Economic models fail. What of climate?

[SkepticLance]

An article in New Scientist points out a problem with certain computer models.

 

http://www.newscientist.com/channel/being-human/mg19926754.200-the-blunders-that-led-to-the-banking-crisis.html

 

I quote :

"Banks pay enormous sums to lure researchers away from other areas of science and set them to work building complex statistical models that supposedly tell the bankers about the risks they are running. So why didn't they see what was coming?"

 

The answer given is that the models were unsuited to the type of extreme event that just happened. However, it still remains that mega-millions of dollars have been spent to create economic computer models that could guide the financial world to avoid such crashes as just happened. In spite of all the expertise and all the money invested, these models totally failed to predict exactly the event they were set up to predict.

 

I have always expressed scepticism at the reliability, accuracy/precision of global computer models of climate. After the example of the failure of global economic models, how can anyone have confidence in global climate models?

[insane_alien]

well, in the economic environment humans play a much larger role than in the climate.

 

the climate obeys the laws of physics and chemistry and human influence plays a small role so can't really cause a sharp, severe disturbance. economics however, is entirely dependant on humans and doesn't really follow all that many rules. and the rules in place can be broken.

[SkepticLance]

Ultimately, though, both sets of models depend on the average actions of large numbers, and are treated statistically. Individual human actions cannot influence the economic models in any significant way, just as minor weather events will not affect climate models. Both sets of models are immensely complex. Should we not regard both as 'waiting to fail'?

[Mr Skeptic]

Another thing is that you assume good faith on behalf of the bankers. Seeing as they (personally) made out like a bandit, that is hardly a reasonable assumption.

[D H]

After the example of the failure of global economic models, how can anyone have confidence in global climate models?

Association fallacy. Just because economic models didn't work in this particular instance does not mean all economic models are wrong, and the failure of economic models says absolutely nothing about other kinds of models.

[SkepticLance]

To DH

 

As I have always said, the problem is that the models are highly complex with many unknowns. Economic models fail when some factor that has not been accounted for plays a role. Climate models likewise.

[iNow]

I'll repost this since you seem to have forgotten the lesson it brought:

 

 

 

On the subject of models:

 

 

I would first ask that you name a specific model with problems so we might have a real scientific discussion. If you refuse to name a specific model, then you are lacking in academic and scientific integrity. You can make all of the abstract and high level claims you want, but until you name a specific model with accuracy problems, you are simply hand waving.

 

If you name specific models with issues, then we can discuss the margins of error and where that error resides. We can discuss problems in the models inputs, and to what extent those problems cause issues in the results (the outputs). We can even discuss ways to improve the models.

 

However, we cannot do any of that if you continue with your blanket claims and handwaving that models are not to be trusted.

 

 

 

Now, with a substantiated and well supported counter argument:

 

U. study substantiates computer models for global warming — Department of Meteorology

Computer models used to predict climate change are remarkably accurate when measured against actual weather, according to a new study by University of Utah meteorologists. The findings are expected to boost the role of such models in shaping public policy to confront the menacing specter of global warming, generally believed to be caused by rising concentrations of atmospheric carbon from fossil fuel-burning industries.

 

<...>

 

The University of Utah study results directly relate to this highly publicized report by showing that the models used for the IPCC paper have reached an unprecedented level of realism.

 

Another important aspect of the research is that climate models built in the U.S. are now some of the best models worldwide.

 

 

AMS Online Journals - How Well Do Coupled Models Simulate Today's Climate?

Information about climate and how it responds to increased greenhouse gas concentrations depends heavily on insight gained from numerical simulations by coupled climate models. The confidence placed in quantitative estimates of the rate and magnitude of future climate change is therefore strongly related to the quality of these models. In this study, we test the realism of several generations of coupled climate models, including those used for the 1995, 2001, and 2007 reports of the Intergovernmental Panel on Climate Change (IPCC). By validating against observations of present climate, we show that the coupled models have been steadily improving over time and that the best models are converging toward a level of accuracy that is similar to observation-based analyses of the atmosphere.

 

Also:

 

 

 

The above looks pretty accurate to me.

 

 

NOAA has a great site discussing some of the main points in this issue of model accuracy (which includes links to other information).

 

Modeling Climate

 

 

And here is a summary for a wider audience done in 2007 as part of a "responding to the denialists" toolkit:

 

Climate is too complex for accurate predictions - earth - 25 October 2007 - New Scientist Environment

 

 

And below is a whole swath of information specific to models and the challenges to them:

 

RealClimate - Articles on Climate Modeling

 

 

Again, I suggest you respond to my original inquiry and offer a specific model with issues of inaccuracy. Perhaps you can start here:

 

Climate models

 

 

You need to find a specific model, show it’s margin of error, and also how predominantly that one model among many is used in the field when doing this work and forming conclusions.

 

 

Until then, you’re still just waving your hands about and expecting us to take you seriously.

 

 

 

Regarding your challenges to model accuracy, I advise you review these studies which examine that very issue:

 

Performance metrics for climate models

http://www.atmos-chem-phys-discuss.net/8/10873/2008/acpd-8-10873-2008-print.pdf

http://ams.allenpress.com/perlserv/?request=res-loc&uri=urn%3Aap%3Apdf%3Adoi%3A10.1175%2FBAMS-89-3-303

http://www.atmos-chem-phys-discuss.net/8/10873/2008/acpd-8-10873-2008.pdf

 

I also advise the reference sections of each.

 

 

 

Regarding the subject of model accuracy, the IPCC 4, states that:

 

There is considerable confidence that climate models provide credible quantitative estimates of future climate change, particularly at continental scales and above. This confidence comes from the foundation of the models in accepted physical principles and from their ability to reproduce observed features of current climate and past climate changes. Confidence in model estimates is higher for some climate variables (e.g., temperature) than for others (e.g., precipitation). Over several decades of development, models have consistently provided a robust and unambiguous picture of significant climate warming in response to increasing greenhouse gases.

 

<...>

 

In summary, confidence in models comes from their physical basis, and their skill in representing observed climate and past climate changes. Models have proven to be extremely important tools for simulating and understanding climate, and there is considerable confidence that they are able to provide credible quantitative estimates of future climate change, particularly at larger scales.

 

 

 

 

The understanding of the numerous model variables is quite strong, it's just that computers are often not powerful enough to account for all of our knowledge and understanding. However, with new computers, the ability to add our knowledge of more variables into the system dynamics is growing.

 

 

nsf.gov - Office of Legislative and Public Affairs (OLPA) News - Climate Computer Modeling Heats Up - US National Science Foundation (NSF)

"The limiting factor to more reliable climate predictions at higher resolution is not scientific ideas, but computational capacity to implement those ideas," said Jay Fein, NSF program director in NSF's Division of Atmospheric Sciences. "This project is an important step forward in providing the most useful scientifically-based climate change information to society for adapting to climate change."

 

Researchers once had assumed that climate can be predicted independently of weather, that is, with weather having no impact on climate prediction. Now they're finding that weather has a profound impact on climate, a result that's integral to the drive to improve weather and climate predictions and climate change projections.

 

With this boost in computing capabilities, research team member Ben Kirtman, a meteorologist at RSMAS, has developed a novel weather and climate modeling strategy, or "interactive ensembles," designed to isolate the interactions between weather and climate.

 

These interactive ensembles for weather and climate modeling are being applied to one of the nation's premier climate change models, NCAR's Community Climate System Model (CCSM), the current operational model used by NOAA's climate forecast system (CFS).

 

The CCSM is also a community model used by hundreds of researchers, and is one of the climate models used in the Nobel Prize-winning International Panel on Climate Change (IPCC) assessments.

 

The research serves as a pilot program to prepare for the implementation of more intense computational systems, which currently remain a scientific and engineering challenge.

 

"This marks the first time that we will have the computational resources available to address these scientific challenges in a comprehensive manner," said Kirtman. "The information from this project will serve as a cornerstone for petascale computing in our field, and help to advance the study of the interactions between weather and climate phenomena on a global scale."

 

While this research focuses on climate science, he said, by-products of the work are applicable to similar modeling challenges in other science and engineering fields, particularly the geosciences.

 

 

In addition to all of the previous references, that last one flatly and directly debunks your comments about "too many unknowns" made in post #6.

 

 

In closing, here's a good assessment of the climate models that recently came out:

 

Final Report, CCSP Synthesis and Assessment Product 3.1: Climate Models: An Assessment of Strengths and Limitations

 

 

Enjoy.

[Pangloss]

well, in the economic environment humans play a much larger role than in the climate.

 

That's an ironic twist of a reply that alone makes this thread worthwhile. It doesn't counter a single aspect of global warming theory, supporting the notion of human contribution, and at the same time pointing out that human's aren't a major factor in GW.

 

Which is probably why it was subsequently ignored, even though it undermines Lance's theory.

[insane_alien]

yes, i should probably clarify that i am not saying humans are not a factor in GW just not the biggest... yet. i'm sure a large change in energy influx from the sun would have a much larger and more sudden effect than us doubling or halfing the rate we pump out CO2 and other gases. or if the yellowstone caldera erupts.

 

nature is currently holding the trump cards in GW. economics is almost entirely a human influenced construct. nature can still play a role if it leads to a bad crop or a particularly nasty storm wipes out a large number of oil pumps but humans can have an even greater effect and nothing has to happen bar a few stock brokers panicing and dumping a company like a tonne of bricks with no real cause other than they thought it was probably a good idea.

 

unlike people, the earth does not panic.

[iNow]

If I read this correctly, the point being made is that climate is a system impacted by many different contributions, human based CO₂ contributions included. However, economic systems are not physical, not subject to the laws of molecular interaction and conservation of energy, etc... economic systems are purely human, and do not follow the principles of physics, only the principles of human behavior and society.

 

One - Calculated based on known physical interactions.

The other - Calculated based on statistical likelihoods that humans will or will not engage in some behavior.

 

One - Calculated based on historical trends and projections of human CO₂ contributions to the atmosphere.

The other - Calculated based on numbers must less real and tangible.

 

 

About the only similarity here I see is that they are both models. Next thing Lance will be trying to convince us all that his model airplane broke therefore climate models can't work. As DH rightly pointed out above, it's an association fallacy, and failures in economic models have zero relevance regarding the ability of climate models to help us better understand our climate.

Edited October 18, 2008 by iNow
extraneous word removed

[SkepticLance]

iNow

I have told you of climate models that showed serious errors before, such as 100% error in predicting sea ice loss in the Arctic. Since you ignored those points, not a lot to be gained going down that track again.

 

I am not terribly surprised that current models track history reasonably well. The first models in the 1980's did not. However, over 25 years of tweaking to make the models track history better leads (surprise) to models that track the past a lot better. 20:20 hindsight!

 

It is their ability to predict the future that is the clincher. So far, they are predicting nothing more than I can achieve by running a ruler up a graph. Like a good hypothesis, accurate prediction is the test. The example of Arctic sea ice was a failure at predicting.

 

Earlier predictions of Antarctic continental warming proved to be a failure. Interesting that the true cause of the continent of Antarctic cooling is still not fully understood, in spite of several different hypotheses, yet the current models track it quite well. That does NOT come from understanding. Just from a fudge factor. If the models would predict a change in Antarctic continental land mass cooling changing its pattern, and then seeing it happen for real, that would be more impressive.

[iNow]

It is their ability to predict the future that is the clincher. So far, they are predicting nothing more than I can achieve by running a ruler up a graph.

 

Let's see you produce the same outputs these models do with pencil and graph. Either you have math skill so profound that you make Isaac Newton look like a retard by comparison, or you're fibbing a bit.

 

Also, models don't predict the future, they model it based on various parameters. "If this, then that. If some other inputs, then some other response." They are not crystal balls and it's not appropriate for you to disregard them for this reason.

 

They are accurate within a small margin of error... accurate enough that the trends they produce can be relied upon to make decisions and changes.

[Pangloss]

I assume economic models can fail regardless of the unpredictability of human behavior. Incorrect calculation of predictable economic pressures and external causes (natural disasters, etc). Am I wrong?

[bascule]

I have always expressed scepticism at the reliability, accuracy/precision of global computer models of climate. After the example of the failure of global economic models, how can anyone have confidence in global climate models?

 

This argument is a category fallacy. Newtonian mechanics is a model too. So are general relativity and quantum mechanics. Because economic models failed should we conclude that Newtonian mechanics, general relativity, and quantum mechanics are failures too? They're also models after all.

 

You seem to be arguing that because something is a model that it's inherently untrustworthy, putting forward one example of a model, and not even one that's rooted in the physical sciences, but in the more abstract concept of economics.

 

If you have problems with climate models, how about you point out the specifics, as opposed to pulling guilt-by-association?

[swansont]

It is much more difficult to test the individual components of an economic model. OTOH, solar radiation, the absorption spectrum of water and CO2, etc. can be independently tested.

 

The bottom line is that economics is not science. D H was right in terming this an association fallacy.

[SkepticLance]

Economics modelling is not science - true. Neither is climate modelling.

 

Science requires predictive testing. Climate models must make predictions that are testable. Until they do, and until those predictions are tested, then there is no science.

[iNow]

Economics modelling is not science - true. Neither is climate modelling.

 

Science requires predictive testing. Climate models must make predictions that are testable. Until they do, and until those predictions are tested, then there is no science.

 

Ermmm... They're tested every time we take an actual temperature reading, every day of every week of every month of every year.

[Pangloss]

It is much more difficult to test the individual components of an economic model. OTOH, solar radiation, the absorption spectrum of water and CO2, etc. can be independently tested.

 

The bottom line is that economics is not science. D H was right in terming this an association fallacy.

 

I recognize the technical definition of science, but if human unpredictability rendered science to be non-science then we'd have to toss most of medicine on the scrap heap as well. Just look at all the early drug testing that gets turned upside down by more detailed testing after the product hits the market.

 

Just because something is difficult to predict doesn't mean there's no value in limiting the variables and testing on that basis anyway. And if we can do that with medicine and the environment and learn something, then we can also do it with economics and learn something.

 

And in all three cases you take the result with a grain of salt, knowing that in the real world the variables aren't limited like they are in the experiment.

[insane_alien]

Economics modelling is not science - true. Neither is climate modelling.

 

Science requires predictive testing. Climate models must make predictions that are testable. Until they do, and until those predictions are tested, then there is no science.

 

lets see, do climate models make predictions? yes, they predict that the temperature should continue to rise, they also predict by how much it should rise.

 

are these predictions testable? yes, we wait and take temperature measurements.

 

is this science? yes.

[swansont]

I recognize the technical definition of science, but if human unpredictability rendered science to be non-science then we'd have to toss most of medicine on the scrap heap as well. Just look at all the early drug testing that gets turned upside down by more detailed testing after the product hits the market.

 

Just because something is difficult to predict doesn't mean there's no value in limiting the variables and testing on that basis anyway. And if we can do that with medicine and the environment and learn something, then we can also do it with economics and learn something.

 

And in all three cases you take the result with a grain of salt, knowing that in the real world the variables aren't limited like they are in the experiment.

 

Medicine straddles the line. There are plenty of non-science-based treatments considered medicine (by some). Acupuncture, herbal treatments, homeopathy, magnetic bracelets, reikei, various "cleansing" routines, etc. The list is really long.

 

Pharmaceutical testing, though, is science. You have a specific reaction that has been isolated and the substance is tested for efficacy and side-effects, using a double-blind procedure. The very fact that detailed testing gets (potential) products thrown out is a point in the scientific column.

 

Do economic models do that? Can you isolate the effect and see if it, and only it, has the desired outcome? Can you test a small population and see if the model works with just them?

 

Economics modelling is not science - true. Neither is climate modelling.

 

Science requires predictive testing. Climate models must make predictions that are testable. Until they do, and until those predictions are tested, then there is no science.

 

How is one supposed to take the claim "climate models are not tested" seriously?

[Sayonara]

This is going to be another one of those threads that goes on and on and on despite the premise being pointed out as a fallacy by everyone replying, isn't it?

[insane_alien]

probably.

[SkepticLance]

I made the comment earlier that I could make predictions using a graph and a ruler. The examples of prediction from climate models that iNow and insane alien are also examples that can be predicted the same way. Temperature increase over the last 30 years averages at 0.08 C per decade, excluding short term fluctuations. I predict, with my ruler, that the world will warm, on average, by about 0.08 C over the next decade (with an error factor of plus or minus 0.03 C). Does this make me a computer model?

 

Predictions that are just plain bloody obvious have no scientific value. For a global climate model to 'prove' itself, it needs to make an unexpected prediction and be shown to be correct.

[insane_alien]

do you realise the idiocy of what you have just said?

 

thats like saying any theory of gravity has to predict an object spontaneously falling upwards at several thousand g for it to be proven.

 

just because the model says nothing spectacular is going to happen doesn't mean it is wrong especially if nothing spectacular happens when it says nothing spectacular will happen.

[Pangloss]

Don't be rude, IA.

 

 

Pharmaceutical testing, though, is science. You have a specific reaction that has been isolated and the substance is tested for efficacy and side-effects, using a double-blind procedure. The very fact that detailed testing gets (potential) products thrown out is a point in the scientific column.

 

I agree that is science, and I think the same thing happens with economics and environmental modeling. Just to be clear, I'm not arguing that the models aren't science, I'm just suggesting that economics deserves a little bit more respect. I recognize the commonly accepted technical/academic distinction that makes economics not "science" per se, but that doesn't mean economists can't take a scientific approach when creating an economic model.

 

In pharmaceutical testing they minimize, eliminate or ignore the variables they don't have any control over, such as, oh I don't know, how about whether or not the patient actually swallows the pills you give them, or whether they were honest about their medical backgrounds or other drugs they're taking. Not to mention stuff they may not even be aware of (environmental factors, dietary considerations, accidental overdose, etc etc etc).

 

So, as with economic models, we have to take those early studies with a grain of salt. And the same can be said of environmental models -- even those who create and run them express their limitations for the record. They're not reality. They're only as good as their assumptions.

 

That doesn't mean they're wrong.