Verification and Validation in Meteorological/Climate Simulations

[Bigwig]

Does anyone know of a clear articulation of the verification or validation procedures undertaken in weather or climate simulations? I know V+V is heavily relied upon in the engineering sciences, and less so in the "theoretical" sciences, but I was wondering if there was a lucid explication of the way in which weather/climate models get their credentials given the difficulty of obtaining good experimental data in such a complex system. Any information (especially references to journal articles) would be helpful.

[JohnB]

The short answer is "No". In general V & V procedures are not done.

 

What is done is "Hindcasting", a process where the climate model is run with a variety of forcing values and the results compared to the historical record since 1850 or so. Once the output roughly matches the record, the model is deemed verified and useful for forecasting the next 300 or 3,000 years.

 

Given the number and values of the various tuneable parameters in a climate model it would be very unusual if you couldn't get it to match the historical record. The varying models use different values for climate sensitivity, forcings, etc but do have one thing in common, they all use a value for negative forcings due to aerosol particulates that is exactly right to balance the other forcings and make the output match the record. Isn't that handy?

 

It's like having a ballistics program where you can vary the gravity, wind speed, air viscosity, etc and you tune these factors until the output matches a known ballistic curve. You then use this to make predictions.

 

This "tunability" means that models are generally good in one area and poor in others. This is why we use an ensemble of models, the theory being that by using the average of the models the mistakes will cancel out. Personally I view this as saying that if I have 4 cars, one has no motor, one has no wheels, one has no doors and the last has no seats, that on average, I have a good car.

[iNow]

Oh, rubbish, John.

 

http://scholar.google.com/scholar?as_ylo=2008&q=climate+model+verification

[immortal]

Does anyone know of a clear articulation of the verification or validation procedures undertaken in weather or climate simulations? I know V+V is heavily relied upon in the engineering sciences, and less so in the "theoretical" sciences, but I was wondering if there was a lucid explication of the way in which weather/climate models get their credentials given the difficulty of obtaining good experimental data in such a complex system. Any information (especially references to journal articles) would be helpful.

 

Winter Simulation Conference is a good place for anyone to learn about any simulation applications. Just go to their WSC archive and search for any keyword of your choice say 'weather simulation models' and it will list a series of full pdf papers concerned with the keyword. WSC is sponsored by six technical societies.

[JohnB]

Oh, rubbish, John.

 

Which part? Are the models not compared to the historical record? Articles from your own link say so.

Second one down Pincus et al 2008 says;

Unlike weather forecasts, the climate models used to make long-term projections have not been subject to uniform assessment over time. Climate model evaluation is, in some ways, more difficult than assessing the skill of short-term weather forecasts because climate models solve a boundary value problem as opposed to the initial value problem posed in weather forecasting. This blurs the association between time in the model and time in nature, so model forecasts can’t be compared with observations on a day-to-day or month-to-month basis. Furthermore, climate models are primarily used to make projections over very long timescales (decades to centuries), and these projections cannot be directly assessed until that time has passed.

[5] In lieu of direct assessment of long-term trends, climate models are evaluated according to their ability to simulate present-day conditions and the historical record.

 

Emphasis mine.

 

Attempts to identify which aspects (if any) of the current climate are the best predictors of climate sensitivity or related quantities have so far been unsuccessful, so a model’s skill in simulating the presentday climate may not reflect accuracy in long-term projections. Nonetheless, it seems unlikely that a model that does a poor job simulating the current climate will somehow produce credible long-term projections.

 

Which was my point.

 

As to forcings;

 

The above graph shows the forcings used by the GISS-E climate model. Note the Solar forcing is .3W/M-2. This is not the value used by the IPCC which uses .5W/M-2. So there are two differing values for Solar forcings. Now if we look at the CMIP5 dataset source found here, we find that they use the Solaris data and Solaris say,

It is recommended to use the TSI time series with varying background (second column in ascii files) for the CMIP5 runs and if desired perform additional sensitivity experiments without the varying background.

 

A comparison between the values involced is found also on the Solaris site here. Note that the two values differ by more than .5W/M-2.

 

But wait, there's more. Let's look at the "Climate Forcing Data" over at NOAA, found here. A nice selection of Solar data sets. What does Lean et al 1995 say?

 

Gee, looks like a low in 1700 of 1364.2209 W/M-2 and a high in 1989 of 1368.2017 W/M-2. Of course I'm not a climate scientist, but I can do simple subtraction. Subtracting the 1700 value from the 1989 value gives a difference of 3.9808 W/M-2. In your world this might match the .3W/M-2 that the GISS model uses, but in my world 4 is more than ten times larger than .3.

 

Maybe you didn't like what I said about the mistakes cancelling out. Going back to Pincus et al, from your own link it says;

No individual model excels in all scores though the ‘‘IPCC mean model,’’ constructed by averaging the fields produced by all the CMIP models, performs particularly well across the board. This skill is due primarily to the individual model errors being distributed on both sides of the observations, and to a lesser degree to the models having greater skill at simulating largescale features than those near the grid scale.

 

Which is exactly what I said with the difference that I don't think the average "performs particularly well across the board".

 

It should also be noted that Pincus makes this point;

Clouds strongly modulate the long-term evolution of the atmosphere, and cloud feedbacks on the climate system have remained the single largest source of uncertainty in climate projections since the Intergovernmental Panel on Climate Change (IPCC; see http://www.ipcc.ch) began issuing Assessment Reports in 1990

 

and

There are, however, no standard metrics for judging model skill in simulating presentday cloudiness or related quantities such as rainfall and radiation.

 

So if clouds are important and there is no way of measuring the accuracy of the model WRT to clouds, then how does a hindcast work and get the right answer? Magic? Or fudge factors?

 

We could look at climate sensitivity values, most of those coming from Forest et al 2006. Just so you know, he's the guy who figured out that the temp rise for a doubling of CO2 would be about 3 degrees due to amplification rather than the simple 1 degree that physics says. It's an important paper and has been cited more than 100 times. Nicholas Lewis (a coauthor of O'Donnell et al 2010 and one of the guys that shredded Eric Stieg and his compatriots over at realclimate) has been looking at the data used in Forest and the methodology involved. Well, he would look at the data, but that dog has been rather hungry again and the original data has been "lost", but it took Forest over a year to admit it. While some data is available the two sets differ remarkably and provide sensitivity values of 3 degrees or 1 degree depending on which one is used.

 

Hop over to Judith Currys and read all about it. There are only 680 comments so far, so make a coffee first. Although I'll bet that actually reading the thing is too much trouble and you'll grab a post from Skeptical Science or Realclimate instead. Why read and think when you can let cartoonists do it for you?

 

iNow, look again at the OP;

Does anyone know of a clear articulation of the verification or validation procedures undertaken in weather or climate simulations? I know V+V is heavily relied upon in the engineering sciences

 

No, climate models do not have a clear V & V process as the term is understood in any other field.

 

Now if you think I've been talking rubbish then you should be able to come up with a paper that provides that "clear articulation". Find that and you can say "rubbish", all you provided was a Google list of papers that had the word "verification" somewhere in the text.

 

Put up or shut up. Either give a reasoned argument or don't bother wasting my time. And don't call a post "rubbish" unless you are willing and able to provide some sort of evidence. And Gish Galloping won't work.

 

As to the accuracy of the models themselves a recent paper by Ross McKitrick has a look at this topic.

 

To be blunt the findings were;

In only 3 of 22 cases do we reject leaving out the climate model data, but in one of those cases the correlation is negative, so only 2 count--that is, in 20 of 22 cases we find the climate models are either no better than or worse than random numbers.

 

So 20 out of 22 IPCC climate models are roughly as accurate as having a monkey throw darts.

Edited June 29, 2012 by JohnB