Does anyone know about methodologies of developing climate models?

[zbigi]

I see that my other thread on this topic got quickly locked (as Stahlman predicted before I got answers to the questions I had, so I'm going to re-ask them in the strictest technical and non-political terms possible (as my question is about facts, not politics).

So, I'm wondering about methodologies of developing climate models. They're all done in software now, so it's essentially a large software project - something I (and many other forum members) have a lot of experience in. The state of the art climate models are reported have at least a couple of millions lines of code in them. That's a number which implies an absolutely staggering complexity, which arguably is too large to have any chance of ensuring correctness.

For comparison, code for critical system like F-35 is of similar complexity, but the military spent more that $1 trillion developing it (I don't think anything near that was spent on climate modelling), AND it's inherently more testable. Whereas my uninformed suspicion is that code for climate models is:
1. Developed by underpaid and overworked grad students and postdocs who have every incentive to overlook potential errors in code (modern academia has the most corrupting incentives) as long as they're reasonably certain that no one will call them out on it.
2. Developed without nearly enough rigor and methodologies across the whole software lifecycle. Scientist are basically not software engineers.
3. Tested against the authors's biases. I.e. they run the model and if it predicts something surprising (to them), they go back to the code and look for bugs - but if it predicts something that confirms their biases (like "+4C till 2100"), then they declare mission accomplished, as the model is clearly working now.
3a. They test against past data for which the correct predictions are already known to see if the model predicts them. That's probably the best they can do, but of course there's a huge risk of overfitting (https://en.wikipedia.org/wiki/Overfitting). You can build models which fit both to the past data and to your biases at the same time.

On a broader level, since it's acknowledged that Earth's climate is a chaotic system [1] (https://en.wikipedia.org/wiki/Chaos_theory), isn't modelling it also an activity ruled by chaos? I.e. one where a tiny change to the model's structure or constants or inputs will affect model's outputs in a major way? Doesn't this make a problem of climate modelling hopeless?

I'd love to have someone knowledgeable answer this doubts of mine.

[1] Or is it just weather? Can we even know if climate is a chaotic or non-chaotic system?

[theanimal]

A quick search of your title shows plenty of results for further study that discuss what you are asking.

https://duckduckgo.com/?q=methodologies ... 6-1&ia=web


Climate change and political threads revolving around theory have worn out their welcome on this board over the years. Jacob is the only moderator on the board so it may be wise to take the locked thread as a hint. Or persist, but prepare to face the wrath of our DL

Also if you go through the previous climate change threads, you will find much of what you are asking discussed before.

[zbigi]

I see @jacob addressing some of the worries about limitations of the models with the following: "From a strategic/scientific/modeling point of view this means that you shouldn’t focus so much on WHEN things happens as on WHAT’S NEXT/WHERE ARE WE". I agree with that. I am convinced that AGW is happening, I'm just not convinced that we can predict the rate of change, which is unfortunately a crucial variable when devising a response.

[zbigi]

Yes, I did some search on the Internet beforehand. I haven't found anyone addressing most of the specific issues I raised, esp. in a satisfactory way (one site for "debunking climate sceptics" said that since the models accurately predicted the past, they will most likely work for future data as well...).

It's obvious for anyone working in software that creating an error-free large-scale software system is virtually impossible (humanity have not figured out how to do it yet, it might be very well beyond our capacities, unless we're willing to spend trillions on it). On top of that, climate models are inherently untestable (I'm talking about verifying for correctness of just the act of mechanical transcription of scientists' ideas from papers into code, not whether the ideas themselves are scientifically correct and accurately reflect reality).
And yet, no one even mentions this specific issue. It's almost as if the scientific community was afraid of admitting that the models are most likely wrong (because chances that they are bug-free are astronomically low) and they only hope that the errors don't affect the calculations in a major way. That's because they know that the public will only hear the "models are most likely wrong" part and any intelligent debate about it will be impossible.

[Stahlmann]

I think if you have time and energy for moral and "higher" stuff, you can sort topics in "Politics and eternal disagreements" by highest number of replies, then read ones about AGW or especially jacob's replies. I think there were SEs with similar questions.

I don't write this to feel better, but simply to transfer your energy in different direction.

[zbigi]

Thanks, I'm unfortunately still working full-time so I don't have that much time for self-education. I was hoping that, since climate change is such a pivotal topic, such fairly basic questions would be neatly answered on the Internet somewhere (and they are, but in a dismissive way). Maybe I just need to luck into meeting a climate scientist sometime and chew his/her ear on it (or, stop working full-time and go down the rabbit hole on my own).

[Stahlmann]

Create simple scrapper which will spew jacob's post from above range .

[Sclass]

. On top of that, climate models are inherently untestable (I'm talking about verifying for correctness of just the act of mechanical transcription of scientists' ideas from papers into code, not whether the ideas themselves are scientifically correct and accurately reflect reality).
And yet, no one even mentions this specific issue.

Earth sciences (among others) don’t have this luxury. I don’t think that makes them any less of a science. It does however temper my expectation when presented with results.

Earth scientists do their best given their lab. It’s kind of insulting to imply what they are doing isn’t science or lacks value because it lacks scientific control for strict falsification.

[zbigi]

Earth sciences (among others) don’t have this luxury. I don’t think that makes them any less of a science. It does however temper my expectation when presented with results.

Earth scientists do their best given their lab. It’s kind of insulting to imply what they are doing isn’t science or lacks value because it lacks scientific control for strict falsification.

I'm not saying that what they do lacks value. I'm only saying that they perhaps should be very cautious about making strong statements on quantitative conclusions (as opposed to qualitative) from their work, knowing that the methods they have at hand are inherently limited.

[zbigi]

Create simple scrapper which will spew jacob's post from above range .

I'm going through the posts manually... So far, I've mostly encountered rebuttals on the physics front, with the only comment about modelling being the one I already cited (and which, if anything, is confirming the doubts that I have).

[mountainFrugal]

If you are perceiving resistance start here: viewtopic.php?p=134324#p134324

If you are still interested in learning more about the following sub-fields of modeling (not specific to climate modeling, but used in nearly every contemporary climate modeling study specifically for addressing study limitations in some combination) you can when you have more time:
1) Ensemble modeling (within model parameters/across models/ across scales / between research groups)
2) Leave one out cross-validation (cross-validation in general)
3) Sensitivity Analysis
4) Model comparison
5) Training and test data standards
6) Open-sourced models
7) Peer Review

[zbigi]

Thanks for the list. Yeah I'm aware of some of these concepts (I did a Masters in Machine Learning back in the day). The question is - to what degree they're used, and (more significantly), what kind of conclusions/guarantees we can draw when applying them.
At least in the ML world, the approach is mostly empirical - i.e. "we see that applying ensemble modeling helps", "we see that adding cross-correlation improves result and therefore should be adopted" - but there are no mathematical proofs on the error upper bounds for either of those. I.e. even if we apply them all, anything can still happen (the model can still be very wrong). These methods are basically "best effort" aka "we do what we can", and are far from guaranteeing any scientific certainty even in the ML field, where the outputs are known beforehand (and, because of those upper bound guarantees on errors do not exist, the ML models can sometimes be embarrassingly wrong). And ML is trivial compared to climate modelling...

[Miss Lonelyhearts]

I see that my other thread on this topic got quickly locked

I'm wondering

it's essentially … something I …
have a lot of experience in

which arguably

my uninformed suspicion

the most corrupting incentives

hopeless

I'd love to have someone knowledgeable answer this doubts of mine.

I'm just not convinced

in a satisfactory way

It's obvious

no one even mentions

Yeah I'm aware of some of these concepts

I don't have that much time for self-education

https://youtu.be/SbF1bRwxIWY

[mountainFrugal]

If you read the above link to Jacob's discussion then you will understand that this is my last post in this thread. Godspeed in your understanding quest @zbigi. .

There is a difference in how these techniques are used in machine learning and how they are applied in process models (based on physics). This is why I said they are general techniques, not specific to climate modeling. If completely different models written at different regional scales written by scientists from (mostly) first principles work across regions (outside of the one the model was developed for) or converge on similar answers with similar sized uncertainty (between model comparison across research groups), this suggests the scientists know what they are doing. Additionally, starting a model back in the 1800s and leaving large pieces of the more recent time series out and having the model predict within a predefined margin of error the climate of a region today suggests they are calibrated pretty well. They are not built to predict the exact temp of X location on Y date some day in the future (which may be a part of the confusion). But they can give you an estimate with the 95% CI of the model(s) runs using the various techniques. These models are then averaged for predictions as part of say the IPCC report (what is meant by ensemble most of the time in this context). All of the individual studies are peer reviewed and then summarized in a multi-national meta-analysis style report made available to the public for free. Read the summary for policy makers for the most recent report here: https://www.ipcc.ch/report/ar6/wg1/#SPM . Maybe an email to one of the thousands of climate scientists on this report (or look up your home country for a local public climate science talk), could be a better use of your time to get to the answers if this is unsatisfactory?

[zbigi]

https://youtu.be/SbF1bRwxIWY

I'm going to give you more courtesy than you gave me and type in my response instead of replying with a link to an off-topic 14-minute video.

I see that as a problem if the education materials on the "biggest issue of our time" don't answer the common doubts that a semi-informed person might have (that's why I started this thread - I couldn't find anything on the Internet). There can be a couple possible explanations to this:

1. It's all so complex that explaining it to people who are not willing to devote years (or at least months, assuming prior knowledge) to building understanding is hopeless.

2. There's no funding for doing any education for people who are not complete laymen - i.e. an unfortunate policy fail.

3. Addressing any of these issues in public is a suicide among scientific community (i.e. you're being labeled as a denier and your career ends there). The doubts can be discussed among the pros, but "we" (the scientists) need to present an unified front to the masses - similar to how there were separate, secret newspapers for the party elity in communist China. I'm far from saying that it's definitely the case, but given how corrupt modern academia is (even Jacob admitted it multiple times), I don't think these guys can be trusted very much. That's unfortunate because it would be great to have trustworthy institutions for scientific discovery - but unfortunately, we've elected politicians who screwed it all up and now we're left with the current cargo-cult, money-driven caricature of academia.


In any case, it's more and more tempting to take a year off at some point to do a Msc in climatology at some research institute, just to see how the sausage is made. For whatever reason (any of the above three, or something else entirely) finding this info on the Internet is hard.

[Quadalupe]

Have you considered corresponding with authors of climate change models or climate science researchers in general? I think if you write polite emails with some well thought out questions, without giving of the vibe of looking for some kind of 'GOTCHA' moment, you might get useful replies.

[jacob]

Short answer: Yes.

For whatever reason (any of the above three, or something else entirely) finding this info on the Internet is hard.

It is on the other hand quite easy to find in undergraduate textbooks which will walk you through the process step by step. The problem with that approach is that, much like the follow-through in reading the economics curriculum, very few care to pick up a book and read it.

Reading would be a much faster way to clear up misconceptions about how the sausage is made than a Q&A session expecting "quick and simple"-answers to complex questions. The reason is that a book presents information within a developing framework whereas Q&A conversations (or google searches) are devoid of context. For example, machine learning and physical modelling are nothing alike (In a former life I was a postdoc creating physics models of stellar atmospheres and I have also worked on (simple) machine learning as a quant.) but you're asking your questions or probing for answers with a ML mindset and would likely interpret my answers within that [ML] mindset as well which would be wrong. "Not having much time for self-educating" doesn't bode well in terms of making any progress on that front.

Due to the [almost universal] lack of effort in self-educating, I'm shitcanning any and all climate threads. "Asking questions" is simply too inefficient relative to the amount of moderation work this topic generates. Same goes for COVID and other culture war issues.