Escaping statistics: the starkest examples of fields not lending themselves to statistics

[guitarplayer]

There are many STEM inclined people here on the forum. What areas of life / fields of knowledge would you say are not particularly suited to be tackled with statistics?

In starting the thread my aim is to avoid the 'when you have a hammer everything looks like a nail' fallacy as it looks like there is a year of full immersion in statistics ahead of me. Also from the point of view of parsimony (in that some fields / areas of life might be tackled by statistics, but there are much quicker and simpler ways e.g. relying on intuition and in-the-moment judgement, like in a day-to-day relationship with SO).

[IlliniDave]

From an engineering perspective statistics have their place, but it's pretty rare that problems get 'tackled' by them--at least in the sort of engineering I was involved in. They help in defining the problem to be solved (e.g., define the load characteristics a power system has to cover) and assessing potential design solutions (e.g., how often might a given air traffic radar design drop track on an aircraft). I imaging the same is true for most of STEM, where in general you are concerned with how things work from theory and usually have a starting point, rather than hunting through information where you don't have the laws of physics undergirding the situation in question trying to extract some sense out of it. I'm sure there are many exceptions though. A lot of signal processing relies heavily on statistics.

So statistics coexist well with most of STEM, and probably have more of a driving role in fields where causal relationships and even correlation aren't necessarily known in advance. I'm thinking a lot of social sciences, certain aspects of biology/ecology, etc.

[Sclass]

Depends on the nature of the problem. Some things can be described by a few numbers. Others need a lot of numbers.

Building a statistical model is more natural than becoming a STEM person. When I see a neighbor on a political, racist, ageist or sexist rant etc. about some observation they don’t particularly like I realize they’re operating off some flawed statistical model in their mind. They couldn’t tell me what a standard deviation is if their life depended on it. But they are making broad sweeping statements based on statistical data they’ve mentally tabulated or other misguided people have tabulated for them.

Maybe they’re explaining how their 401k will work out in the long haul. That seems to be the topic of the day in my locale.

My favorite ignorant being is the brush rabbit in my backyard. He isn’t the least bit scared of me. I can get up to 6’ of him before he adds a few feet between us and continues to eat my lawn. We just ignore each other now. He’s a wild animal living in an area with coyotes, bobcats and mountain lions but he realizes I’m no threat. The fact he’s still alive here means he knows the difference. However he’s ignorant. I have a fully pressurized precharged pneumatic air rifle with an integrated silencer on a rack just inside the door. I’ve shot, dressed and stewed so many rabbits I cannot even count them. But to him I’m just a slow human who eats pizza like 68% of the other humans he sees in the backyards.

For specifics I see wealth distribution as a problem where simple statistics falls apart. I read a lot of reports that seek to understand wealthy clients and I realize there is little data to constrain a good statistical model. Figures for median and mean net worth indicate non normal stats. They don’t represent the current state nor do they predict any dynamic responses. I.e. you don’t know how full the bucket is nor understand whether it leaks or tips over. Therefore any simplified figures that follow are likely misleading.

[7Wannabe5]

Situations involving small sample size, lack of independence, feedback, discontinuity, and asymmetry.

[Sclass]

I can see the others, but how does feedback mess up a statistical model? Just asking out of ignorance.

I was thinking, maybe wrongly, that STEM people are good at deterministic models. That’s what we are hammered with in school. Mass on a spring, ball rolling down a hill etc.

Non STEM seem to use stochastic models conditioned by experience or narratives. Maybe I’m wrong on this. I have this image of friends ranting “you know those people, they all do that!” Conditioned by TV, political narratives and wizdumb.

[7Wannabe5]

@SClass:

For instance, statistics on efficacy of a pesticide which do not account for feedback mechanism of evolutionary or behavioral adaptation of pest and/or tendency of farmers to start watering down the pesticide to save $$ once it has proven highly effective. Since humans are HIGHLY adaptable, almost any statistic having to do with human behavior will be subject to this problem.

[jacob]

There are lies, damned lies, and statistics.

I can think of three problems.

Generalization fallacy: The sample space does not represent the population space. For example, some people deliberately lie in political polling for shits'n'giggles or due to shame (The shy Tory effect). Much psychological research has been carried out on WEIRD people (google it) who may not be representative of humans.

The population space itself is changing. This is a big issue in finance and economics. Captured by the comment that "in a bull market, everybody is a genius". Beaten to death in most discussions about the 4%-rule. Is the 20th century representative of the 21st century. Even at shorter time intervals, a model that was fitted to 2006 and backtested to statistically confirm it in 2007 would fail in 2008 because the population fundamentally changed. Technically, this is similar to the problem above, but it appears different because the attitude is that "I already have decades of data". If the population space is nonstationary, you gotta decide what to do about that and that requires an executive decision that is outside the realm of statistics.

Statistics can be compared to reductionism as a way of trying to understand a population. With reductionism (scientific experiment), one isolates a small part of the population by cutting off dependencies and data that one believes to be irrelevant. Cause and effect, which is really nothing but a ~100% correlation, is then studied. Statistics goes the other way by looking at the whole population (or at least a relatively huge sample) but ignoring all dependencies by turning everything into "independent variables" ... or at least variables with much simpler dependencies than reality admits. From a STEM perspective, think of understanding water. Reductionism would study a single water molecule and maybe even hydrogen and oxygen in isolation. Write down the Schroedinger equation and claim that water has been understood, in principle. However, knowing the Schroedinger equation says nothing about the color, wetness, steam, etc. of water. Going the other way---statistical physics was invented in the late 1800s---ignore all the dependencies (Collisions between molecules) and we can not use statistics to relate temperature and pressure to the statistical distribution of velocities. Again, though just because you know the temperature and the pressure of the gas doesn't mean you understand wind and weather.

So beware of statisticians without domain knowledge. Statistics, like mathematics, is a tool that destroys (ignores) certain structures to draw conclusions about the whole. Accidentally (or deliberately, remember the lies) eliminate crucial structures and one will confidently derive a false conclusion.

[guitarplayer]

Thanks all! Apologies for not specifying, I actually considered statistics to be part of STEM! But no worries, I get the responses.

@ID, in other words statistics are of not much use in tightly coupled relationships between things.

@Sclass, so the overgeneralizations are a lack-of-time-or-will-to-ponder situation, ignorance. Rabbit story is the fat tail Taleb's style situation. Wealth distribution is the lack of enough data situation.

@7w5, @Sclass, yes feedback is a good one actually, I see how it can ruin any model. Self-referentiality of a system, or autopoiesis. Related also Goodhart's Law in the sense of the statistical model applied to some objects who are also subjects in their own right changes how the subjects govern the objects they have power over. In other words:

Here is a model that shows what people do. Then people see the model and based on it change their behaviour.

Hence, build a model without ever telling anyone!

@jacob, feedback seems to relate to the example of generalization fallacy you have given. An act of asking question impacts its answer. The population space changing is related. For a long time I had the 'small-slow-simple' triangle on my whiteboard for readily actionable problems, that was from your book. The last point - they do talk about statistics being an art as much as it is a tool of science and in this statement I think they hint at not falling prey of reductionism. But the point is perhaps the most vital in terms of having a hammer and not trying to hammer meaning, soul or god with it.

[IlliniDave]

Thanks all! Apologies for not specifying, I actually considered statistics to be part of STEM! But no worries, I get the responses.

@ID, in other words statistics are of not much use in tightly coupled relationships between things.
...

I'll have to think about that ... it's true in a sense. But the wider your perspective the less true it gets.

In my mind I always think of statistics as "probability and statistics" because that is what my math courses on the subject were titled. Worth noting that at my university at least one semester of probability and statistics was required for an undergraduate degree in electrical engineering, so while the design of a widget or system is typically not carried out by gathering a bunch of data and reducing it with statistical techniques, prob/stats is still part of the toolbox.

In school a student might be assigned a problem on a test like: Design a single-pole low pass filter with a gain of 2.0 and a 3dB frequency of 10kHz using an ideal op amp." That's something one can do with a pencil and a calculator.

But out in the real world much more of a systems approach is necessary. I don't mean the type of system theory often discussed on this forum. I mean both the process of going from an idea to a thing that successfully operates out in the real world being a system, and the thing itself often being a system, or "system of systems" in its functionality. When you widen the lens beyond the paper design (what is emphasized in school) to include sourcing parts and material, building the thing, integrating the sub-parts, evaluating the thing, then turning it loose in the wild, prob/stats often come into play.

In part it is because humans can't build things with infinite precision. So if you are going to build a million widgets each comprised of a thousand parts where across the million each of the thousand parts none of them are completely identical, you might want to know how many of your million widgets will work as intended when put together.

In part it is because the universe is a noisy place and often the only practical way of describing a situation is as a random process, A simple example is the problem of extracting signal from noise in radio frequency transmission where statistical models/distributions are used to characterize the environment.

Together those get me to the point I'm wandering towards. One can oversimplify a little and divide the upstream part of engineering into two primary tasks: design and analysis. At least in the field of electrical engineering the actual design generally does not directly use statistics. That's the realm of Ohm's Law and Maxwell's Equations and semiconductor theory and Bode Diagrams and such. Stuff that you can work out even with a slide rule. But the analysis side in some applications leans heavily on prob/stats in the sense of modeling the environment and/or the object to be manipulated as random or partially random processes. In my 34 years of herding electrons I couldn't begin to count how many Monte Carlo trials I churned out and/or crunched through. In many of the systems I worked on the probability of success was never 100%, and being able to describe the range of outcomes as well as understanding correlations and their impact were important "products".

And nearly all the systems used feedback. Some even employed Kalman Filtering which is sort of an exception in that it's a case of a "design" technique that uses some statistical calculations to reduce measurement errors.

Engineering is a little different in that it is not a science that observes nature (nature writ large to include the whole physical universe) and attempts to figure out how nature works. It's a discipline that manipulates nature to achieve an outcome. So in an engineered feedback system, and even one with built-in adaptation mechanisms, statistical analysis has a place, but statistics aren't what tackles the primary problem (e.g., how to make an autopilot to land a man on the moon or how to make faster RAM). Rather, it is more broadly useful for answering the question, "If we try to solve the problem this way, how well will it work?"

[Sclass]

@ID I get your point that an electrical engineer designs by solving a bunch of deterministic approaches. Some electrical engineering problems won’t have a shred of stats in them till your academic paper is rejected for not showing error bars on a figure.

However statistics is incredibly important when designing with parts and processes that have tolerances based on real data. The most basic example is the gold and silver band on a simple resistor. The electrical engineering student solving homework problems may only worry about the resistor’s nominal value but the tolerance band can have a huge influence over the design process. Generally if you’re going to make more than one copy of a thing you need a lot of statistics. The most involved statistical analysis I worked on was a Monte Carlo simulation of the tolerance stack up of a collection of several components to see how their uncertainties influenced the sensor’s accuracy. It took more engineering man hours to pull this off than inventing the actual sensor in the first place. Today there are billions of copies of this chip in use. The copycats not only copied the design, they undoubtedly copied the tolerance specs which make the sensor cheap yet reliable.

As I am writing this I’m trying to work out a test parameter that I got wrong on a device I’m manufacturing in the low thousands. I have one perfectly good board that doesn’t pass due to a LSB tolerance issue from my chip vendor. They’ve either had a process walk out or I missed a footnote in their data sheet regarding variation in their expected performance.

Just saying, it depends on what you’re engineering. I get what you’re saying but it’s only one side of the coin.

Wait…my apologies that’s what you just said. I didn’t read your post carefully enough the first time.

[7Wannabe5]

OTOH, it can sometimes be fun to use statistics for purposes for which it was not intended. For instance, I used* to generate statistics such as 72.5% of grouchy old men will offer to buy me dinner after they meet me for coffee. Obviously, "grouchy old men" are not really as indistinguishable in their behavior (or emotions likely to motivate behavior) as molecules in a box, so it would be "wrong"** on multiple levels to include such a statistic in a spreadsheet projecting monthly grocery budget.

*Before I declined into complete Old Pumpkin Lady functioning.

**In fact, I recently learned that the tendency to use statistics in such a cold-hearted manner may be indicative of level Red functioning on the spiral dynamics model.

[daylen]

Philosophy, especially when considered as a dialectical activity. Philosophical questions which can be illuminated from data quickly get re-framed into scientific questions. Perhaps also something like "common sense" or "pragmatic living" where life experience tunes judgement yet experiential replication is of low fidelity.

[WFJ]

Fundamentally, statistics mainly informs one the probability between variables is not random. The probability X to Y relationship is not random is 10%, 5%, 1%. This is why out of sample tests and robustness tests using alternative variables is vital to using stats for anything but brain exercises. It is CRAZY east for someone marginally trained in stats to "torture the data until it confesses".

The weaknesses in stats reveal themselves when samples do not represent populations, populations are unknown or change over time, have different regimes, distributions are not known, populations have outliers, variables are not measurable, probably a few more hundred critiques and why statistics should be used with great care and with great skepticism. Especially today where packaged stats programs can allow anyone in the top 80% IQ to produce output in a few minutes that required a room full of PhDs to produce 20+ years ago. I almost can't contain myself when I see/hear someone provide junk stats to further their opinion and browbeat anyone who mentions the obvious errors because they pressed a few buttons on a canned program wearing a white lab coat and have letters behind their name.

In STEM, my training is 10+ years ago, but don't remember any stats in physics classes (mostly calculus) but remember a lot of stats in Biology (genes, viruses, some others but fuzzy).

Although above is my opinion, stats are used successfully in more areas than anyone could imagine. I assume everyone on this forum has watched all 35 Fast and Furious movies several times and wondered why the movies have progressed from a street racing series to a gun-wielding car chase series... statistics.

https://www.bloomberg.com/graphics/2017 ... ify%20wall

https://www.youtube.com/watch?v=iIY5b1JMvGs

[IlliniDave]

...

Wait…my apologies that’s what you just said. I didn’t read your post carefully enough the first time.

Ha, yeah, easy to do as I tend to get wordy. No worries. My very first assignment coming out of school was running "tolerance analysis" on circuit cards using SPICE (company was still running on mainframes then and PSPICE was still a few years away).

[IlliniDave]

...

In STEM, my training is 10+ years ago, but don't remember any stats in physics classes (mostly calculus) but remember a lot of stats in Biology (genes, viruses, some others but fuzzy).

...

Mine is even older. Aside from prob/stat class I didn't encounter much in the way of stats until grad school when I had classes that covered things like stochastic systems, adaptive control, and advanced signal processing/Kalmann filtering.

But as I recounted above, once I moved on to the real world (which happened before graduate school) the first engineer thing thing I did was to start running Monte Carlo analyses. As soon as one sets foot in the nonidealized real world stats begin to become relevant. Manufacturing is an imprecise endeavor, and measurements are never perfect. A dynamic universe presents never-ending variability. I spent a significant chunk of years in the world of modeling and simulation. Despite the fact that we don't have precise deterministic equations to model things like future atmospheric conditions/weather or ocean waves, we still have to make planes that fly and boats that traverse the sea, and use of stochastic models derived from statistics of observations is commonplace and often adequate.

And you're right statistics are nearly ubiquitous. Offhand I really can't think of a whole lot of endeavors that don't employ them to some degree.

The challenge is, as you mentioned, to avoid misusing them either out of ignorance or malice. An average is a simple calculation but what the resulting number means varies significantly (based on factors such as the ones you listed). The calculations are benign but how they're used can be fraught with problems. Unfortunately in many areas of extraordinary complexity they are the only concise way to characterize what we can observe. And due to inherent limitations they can easily lead to fallacies like confirmation bias.

[WFJ]

One of the largest issues in stats is the tendency for "leaders" to shop for analysts to torture the data until it confesses directly in line with the "leaders" opinions. If you don't find what the leader believes, they will just replace you with a more complaint (corrupt) analysts who are unfortunately plentiful. It is monumentally easier to do this today vs 20 years ago and canned programs and almost unlimited secondary data sets allow almost anyone with marginal intelligence to press a few buttons and find "significance" in many fields. When someone with a brain brings up simple violations with the underlying data, methods or robustness critiques, they are shouted down, ostracized and badgered into submission, the "science" becomes meaningless and nothing more than propaganda. Many fields and have fallen into this trap for a variety of reasons and not sure how to solve this without massive accidents revealing the lunacy.

[zbigi]

Hah, that sounds similar to what I've seen in the large bank I worked at. Once, a very important (and self-important) director was touting the benfits of our new shiny big data platform. The biggest benefit he ultimately saw was the ability for our data scientists to comb through the data and find all kinds of correlations that no one in the bank knew existed. I pointed out to him that this is basically considered a huge no-no amongst scientists and even has a derogatory term for it ("going on a fishing expedition"), because, if you have large amount of independent random variables, just pure randomness dictates that some of them will be correlated by accident. He paused, pondered it, and then basically went on with his presentation. A couple weeks later, I heard him repeat the same sales pitch to another group of people in the bank...

[Mister Imperceptible]

Potential future central planners:
https://m.youtube.com/watch?v=jIE6F3TOGug


excited for the future

[WFJ]

Potential future central planners:
https://m.youtube.com/watch?v=jIE6F3TOGug


excited for the future

That was an SNL skit, right? These people can buy and sell trillions of dollars in any asset for any reason at any time, swamping any prudent financial decision and wipe out years of capital saved from labor .

I'm not strong enough to carry gold but might buy a wheelbarrow after watching the first few minutes of that cult meeting.

[Jean]

the most common mistake i see is to treat individuals like they are the average of the population you assigned them to.
The other mistake i see is completly refusing statistics because of the first mistake i mentionned.