Empirical Evidence of Mt. Stupid

Move along, nothing to see here!
Post Reply
Tyler9000
Posts: 1701
Joined: Fri Jun 01, 2012 11:45 pm

Empirical Evidence of Mt. Stupid

Post by Tyler9000 »

I found this article and associated study co-written by David Dunning (of Dunning-Kruger fame) interesting:

Research: Learning a Little About Something Makes Us Overconfident

It's a nice primer on how overconfidence works in beginners and experienced people alike, and it covers examples in several different fields. One of the more interesting details to me is how measurable accuracy never reaches confidence level even for subject-matter experts. So keep that in mind the next time you look down a few Wheaton levels completely sure of your own footing on the ladder. ;)

jacob
Site Admin
Posts: 13168
Joined: Fri Jun 28, 2013 8:38 pm
Location: USA, Zone 5b, Koppen Dfa, Elev. 620ft, Walkscore 73
Contact:

Re: Empirical Evidence of Mt. Stupid

Post by jacob »

Eh, that paper was just about beginners, e.g. 60 trials and <1000hrs of flight time. Maybe they can do the other end too?

The full spectrum looks like this: https://graphpaperdiaries.com/2017/08/2 ... ger-graph/ but this is about your own perception of your percentile vs your actual percentile. Beginners demonstrate the DK-effect (the gap(*) between the graphs). Experts demonstrate Impostor Syndrome which is the opposite (experts are actually more skilled than they think they are ... at least relative to other people). If anything, this should indicate that once you make it out of the Mt Stupid valley, you're probably further ahead than you think.

(*) Plotting the difference instead creates the Mt Stupid graph.

Noting these are two different issues: 1) perception of skill relative to others; and 2) perception of skill relative to reality.

Tyler9000
Posts: 1701
Joined: Fri Jun 01, 2012 11:45 pm

Re: Empirical Evidence of Mt. Stupid

Post by Tyler9000 »

Ah -- good point. There's hope for the experts after all! Assuming they don't think they're frauds, of course. :)

People are complicated.

ThisDinosaur
Posts: 997
Joined: Fri Jul 17, 2015 9:31 am

Re: Empirical Evidence of Mt. Stupid

Post by ThisDinosaur »

How does one know what side of that curve they fall on?

Confidence is too subjective to rely on. DK and Impostor Syndrome prove its unreliable. It seems to me that most people talking about DK are confidently discussing how stupid *other people* are.

jacob
Site Admin
Posts: 13168
Joined: Fri Jun 28, 2013 8:38 pm
Location: USA, Zone 5b, Koppen Dfa, Elev. 620ft, Walkscore 73
Contact:

Re: Empirical Evidence of Mt. Stupid

Post by jacob »

The "advanced beginner"(*) level for Dreyfus's learning levels is at the peak of Mt Stupid.

(*) This is the point where a person knows a lot of facts but have no real experience putting them into context.

You can also just test the population and plot confidence vs performance, that is, ask what grade people think they're going to get vs the one they actually get. This is what DK originally did. The Mt Stupid shape comes from differencing the confidence with the performance and plotting it as a function of percentile.

In any case, other than using DK/Mt Stupid to talk about how stupid other people are, I also use it to realize when/where I'm on top of Mt Stupid myself and should therefore best not opinionate. (Note how I rarely get into dietary discussions.) Normally, though, Mt Stupid can be used to explain why most internet debates are usually between experts (who know they know) and stupid people (who think they know but don't know enough to know that they don't) with relatively few numbers (who know enough to know that they don't know but not enough to know) in between (not that they aren't there but they aren't talking).

ThisDinosaur
Posts: 997
Joined: Fri Jul 17, 2015 9:31 am

Re: Empirical Evidence of Mt. Stupid

Post by ThisDinosaur »

jacob wrote:
Mon Jul 23, 2018 8:23 am
test the population and plot confidence vs performance, that is, ask what grade people think they're going to get vs the one they actually get.
But, as an individual, what else can one do? I assume the answer is to put your money where your mouth is and test out your beliefs IRL. So, experience trumps strength of argument. But experience can be "wrong" in so far as one can be incorrect and lucky. Or they have the causation all wrong.

Sometimes you'll only have one opportunity to make the right decision, and anyone with experience-based advice has survivorship bias. Example: treating cancer with western, eastern, or holistic medicine. Very articulate people can make convincing arguments for any of those as having the "correct" framework. But they can't all be right.

User avatar
jennypenny
Posts: 6643
Joined: Sun Jul 03, 2011 2:20 pm
Location: Stepford USA

Re: Empirical Evidence of Mt. Stupid

Post by jennypenny »

For anyone interested in something other than COVID, David Dunning was on Ritholtz's podcast this week. It was good. There was a brief discussion about how we view our own work from 5 years previous that I found interesting, Basically, if you cringe at past work, that's good ... if you don't, time to grow a little. Fun fact from the show -- they didn't come up with Mt. Stupid.

Anyway, an enjoyable respite from the pandemic deluge.

User avatar
Seppia
Posts: 1549
Joined: Tue Aug 30, 2016 9:34 am
Location: Hong Kong

Re: Empirical Evidence of Mt. Stupid

Post by Seppia »

Barry is the best, MIB podcast is probably my favorite of all podcasts, I never miss an episode.

theanimal
Posts: 1714
Joined: Fri Jan 25, 2013 10:05 pm
Location: AK
Contact:

Re: Empirical Evidence of Mt. Stupid

Post by theanimal »

Came here to post exactly this! I thought it was interesting how they didn't come up with the chart. Good conversation.

ertyu
Posts: 1680
Joined: Sun Nov 13, 2016 2:31 am

Re: Empirical Evidence of Mt. Stupid

Post by ertyu »

I am listening to it now. I definitely recognize myself as an investor in Dunning's description of the Dunning-Kruger :lol:

theanimal
Posts: 1714
Joined: Fri Jan 25, 2013 10:05 pm
Location: AK
Contact:

Re: Empirical Evidence of Mt. Stupid

Post by theanimal »

The Dunning Kruger Effect is Probably Not Real
The two papers, by Dr. Ed Nuhfer and colleagues, argued that the Dunning-Kruger effect could be replicated by using random data. “We all then believed the [1999 Dunning-Kruger] paper was valid,” Dr. Nuhfer told me via email. “The reasoning and argument just made so much sense. We never set out to disprove it; we were even fans of that paper.” In Dr. Nuhfer’s own papers, which used both computer-generated data and results from actual people undergoing a science literacy test, his team disproved the claim that most people that are unskilled are unaware of it (“a small number are: we saw about 5-6% that fit that in our data”) and instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
I finally reached a point where I was fairly certain the Dunning-Kruger effect had not been shown to be a bias in our thinking but was just an artefact. Here then is the simplest explanation I have for why the effect appears to be real.

For an effect of human psychology to be real, it cannot be rigorously replicated using random noise. If the human brain was predisposed to choose heads when a coin is flipped, you could compare this to random predictions (heads or tails) made by a computer and see the bias. A human would call more heads than the computer would because the computer is making random bets whereas the human is biased toward heads. With the Dunning-Kruger effect, this is not the case. Random data actually mimics the effect really well.
Are there dumb people who do not realize they are dumb? Sure, but that was never what the Dunning-Kruger effect was about. Are there people who are very confident and arrogant in their ignorance? Absolutely, but here too, Dunning and Kruger did not measure confidence or arrogance back in 1999. There are other effects known to psychologists, like the overconfidence bias and the better-than-average bias (where most car drivers believe themselves to be well above average, which makes no mathematical sense), so if the Dunning-Kruger effect is convincingly shown to be nothing but a mirage, it does not mean the human brain is spotless. And if researchers continue to believe in the effect in the face of weighty criticism, this is not a paradoxical example of the Dunning-Kruger effect. In the original classic experiments, students received no feedback when making their self-assessment. It is fair to say researchers are in a different position now.

The words “Dunning-Kruger effect” have been wielded as an incantation by journalists and skeptics alike for years to explain away stupidity and incompetence. It may be time to break that spell.

classical_Liberal
Posts: 2147
Joined: Sun Mar 20, 2016 6:05 am

Re: Empirical Evidence of Mt. Stupid

Post by classical_Liberal »

...

Post Reply