I personally am an exception because I worked abroad and accumulated savings, and I'm always "one foot out," ready to leave again at short notice. So, I in particular don't worry about them because I know little in my life depends on their opinion. But whenever I return, every 4-5 years or so, people seem awfully eager to compare themselves to me and very pleased to find me lacking. I guess it has to do with how I left and they didn't, and they look at me and see this fat, balding, middle-aged office dweeb who, in their eyes, isn't nearly that cool and needs to be brought down a notch. This is a very different dynamic from that in, say, NYC or Berlin, where people wouldn't be thinking about how to bring you down, they'll be wondering about how to lift themselves up: e.g., in NYC, through furthering their career, and in Berlin by making cool connections in the local hipster art scene.
Your outcomes in a small place depend on the opinions of others. This is not who I am as a person, but I remember both male and female acquaintances of mine passing over dating prospects that aren't "cool enough" because they don't want this to reflect on the perception of their own coolness and impact their future dating prospects. Guys are notorious for this everywhere - what would my mates think if she isn't hot - but I was surprised to discover my female acquaintances be concerned with this, too. You read this and you think, god what immature bs, don't most people grow out of this in high school - but in a small middle-of-nowhere town, no they don't. Here is an example from 2 weeks ago. A classmate of mine, while hanging out with other classmates, resorts to schoolyard bullying: "Come on, ertyu, go get the beers, might serve you well to get some exercise hahaha." Why? Dude enjoys it. Later on, we move on to a different bar and I meet another friend of his I don't know. My classmate talks me up - how many countries I've been to, etcetera. Why? He needs to justify associating with such an apparent loser, or else, his acquaintance might see him as uncool by association. I spent this entire night mildly bemused at the level of effort people invested in games like these - how much effort and manoeuvring goes into impressing the other 50-60 village bumpkins they're stuck with. I then decided I'll not spend much time with this classmate because games like these are not where I want to invest my energy in life - but I can only do this because I have the option to leave the town. Most who stay here know that they don't have the option to say, "i will spend less time with this person" because this person will always be at one of the 2 or 3 places everyone gathers to hang out.
I guess the tl;dr: here is the smaller the place you live in, the more your social image contributes to your outcomes. What they will do that's so awful is keep bullying you whenever you decide to go socialize. And while you can say we're adults and shouldn't pay attention to this shit, constantly being in a hostile environment takes its toll. There's no winning this game - "I told him he's a loser so he cleaned up his act (by implication, to impress me and gain my good opinion) is a favorite, of my father as well as of the local town bullies. You're not gonna hear, "oh, that guy really turned his life and health around, good on him, glad to see it" because people are small and so is their world. The only way to "win" the game is abandon it in its entirety and leave, which is what I have chosen. i find myself back in my home town for reasons of covid + stash is too small to afford living elsewhere if i stop working now. I will take 6 more months or so, and then I'll see what makes me feel more exhausted - the thought of working again, or the thought of my family, my classmates and their bullshit.