Is Software Development/ Programming Oversaturated?

[zbigi]

“this dovetails in sinister fashion with the basic idea that any sufficiently advanced technology cannot be distinguished from magic. highly evolved capitalism becomes such a technology and the largess and plenty it produces gets mistaken for a property of the universe rather than a made thing, a thing that must be created rather than simply reaped.”

https://doomberg.substack.com/p/where-stuff-comes-from

I'm not convinced that production of software and of other digital creations is heavily constrained by physical reality. I can envision an earth where billions of people make games, apps and other software and/or digital art while creating very little waste and consuming little of earth's resources.

[Qazwer]

https://noahpinion.substack.com/p/the-m ... e-economic

Recent article on infinite growth to follow up the last point

[white belt]

I'm not convinced that production of software and of other digital creations is heavily constrained by physical reality. I can envision an earth where billions of people make games, apps and other software and/or digital art while creating very little waste and consuming little of earth's resources.

The networks, hardware, and infrastructure that the software require to function are absolutely constrained by physical reality. Here's an old article using data from a decade ago:

If we were to try to power the (2012) internet with pedal-powered generators, each producing 70 watt of electric power, we would need 8.2 billion people pedalling in three shifts of eight hours for 365 days per year. (Electricity consumption of end-use devices is included in these numbers, so the pedallers can use their smartphones or laptops while on the job). Solar or wind power are not much of a solution, either: 1,815 TWh equals three times the electricity supplied by all wind and solar energy plants in 2012, worldwide."

Source: https://www.lowtechmagazine.com/2015/10 ... nergy.html


I'm assuming the absolute energy consumption of the internet has only increased since then, although I haven't dug into the data for recent years.

[zbigi]

The networks, hardware, and infrastructure that the software require to function are absolutely constrained by physical reality. Here's an old article using data from a decade ago:

Source: https://www.lowtechmagazine.com/2015/10 ... nergy.html

I'm assuming the absolute energy consumption of the internet has only increased since then, although I haven't dug into the data for recent years.

I agree that there are some constraints in the digital realm, but they are much weaker than in physical realm. If those billions of people from my example were to produce cars and clothes and cruise ships instead of making video games, the impact on the planet would be probably hundreds of times more severe.
At the margin, a single person creating a video game uses maybe 100 W of electricity for their computer for 8 hours a day, and similarly for consumer of this video game (100 W for (hopefully) less than 8 hours per day). Compare to an average car maker and average car owner - on the car owner side, just 1 gallon of fuel contains 33.7 kWh of energy, which could power a digital entertainment system for hundreds of hours.

[JCD]

During the 11 years before Ross Perot became known for his anti-globalization stance, manufacturing demand in the US grew by 40%.  Output grew only by 34%.  The difference is the outsourced labor, costing 6% of work.  Employment in manufacturing had fallen by 9% during that time.  Therefore 85% of all job loss in manufacturing during that time period was technology. [Source: https://www.youtube.com/watch?v=O7jyh0ClcSA&t=1179s ]  I think the question of "is it worth learning to program with all these other folks piling in" is not as useful a question.  The question I see as useful is what work is likely to be destroyed or created by use of technology?  

Just to give an example, construction work seems currently immune.  I've not seen robotic roofers or plumbers, in part because each house is different and tech doesn't handle that level of difference well or easily.  Some other examples include work that is mostly socialization and work that has cheap employees, as automation is not free.

Back to the topic, software has been immune to tech job loss, but does show signs of vulnerability.  There seems to be two ways this might go, either these mass schools are training to level up a few people or it is all about socialization work and low wage work.  Those who learn in code camps to write boilerplate code are likely to be attacked on that front.  If it is too boilerplate, we'll want to automate it and fire them, just like the manufacturing workers.  If that boilerplate work is to train them to become better employees, we need to have lots of harder work to justify that (otherwise existing CS programs would be good enough).  Thus if it really is about doing technical work, only a few of these "6 weeks of tech school" people will get to be good enough to get to the point that they can do the work that is actually needed.  Those that fail to adapt will be let go.  I'm not sure such schooling was useful for them.  If on the other hand, you don't believe we will have enough harder tech work to do, then software work will move to socialization work.  If it's really about translating from customer requirements to boiler plate code, it's just socialization work and thus it will become low pay, regardless of the number of people these schools pump out.  Just how automated the work can become is something less than clear today, but we know that idea has been the holy grail of tech since the 50s but has not been achieved... yet.

[GandK]

Those who learn in code camps to write boilerplate code are likely to be attacked on that front.  If it is too boilerplate, we'll want to automate it and fire them, just like the manufacturing workers.  If that boilerplate work is to train them to become better employees, we need to have lots of harder work to justify that (otherwise existing CS programs would be good enough).  Thus if it really is about doing technical work, only a few of these "6 weeks of tech school" people will get to be good enough to get to the point that they can do the work that is actually needed.  Those that fail to adapt will be let go.

This is right. In the 20 years I worked in software development (1993-2013) I saw PC development begin as one where IDEs as you think of them today did not exist, where you had to type all code yourself, in a language that often did not remotely resemble human communication. Fast forward to today and a lot of the simple stuff is drag and drop, and the rest of the commands you need are basically in English, though you still need to know the magic words, and great wizards are highly prized.

As I see it, the developers who will lose this battle are the ones who are in some way not flexible. Most education comes on the job, but the ability (and desire) to adapt every 2 years to a new language, a new framework, etc. isn't something you learn. And it isn't something machines can be taught either... yet.

[Blackjack]

This is right. In the 20 years I worked in software development (1993-2013) I saw PC development begin as one where IDEs as you think of them today did not exist, where you had to type all code yourself, in a language that often did not remotely resemble human communication. Fast forward to today and a lot of the simple stuff is drag and drop, and the rest of the commands you need are basically in English, though you still need to know the magic words, and great wizards are highly prized.

As I see it, the developers who will lose this battle are the ones who are in some way not flexible. Most education comes on the job, but the ability (and desire) to adapt every 2 years to a new language, a new framework, etc. isn't something you learn. And it isn't something machines can be taught either... yet.

Only 7 years in but generally agreed. I also think software languages (and thus employment) generally follow Jevons paradox, wherein the easier it is to use a language, the more induced demand there will be to software-ize everything, thus we will have more people writing software (and being drastically more efficient in the new languages), and thus the more software to be maintained (which requires people to do it too).

The old guys / people who learn low level languages will then complain about how easy it is to write software these days / loss of control of things that usually don’t matter ( and can usually be configured if you need to) and the new software engineers use these higher level languages to be massively efficient, puzzled about why the other people are complaining about efficiency losses that are (usually) small compared to development cost, software bloat, etc. Then we see more induced demand (I.e the automation / devops world who maximize the efficiency or try to minimize the friction costs at least of writing and testing and releasing software), and I think that’s the edge of the induced demand cycle we are currently at?

Maybe add data scientists as the new induced demand users who everyone above them doesn’t like because they can (often) barely code (compared to the other tiers?).

As far as I can tell the new job types do not have lower salaries, salaries jumping up 20-30% YoY, and the 36 people messaging me on LinkedIn every week signals that demand is steady or increasing at the current moment, likely due to Jevons paradox.

Side note here that everyone “wants to hire” more people but the friction costs of changing jobs is enormous because you often have to play a silly algo game that has nothing to do with 99% of these jobs, and it’s often 15+ hours of interviews of “culture fit” and other B.S. to get hired at a new company. so nobody keeps those skills up unless they are trying to deliberately game the system towards one of the high tier cushy jobs (300k salary +). The funny side effect of that I have seen is that many of the people gaining those jobs are not necessarily the hard workers or good software engineers, just the ones who game the irrelevant metric.

[Scott 2]

In the 20 years I worked in software development (1993-2013)

Have you kept up on progress in the field, since then?

Since leaving work in February, I had been paying attention - watching new developments, picking up a trick here or there. Last month, I turned off my LinkedIn. It seemed to flip a switch. What I thought was genuine interest, was subconscious competition with my social media feed. Insidious.

More than ever, I don't think over saturation is anywhere close. Remaining in the field is an enormous pain. It's hard to overstate how different the pace of life can be.

[zbigi]

If on the other hand, you don't believe we will have enough harder tech work to do, then software work will move to socialization work.  If it's really about translating from customer requirements to boiler plate code (...)

There's a small chance of this happening. In reality, the industry is taking an exactly opposite direction - the job gets more and more complicated by the year. 20 years ago, you could get a web job by knowing PHP, HTML and a little bit of Javascript and some SQL. Now, you need advanced Javascript, Kubernetes, CI/CD pipelines, huge frameworks like React, different DB paradigms (SQL/NoSQL), some backend language and web framework, you need to be aware about security, sophisticated version control approaches etc. etc. There's a lot of stuff to know, and not many people have had the time to learn all or most of it - hence the ballooning salaries of senior developers who have this knowledge.

Also, there's basically no major movement towards simplification or "boiler plate code" in the industry as far as I can tell. I mean, ideas like that have been floated since at least 70ties, but the actual implementations have always very limited (usually - you could maybe make a simple database-backed report for internal company usage or MAYBE some crude interactive, database backed forms with a very simple workflow behind them - but that's pretty much as far it goes, at least based on what I saw).

EDIT: forgot message brokers (Kafka etc.), monitoring systems, automated testing and cloud on my list of thingies to know to build a bloody web app in 2021. Also, you should be familiar with and have opinions on various programming paradigms and philosophies (monolith vs microservies, SOLID vs YAGNI etc.), because you'll be asked about them during interview. As a bonus, if your opinions don't match with the hiring team's, you'll be passed as poor fit.

[ducknald_don]

I agree. I started in the industry working on 8-bit machines where you could hold the whole system in your head. We didn't have to think about concurrency, security was a non-issue because this stuff wasn't online, it was unusual to rely on 3rd party code, teams where smaller, the whole industry was completely different. I've been working in the same domain for nearly forty years now and the one consistency has been the increasing complexity of the systems we build. I'd hate to start from scratch because there are so many layers you build on. I don't know how anybody can think that 3 months in a bootcamp is going to do anything other than scratch the surface.

[JCD]

There's a small chance of this happening. In reality, the industry is taking an exactly opposite direction...

I have seen efforts to convert what was hard coded software routines into "softcoded" xml style documents that decide workflows in the insurance industry. The tool that did it was often too clunky to hand off to those who came up with the requirements. That is the sort of example I was thinking of. The logic was all business owner, the code writing was all UI driven. I too doubt it will become the majority way development is done.

However, there are two philosophies I see in software throughout history. One is where the developers are small g-gods: they write the tests, they manage the deploys, they manage the infrastructure, etc. This comes from the agile sort of view. Thus view is absolutely dominate today. Then there are the waterfall style, separate work divided up in to the smallest chunks possible. In this world you have lots of dedicated roles, and like manufacturing you try to automate each element possible to eliminate roles. I agree there is no reason at present to believe that the tilt towards the agile is over (I stopped worked in software in 2019), but I know these things come in cycles and I expect a turning will eventually happen. When, how or why are the part I can't say. AI? Better autogenerated code? Competition from China? Cheaper labor? Monolithic monopolies? Union creating? Union busting? I don't know.

Have you kept up on progress in the field, since then?
Since leaving work in February, I had been paying attention - watching new developments...

Funny, I enjoy software development even now. I just didn't enjoy the politics of it, so when I write code it is just for problems I currently have. I've even started a book on a topic around software development--got about a hundred pages in right now.

[GandK]

Have you kept up on progress in the field, since then?

I have not kept up with specific languages. But I continue to read voraciously about how code is being used.

What bothered me for a long time was the reflex to answer "What do you do?" with "I was a software developer." That isn't what I do now, and I knew my answer was about status anxiety. It took me years to stop.

[Scott 2]

The status thing took me by surprise. I have always been one to downplay success at work, avoiding related conversation whenever possible. Once it was gone, finding myself feeling "less than" was very off putting. It must have been motivating my drive for continuing education. I am sure keeping my LinkedIn active dragged that feeling out.

My stock answer so far has been "I am taking a break from work". Close relationships know the truth. So far, my new path hasn't offered much in the way of identity, so the others will probably figure it out eventually.

[guitarplayer]

I have been reading threads related to software/coding jobs on the forum. I have a feeling that there is a bulk of folk saying something along the lines of ‘people who are good at it have been doing it since they were teenagers’ etc. basically it is hard to enter the field without much prior experience.

But then people also say that it is an ever changing field. Taking Margaret’s Mead’s theory of culture, software jobs would then be within a configurative or prefigurative culture. That is, the change is so rapid that premium is put on the ability to sort out before unseen problems amongst participants (configurative) and perhaps even then having to teach solutions to ‘older’ workers already set in their ways (prefigurative). Looking at it like that, software and coding industry should be a manageable to enter by anyone who has good skills of acquiring skills, as opposed to some other fields where premium is on ‘respecting the way we have been doing it’.

I now do care and it is a postfigurative culture field. Clients are set in their ways in a society set in its ways. the task is to have clients ‘being able to enjoy what we all get to enjoy’, like going to the movies to watch something, have coke and popcorn, go for a tea break to a cafe etc.

[zbigi]

Looking at it like that, software and coding industry should be a manageable to enter by anyone who has good skills of acquiring skills, as opposed to some other fields where premium is on ‘respecting the way we have been doing it’.

That is the case. I've recently read about someone making $300k a year at a FAANG just 3 years after finishing a bootcamp. If you can learn quickly, are disciplined/dedicated and have high tolerance for frustration, then this career is really the golden ticket, esp. in the US.

[alex123711]

That is the case. I've recently read about someone making $300k a year at a FAANG just 3 years after finishing a bootcamp. If you can learn quickly, are disciplined/dedicated and have high tolerance for frustration, then this career is really the golden ticket, esp. in the US.

It also requires constant learning retraining though to keep up. I was reading about bootcamp stats and it was mentioned the number of bootcamp grads still in related employment was an extremely low figure. Also the bootcamps have certain ways to inflate their employment statistics (By only including a small subset of grads in the statistics due to very strict measures/ benchmarks of what defines 'looking for employment' and 'in related employment'.

[zbigi]

It also requires constant learning retraining though to keep up.

Yes, I included that in the "disciplined/dedicated" trait. It's certainly not a job for the lazy or people who are looking for an easy ride (this perhaps explains the low long-term retention ratio of bootcampers). It can eventually get so if you stay at a single project for a long time and eventually know all there is about it - but unfortunately the way to get raises in this industry is to change companies often, so staying in a particular cushy job means leaving a lot of money on the table.

[guitarplayer]

I just got enrolled into a funded 1/3 of a bootcamp, basically Python section of the bootcamp. I am getting serious gearing up for professionally working with code.

I am pretty competent with LaTeX, work it in on a regular basis. @zbigi, would this be a good 'open source project' to get engaged in? If not, what else?

I have a neat LaTeX CV, a(n empty so far) github account.

My stats teacher suggested me looking at this website for time series, it looks at it via python lens. The plan is to work through it.

And yes, I do BSc Maths and Stats, past 50% of the curriculum now. I have some rudimentary dusted knowledge of R and MatLab. Now I work with SPSS and Minitab and Maxima (the last two look like something used more in the school setting than industry but maybe I am wrong).

Sorry for derailing the thread somewhat, I might set up a new one or continue in my journal and via private messages.

[Blackjack]

@guitarplayer , I’m not certain that anyone will care about latex, or anything like that. Open source projects are cool to contribute to, but not the key to landing jobs. The key to getting the first job (in terms of the interview) is always gonna be algorithm knowledge and ability to actually solve problems. Unfortunately, time is best spent on hackerrank, codewars, etc. Do codewars first for a long time (few hundred problems, till you can do the hard ones) to level up your skills in semantics, then spend time in hackerrank to learn how to solve algorithm problems. “Grokking algorithms” is a classic if you are learning algos in python, and im sure other more thorough books can be recommended. With codewars, understand clear code is the best, and the code golf b.s. being uploaded is just a silly nonsensical contest (in terms of training for actual jobs).

When I hire fresh level 1s (straight out of college, or for internships), I’m barely looking at the resumes. They all basically look the same, and if I look through a resume and see a decently filled out GitHub account with personal silly projects of not horrible code (like you will do while learning grokking algorithms) or a decent account on hackerrank / something similar, then they make it to the interview stage. Then I’m mostly filtering through 15 minute chats with candidates to try and understand if they have the basic skill knowledge to actually do the work, then the learning ability to get up to speed and be able to contribute, and then the ability to work with others and have a good / fun disposition.

[zbigi]

The bootcamp you've linked is geared toward teaching you back-end software engineering, in Python. So, if you want to go into that, then none of the LaTeX/R/MatLab/SPSS etc skill wills be relevant. These are tools used in the scientific world, including the "data science". It's a very different path than software engineering. Essentially, at some point you'll have to decide which route you want to follow - they both have their pros and cons.