Early Retirement Extreme Forums

As quite a few of you are interested in data and analytics, I'll just leave this here:

Article about the course and the course.

Yeah, these free courses are awesome. At one time, I thought I was going to be a coder (LOL) and found a bunch of free courses from Harvard.

Looks like they have a free data science course too:

https://online-learning.harvard.edu/cou ... e-r-basics

For better or worse, data science, which is really more like a grouping of several other skill sets, is now something everybody should know something about to retain informed citizen of planet status. I completed several online courses and read quite a few books related to the topic, and I would like to someday do a project, although I decided that I did not want to move to some HCOL realm and undertake full-time employment in the field.

Having had the experience of observing children using similar, although much lower level, online learning programs, one very important thing to know before making the attempt is that it is easy to find yourself unexpectedly stuck at some point in your studies. So, you have to have back-up plan to look to other learning resources, such as books or knowledgeable human being when/if that happens. Also, at least reading about the problems other students are having in the discussion groups can be very helpful, because it is pretty common occurrence that, for example, one answer in a problem set will be just plain wrong or the bleeding edge will move even quicker than an online course can be updated/revised.

What are your favourite IDE for python?

Thanks for this thread @FIPS. I enrolled myself.

A lot of my friends have become data scientists. Mostly physics PhDs. There is a cohort of colleagues that don’t get absorbed in academia that have ended up there after struggling in industrial research. It reminds me of my friends at Fermilabs and CERN who went on to be commodities traders. I guess the transition is pretty straightforward. They basically had to bolt on a few new skills and bam, they were off and running.

I’m wondering if this field will be flooded and glutted soon. There are a lot of hungry people with enough mathematical skills to jump the small gap of Python and SQL. Why hasn’t there been a flood of people from Bangalore applying for data science positions and killing the demand? (Kind of like the engineering shortage during the space race and the glut that followed).

Just curious. I guess the market demand is huge right now?

Manipulating and analyzing data is the 20%. The 80% is interpretation and implementation of data products which requires domain specific skills. Such skills are challenging to learn from books because industries are always evolving and experts usually do not bother with teaching. Hence obtaining a data science position involves either..

1. Learning an industry from the inside with some other position while training to apply data science skills.

2. Learning how to extract sparse insight from coarse data-sets while convincing an employer that you can someday extract dense insight from domain-specific data-sets.

@Sclass - Likely the field of data science will be standardized soon. Fallen physicists are often the first species to colonize any new field that involves some kind of mathematical analysis or modelling. It's generally easier to teach physicists field X than it is to teach people from field X how to do analyze and present math stuff. This even goes to some degree for actual mathematicians who tend to lack sufficient appreciation or respect for "reality" to do a good job.

Data science and programming languages are, as you say, pretty much just a matter of bolting something onto the generic physicist. This process only takes a couple of books and a couple of weeks of self-study. It is expected by physicists but not so much by everybody else. Thus employers might ask "What can you do? What are your skills?" and the newly minted physicist answer would be "I don't know, anything you want and if I don't know, I'll figure it out."

Also since the end of the cold war there has been a glut of phd-level physicists, so they have to go and do something. In my cohort, insofar it wasn't academic research or teaching high school, the destination was most frequently selling or supporting software and then onto management or management consulting insofar a modicum of people-skills were detectable. There were a few who went into weapons or finance too but neither are big industries in my country of origin. I think that covers about all.

As for data science itself, I think it's already reaching the point of glue coding similar to software engineering. The tools are there and function as black boxes. Newly minted "data scientists" who took a 2 month boot camp just have to know which tools to use in which situation and turn it into a presentation. At this point physicists who are innately driven towards (re)building everything from scratch, because they came in during the era where such tool libraries weren't widely available, would have to find some other kind of value-add.

so are physicists final form of human being?
<tongue in cheek post, btw>

Haha! More like the universal STEM character. Physicists are close to the middle of the process and therefore knows a little bit of everything/have fewer blind/sticking spots. To wit,

Mathematicians create equations and methods, which are used by
Scientists to create theories that describe reality, which are used by
Engineers to create machines and tools, which are used by
Technologists to create products and solutions at the end-user level

(Physics is often called the king of science, because it's the foundation of all the other sciences. Mathematics is often called the queen because without it, the king would be nothing. Whereas other sciences are more specialized and thus more locked-in or prevented from questioning everything.)

In any case, you can see how mature a field is by which type is dominating it. For example, spreadsheets, word-processors, and some forms of webdesign (e.g. blogging) are already at the technologists level if not beyond. You don't need to hire an engineer to create a spreadsheet or type in your book manuscript. Data science is definitely at the engineering level now; you don't need scientists so much anymore because the theories used are already getting standardized and locked-in. Like, "For type x problems, use type y solutions from the z-library" ... essentially gluing known components/demonstrated theories together into a product.

Mathematicians and physicists get deprecated as the concentration moves further and further towards to end-user level. Most [mathematicians and scientists] have little interest in what's going on with customers and are therefore not very good at it. They just want to solve interesting problems, once.

As an exercise for the student, you can see a similar transition in how FIRE developed over the past 10 years; who pushed it; what was pushed; how it was pushed; etc.

@jacob:

Interesting analysis. I would estimate that I am slightly less than 50% ( )a STEM character, but my STEM character would wobble somewhere between Mathematician and Scientist since I don't even like Applied Math, but my 53% non-STEM brain also believes that math should be considered a sub-topic of Anthropology.

@fiby41:

I do not have favorite IDE. They all seem "Wow, so easy!" to somebody who took a course in Fortran on main frame in 1984. The new-to-me product I liked using was Jupyter notebook, because allows for more creative across multiple barriers communication of concepts.

+1 to everything sclass, daylen, and Jacob said. Is wasn't there for it, but the evolution of data science reminds me of the steps that I believe web design went through in the late 90s and 2000s.

Whoa that is a pretty damning analogy. But you may just have a point there.

The physicists I know actually wrote some extensive howtos on LinkedIn about what a physics PhD needs to learn to become a data scientist. It didn’t seem like much. It was like unemployable + epsilon ~= unemployable. But it wasn’t the case. They got hired. Given the kind of jobs these folks landed I kind of wondered if the job requirements were demanding. They do things like use metro data to predict human activity around the holidays. Another guy tries to figure out what you’d like to eat based on your restaurant reviews and your friends’ reviews. I know from personal experience his algorithm sucks. I guess it is data science not rocket science.

There must be a sliding scale. I saw a program on plastics recycling and this guy training a machine learning robot to separate plastic bottles from trash. This one looked a significantly tougher than looking at subway ticket sales. Self drive is another demanding one.

If it doesn’t take too much to become a data scientist, isn’t there a risk because of the low barrier to entry? My physics PhD friends were basically dropouts...I mean they have degrees but they basically had to leave their fields of expertise. And I don’t think it’s good enough to say their foundation from being physics majors carries them because they kind of sucked at that too. Literally the bottom of the barrel in @jacobs hierarchy of physics graduates.

So I’m watching from the sidelines to see what happens. It reminds me of the flood of programmers who arrived from China and India in the mid 2000s. All those 386 PCs we recycled got set up with pirate compilers over there. I don’t see why some kid in Bombay can’t install a free python ide and kick butt on a physics dropout. In my recent experience math skills of an IIT grad whoop ass on a typical US graduate. If python, sql and ML algos are the differentiator look out.

So where is the value add? Take some free classes get a six figure salary? Install NumPy and make a call? Or worse, what happens when the bean counters install trials of Alteryx software and see how stupid easy it is? Seriously? How long can that last? Or is the prerequisite math that hard to teach mere mortals? Or is this stuff so boring most neural typical humans will be looking for the exits. It smells like a bubble of sorts.

The value add is in the ability to creatively frame problems and present solutions in a manner that is meaningful for decision makers. For instance, one problem I created for a course I took :

A rural region is currently served by 5 small affiliated hospitals. Due to falling population and rising overhead, one of the hospitals must be closed. The regional board in charge of making this decision would like to have some information about possibility of increased death in transit to hospital rate dependent upon their choice. The results of this analysis project to be presented should include animated map of region running model of mortality instances over course of 5 year period.

@Sclass -It probably follows the Gartner hype cycle. Physicists enter shortly after the "technology trigger" because they're the only ones (short of the original inventors) who can learn the techmology on their own fast enough. Schools start offering [named] degrees and entrepreneurs offer bootcamps at the "peak of inflated expectations".

The value-add at the time of the data science technology trigger was that you could hire to physicists or physics-dropouts(*) and they'd understand the basics of the field on their own within a few weeks and up and productively running within a few months. That's allows the company to be first and being first is usually a huge advantage. This kind of innovation is not automagic with people who aren't "trained" in finding a creative solution while starting from scratch for each and every problem they've ever encountered.

(*) In my hierarchy they'd still be stronger than most anyone else at reality-applied math even if they're dropouts. There is of course highly specialized math where other fields would be stronger but such is generally not used in data science. I have not seen anything in a data science that a second-year physics student couldn't understand. There are certainly different levels of data science but none of it is all that hard compared to physics.

The value-add now from taking a few classes to retrain now is less because a lot more people can learn from classes than learn creatively on their own. This radically expands the supply of "data-operators". The train is about to leave the station.

The next technology might be AI. This is actually something where I would say that physicists don't have a natural advantage and where computer scientists and maybe even retrained biologists would probably do better due to the non-linear black-box aspects of various AI methods. Black boxes make physicists very uncomfortable.

What will eventually happen? Same thing that happened with spreadsheets and webdesign. Consider word-clouds. These used to be a rather innovative visualization that certainly requires a combination of statistics and creativity. Now everybody understands them and a business major can probably get a plug-in for Word if they want to insert a word-cloud in their presentation.

AI is already imbedded in Data Science. In fact, conventional statistical analysis is being used less than other methods. For instance, in the example I offered above, ML trained on very large collection of hospital and death in transit node maps might be method towards model. Another example would be AI image identification on millions of related images used to generate Word Map. It’s more like cooking than physics.

sounds exciting. I’m glad there is work for my old classmates. The data science has enough science in it to keep them happy.

Yeah, the only reason I was interested is because it is a pretty good field for a generalist. It requires at least 5 different skill sets, so if you were bored with one you could switch gears and focus on another. BTW, the courses I took on Coursera were "The R Programming Environment" offered by John Hopkins and "What is Data Science?", "Open Source Tools for Data Science", "Data Science Methodology", and "Python for Data Science and AI" all offered by IBM. These courses would all be quite easy for almost anybody on this forum. If you pay the $39.99/month to take the IBM classes, you get access to most of the IBM tools available such as Watson Studio. To give you some idea of how advanced some of the AI image recognition is becoming, I uploaded a photo I had taken out in the wild and the IBM based AI immediately correctly identified the plant as Bear Berry. This would not be easy to do based on just a photo for even a moderately experienced human forager. So, I immediately started wondering whether a solar powered robot could forage/sort bear berries, blueberries, huckleberries, and wintergreen berries more efficiently than a human (or a bear?)in the Northern Woods environment.

A good book for anyone who is interested in doing some exploration of the topic without access to big tools would be "Data Smart:Using Science to Transform Information into Insight" by John W. Foreman. Some other interesting books I sorted into the general category of Data Science in my stacks along with the usual/obvious textbooks would be "Storytelling with Data" -Cole, Nussbaumer, Knaffic, "The Model Thinker" by Page, "Information Theory: A Tutorial Introduction" by Stone, "Algorithms to Live By: The Computer Science of Human Decisions" by Christian and Griffiths, "The Signal and the Noise" by Silver, and "The Theory that Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines and Emerged Triumphant from Two Centuries of Controversy."

Trying to visualize people with PhDs downgrading to programming or trading is like a person drilling a hole in a wall using a rocketlauncher (carrying too much baggage.)

@fiby41 - I think you're somewhat right about that. There's a material risk of [phds] overthinking a problem before starting on it as others "go ahead and just do something". Overthinking is useful at the beginning of the technology trigger but later it can become a liability. What is "good enough for the job" varies as a field matures.