What IT skills are needed to implement this?

[ertyu]

I recently read this article on macro portfolio construction.

For those who can't be f*ed to click, the idea is:

1. Build a macro model
2. See what the market is pricing in
3. On the basis of the macro model + an analysis of current and historical cross-asset correlation + your own subjective assessment, decide what is mispriced.

The macro model in the example is this:



The cross-correlation table in the example is this:



It is clear that replicating the above involves both macroeconomic knowledge and technical knowledge. I am assuming that one would write code that would pull certain publicly available data from various sources and one would then have an application that analyzes this information and produces the spreadsheet or graphic that is needed.

While I don't walk around with financial formulas in my head, the macro/finance knowledge isn't the limiting factor for me. I am fairly confident that I have an alright-ish baseline understanding of finance and macro on which to build should I decide to go look up the formula for "volativity adjusted cross-asset correlation." I assume any averagely intelligent monkey with an econ degree can accomplish this, and I am an averagely intelligent monkey with an econ degree.

I am, however, completely at sea when it comes to the IT aspect of this. What IT skills does one need to pull this off?

I assume that as a solitary muggle, I would not be able to replicate the full, detailed work of a professional whose goal is, essentially, to build these tools then make them available to muggles for subscription. I assume he has at least one actual quant/programmer working for him as well. So I am aware that what I build will be unlikely to match these tools in their full sophistication. I am also aware that I'm not likely to generate any significant advantage by building them because everyone and their mom can build them with a bit of IT/econ training. But I'm still curious to make inroads into developing a simplified version of some of the above.

What should I look into learning? Also, will some of this be already developed and available on, say, github?

[mathiverse]

It's possible a spreadsheet is all you need. The screenshots in your post and in the linked article even look like a screenshots of a spreadsheet. However, I didn't dig into what's needed or the link you posted, so maybe I missed something.

[jacob]

It is clear that replicating the above involves both macroeconomic knowledge and technical knowledge. I am assuming that one would write code that would pull certain publicly available data from various sources

This is the hard part. Not only finding the data but more importantly cleaning it. Google sheets actually provides a ghetto way of doing this. For example, I pull the gold price from the webpage of a bullion seller using the IMPORTHTML function. However, once you start importing from multiple sources, you got to stay on top if it in case something breaks---usually because a website changes its formatting. This is why clean data is often sold. One of the (ex-)bankers will probably tell you to just get a Bloomberg terminal (which of course costs money).

and one would then have an application that analyzes this information and produces the spreadsheet or graphic that is needed.

This is the easy part. You could do this in a spreadsheet or use R if you want to get really fancy. Those "sophisticated models" aren't more sophisticated than that. For the most part, people just use what they already know. I'd highly recommend splitting up the task in a computational part (numbers in, numbers out) and a presentation part (numbers in, graphs out). This way you can run them separately. At one point, I used a combination of R and LaTeX. R would get me the numbers, and LaTeX would create a nice presentation. When the numbers changed, all I had to do was rerun. Why this choice? Because those were the languages I knew already. I have also programmed postscript directly from fortran90 to create graphic animations and I wouldn't recommend that

[ertyu]

It's possible a spreadsheet is all you need. The screenshots even look like a basic spreadsheet. However, I didn't dig into what's needed or the link you posted, so maybe I missed something.

Well, I'm still assuming that one would need to source data and feed it into whatever data-crunching app (I agree this looks like excel). Or am I making this bigger than it is in my head due to my IT ignorance?

[jacob]

At one point, I used a combination of R and LaTeX. R would get me the numbers, and LaTeX would create a nice presentation.

After googling around a bit I'm 80% sure I used https://en.wikipedia.org/wiki/Sweave to do this.

Keep in mind that since sweave doesn't split between the data crunching and the graphics, you have to rerun the entire @#$@#$ thing every time you change a comma in the report. With macro data it's quite possible this lament will be a non-issue. I was analyzing ticker data so it was a very large (many MBs) data set.

[Slevin]

Most data scientists use python these days, so it’s probably good enough for this too.

Basically step 1: find the data sources, then write an application to scrape data sources. This basically just means writing functions / methods /classes to read the internet data to a file, then “sort and clean” the data to get it into the proper formats, then put it into some “nice for you to handle” data structure (just a multidimensional array probably, in python a dictionary would probably work, maybe use pandas data frames if you need a complicated data structure).

Step 2: data processing. Take your numbers and do whatever the calculations are for getting the 1m delta.

Step 3: output the data all pretty. I’m not a pretty graphics guy, can’t make too much of a recommendation here. If you use pandas and keep everything in data frames, you can use the to_excel() function to write the data to a spreadsheet and there’s probably some nominal formatting you can feed in to make it fairly pretty. Pandas also does graphs, and other types of formatting, so basically you just need pandas for this as far as I can tell, but ymmv.

Step 2 and step 3 are very easy but will just require some messing with. Step 1 is the actual hard part. Like Jacob said, getting clean data is hard. And sourcing data is hard. But in the end this is basically a scraping and processing script, that outputs the end data to some sort of file for the end consumer.

To finish it off as a whole product you would want to write a wrapper script that basically runs this script as an automated job at certain times (cron), then sends the data to you or all the customers at certain times. But you probably just send it to yourself so you can view the data and make sure everything processed correctly before sending it out. And you can add testing to make sure this whole debugging process goes more smoothly when something breaks.

[ertyu]

Thank you, Slevin and Jacob. Slevn, your point by point write-up is especially helpful. There appear to be multiple data scraping tutorials on youtube, so I'll start there and see where that takes me. I will start with this with the goal to understand the process involved rather than to be the most efficient or up to date or industry ready (which is not the goal here, understanding is).

[sky]

If you have not used python before, a few key things you have to understand to get started:

There are different versions of python and you need to select the correct version based on the applications you want to use. It comes down to python and python3. In most cases you will use python3. You need to know this because many old tutorials are not in python3 and you may experience errors.

You need to know how to find out which version of python you are using. You need to know how to set up your python editor to use a specific version.

Python is a base language and there are many extensions, which generally are libraries of functions that do useful things. You can install these packages (libraries, extensions) using pip, which is package installer for python.

Because different applications use different versions of python and different packages, python developers tend to put a python application in an environment. An environment is a folder with an instance of python of a specific version, and where you can install any number of packages. You write your script in the environment folder. You set your python editor up to run the instance of python in the environment folder. This way each application you write in an environment folder has its own python version and packages, and you don't inadvertently install or upgrade something in the future that causes errors.

One of the packages most used for web scraping is "beautiful soup". If you do a search on that, you will get good web scraping tutorials. I would not be surprised if you do a search using terms related to your programming goal, that you find that someone has already built such a program and it is available as open source on github or one of the other open source repositories.

The best source of up to date python tutorials is Youtube. I find website results from a web search on the www to often be outdated. Do your searching on youtube, and use www results as a backup.

I have not yet found a good book which works as a manual on how to use python.

This is based on having spent a lot of time and trial and error trying to learn to use python. These items probably took a year to discover on my own, so I hope I am saving you some time.

[fingeek]

Take a look at something like pipedream.com first - it's No Code/Little Code. It allows you to connect APIs (which is where you would get source data from), do some chewing on the data and then send it to some data sink (for example, Google sheets). That + sheets/excel formula might be just enough for what you need.

If you do need something deeper, +1 for python(3) and something like Beautiful Soup yes or Scrapy, which would pull the data. And then something like pandas to do data analysis.

In opposition to @jacob, I'd suggest that "actually gathering the data" bit is the easier part of the puzzle (although I agree with his point). The harder piece is to work out the model and analysis algorithms so that data in=>useful information out (sentiment, underpriced/overpriced etc).