Cloud Solution for my trading tool?

Ask your investment, budget, and other money related questions here
Post Reply
FreeLiving
Posts: 2
Joined: Thu Sep 24, 2020 7:41 am

Cloud Solution for my trading tool?

Post by FreeLiving »

Okay, so I won’t go into this all too deep, but the thing is this: For my early retirement I’ve put together a tool to help me with trading decisions.
This tool analyses a lot of data from the internet and forms viable trading advice for me. My Data is mostly organized through Grafana and it’s been working great so far.
The only issues I keep running into occur, when there is too much data for my server to process all at once. So I’ve been thinking about switching to a cloud based provider like Aiven in order to have higher capacities in times of need.
My question is, if somebody here has some experience with that?

Thank you in advance!

User avatar
fiby41
Posts: 1341
Joined: Tue Jan 13, 2015 8:09 am
Location: India
Contact:

Re: Cloud Solution for my trading tool?

Post by fiby41 »

I'm golang illiterate but couldn't you set up parallel processing?

Optimal_Solution
Posts: 51
Joined: Tue May 07, 2013 4:56 pm

Re: Cloud Solution for my trading tool?

Post by Optimal_Solution »

I have professional AWS and Azure experience.

It may not be cost effective if you are running your severs 24/7. But if you can organize your computations into batch jobs that run periodically, then it doesn't cost nearly as much. You should look into Spot instances to save even more money. Spot instances are excess servers capacity that AWS auctions off. You can save a lot of money that way (think 50% or more off). You set a max price that you are willing to pay and you get the server as long as the market price is less than your max price.

The trick is having your data available in the cloud for the batch jobs to consume. Commonly the data will be stored in blob storage like S3 or Storage Accounts and accessed by the servers as needed.

Edit: I don't know anything about Aiven or Grafana. Are you looking for a Grafana cluster that can increase in capacity while you are ingesting batches of data?

xmj
Posts: 4
Joined: Tue Apr 14, 2020 6:26 am

Re: Cloud Solution for my trading tool?

Post by xmj »

On the note of AWS -- if you can write whatever processing jobs in such a way that it can run in Docker, you can get by with using AWS Batch.
Couple this with a data lake on S3 to be queried with Athena displayed via QuickSight, and you have pretty much state of the art Enterprise stack which satisfies pay-as-you-go.

just
Posts: 32
Joined: Fri Feb 16, 2018 12:16 pm

Re: Cloud Solution for my trading tool?

Post by just »

@FreeLiving
I'm curious to what this tool is doing? Are you willing to hint about what the overall idea is?

I'm building a screening tool myself. It scrapes some Morningstar data and then does a ranking of the stocks based on some financial and valuation ratios - inspired by Joel Greenblatt's "magic formula". It ended up being the best way for me to get consistent data for both US and European stocks without having to pay for it.

Post Reply