Okay, so I won’t go into this all too deep, but the thing is this: For my early retirement I’ve put together a tool to help me with trading decisions.
This tool analyses a lot of data from the internet and forms viable trading advice for me. My Data is mostly organized through Grafana and it’s been working great so far.
The only issues I keep running into occur, when there is too much data for my server to process all at once. So I’ve been thinking about switching to a cloud based provider like Aiven in order to have higher capacities in times of need.
My question is, if somebody here has some experience with that?
Thank you in advance!
Cloud Solution for my trading tool?
Re: Cloud Solution for my trading tool?
I'm golang illiterate but couldn't you set up parallel processing?
-
Optimal_Solution
- Posts: 57
- Joined: Tue May 07, 2013 4:56 pm
Re: Cloud Solution for my trading tool?
I have professional AWS and Azure experience.
It may not be cost effective if you are running your severs 24/7. But if you can organize your computations into batch jobs that run periodically, then it doesn't cost nearly as much. You should look into Spot instances to save even more money. Spot instances are excess servers capacity that AWS auctions off. You can save a lot of money that way (think 50% or more off). You set a max price that you are willing to pay and you get the server as long as the market price is less than your max price.
The trick is having your data available in the cloud for the batch jobs to consume. Commonly the data will be stored in blob storage like S3 or Storage Accounts and accessed by the servers as needed.
Edit: I don't know anything about Aiven or Grafana. Are you looking for a Grafana cluster that can increase in capacity while you are ingesting batches of data?
It may not be cost effective if you are running your severs 24/7. But if you can organize your computations into batch jobs that run periodically, then it doesn't cost nearly as much. You should look into Spot instances to save even more money. Spot instances are excess servers capacity that AWS auctions off. You can save a lot of money that way (think 50% or more off). You set a max price that you are willing to pay and you get the server as long as the market price is less than your max price.
The trick is having your data available in the cloud for the batch jobs to consume. Commonly the data will be stored in blob storage like S3 or Storage Accounts and accessed by the servers as needed.
Edit: I don't know anything about Aiven or Grafana. Are you looking for a Grafana cluster that can increase in capacity while you are ingesting batches of data?
Re: Cloud Solution for my trading tool?
On the note of AWS -- if you can write whatever processing jobs in such a way that it can run in Docker, you can get by with using AWS Batch.
Couple this with a data lake on S3 to be queried with Athena displayed via QuickSight, and you have pretty much state of the art Enterprise stack which satisfies pay-as-you-go.
Couple this with a data lake on S3 to be queried with Athena displayed via QuickSight, and you have pretty much state of the art Enterprise stack which satisfies pay-as-you-go.
Re: Cloud Solution for my trading tool?
@FreeLiving
I'm curious to what this tool is doing? Are you willing to hint about what the overall idea is?
I'm building a screening tool myself. It scrapes some Morningstar data and then does a ranking of the stocks based on some financial and valuation ratios - inspired by Joel Greenblatt's "magic formula". It ended up being the best way for me to get consistent data for both US and European stocks without having to pay for it.
I'm curious to what this tool is doing? Are you willing to hint about what the overall idea is?
I'm building a screening tool myself. It scrapes some Morningstar data and then does a ranking of the stocks based on some financial and valuation ratios - inspired by Joel Greenblatt's "magic formula". It ended up being the best way for me to get consistent data for both US and European stocks without having to pay for it.