Handling 1 Million Requests per Minute with Go

They’re uploading each POST payload to S3 at a rate of up to 1M uploads a minute? They’re going to go broke from S3 operational fees. PUT fees are $0.005 per 1k, or $5/minute, or $7200/dayS3 is an absolutely terrible financial choice for systems that need to store a vast number of tiny files.

reply

They’re batching the requests into larger files on S3. The 1M refers to the number of HTTP requests hitting their server.

reply

Can you show me where in the post that is described because I do not see it. All I see is a description of how they moved the UploadToS3 aspect to a job queue, but it’s still sending individual files to S3.

reply

Storing millions of tiny files in any filesystem is a terrible choice.

reply

Fair point, I was mostly focused on the absurd cost for that specific implementation. What would you suggest as an alternative? A document-oriented database?

reply

If you’re on AWS I would probably go with DynamoDB, if you’re on GCP Datastore. They aren’t drop-in replacements for one another but the way you architect your system will be similar(ish). The main benefit is that it’ll cost less upfront and require less to manage. Now that AWS have simplified back-ups it’s a pretty simple system to operate. If you’re looking for better controls over latency then I’d probably go with Cassandra.There’s a big caveat to any NoSQL database and that’s how you handle aggregates/roll-ups. With a standard database it’s easy to write these queries. If you do it without thinking on a NoSQL system it’ll cost you in performance and where billed per access, money. There’s a few ways to address this;

– batch ala map-reduce.

– streaming ala Apache Beam, Spark, etc.

– in query counting (aka sharded counters).

reply

An underused option is actually SQLite. That gives you a surprisingly feature-rich system with very low overhead. In fact, you may see benefits: faster access and less disk usage https://www.sqlite.org/fasterthanfs.htmlA key-value store would probably work well, depending on how well its storage layer is architected.

reply