And so we tried to build the Census Platform.
REA runs a plethora of special interest guilds that anybody can join. These guilds get together every now and then to trade smarts, share stories, or just to talk about new, shiny tech.
One of these guilds is the Ops Guild – which over time, has morphed into a challenge/gauntlet where everyone with an interest in Operations can get together and sharpen their Ops skills. We run troubleshooting dojos, breakfix scenarios, and a couple of games – and given recent events, we felt the need to be relevant…
The Game
The Census game was simple. We were to build infrastructure that could handle the traffic of 11 million households from 6PM-9PM on August 26, 2020 – serving out a 100-point questionaire. The infrastructure was to be resilient, secure, and performant.
We had 30 minutes.
Three teams were put to the task of collaborating and figuring out a solution to this conundrum. But they were trying to solve different problems – as each team had a different working parameters:
Team Datacenter had to figure out a solution within the scope of the Australian Signals Directorate Evaluated Products List (see here: EPL – Evaluated Products List: ASD Australian Signals Directorate ). This put a heavy restriction on the technologies they could use.
Team AWS had to work within the bounds of the Certified Clouds list (see here: ASD Certified Cloud Services – Information Security Registered Assessors Program ) . If they were to use AWS, they were bound to what has been certified (which at the time of writing is EC2, VPC, S3, and EBS).
Team No-Restrictions could work with no technical restrictions – but they held the most difficult burden of all. They had to show what they could do with no boundaries. The onus of delivering something…interesting.
The Solutions
Team No-Restrictions Fires the First Shot
Team No-Restrictions decided to use multi-region S3 fronted by Cloudfront to deliver static pages, which then talked to API Gateway to launch magic Lambda functions that save state to DynamoDB. The questions and answers would be saved as JSON. Route53, Cloudfront, and WAF in front of all of it. Oh, and they send an email on completion!
Team Datacenter Spends All The Money
Team Datacenter decided to have two datacenter setups for each state – one for DNS, one for the app. Load balancers, state in each virtual server, saved to SAN as encrypted volume. Anycast for the two outside links, and they would advertise only to Australia to reduce the surface area they need to secure.
Team AWS Makes A Page For Every Form
Team AWS decided to have a farm of instances creating files and forms for each unique 16-character alphanumeric key assigned to each household. The forms would be in S3, which would secure the responses with a public key from the census site – saving to another S3 bucket for collection later.
The Takeaway
All in all, this was a really fun exercise for everyone involved. But at the root of it is the reiteration of the fact that everybody in REA fills different roles. We are good at different things. And the best way to solve Big, Complex problems is to get all of those minds working together.