Protecting your AWS keys with Credulous

Every week, AWS credentials leak into the wild and are used to mine bitcoins or worse.

In April 2014, DrawQuest closed down after a security breach in which their Amazon Web Services credentials were used to create hundreds of EC2 instances, probably for mining bitcoins. DrawQuest decided they could no longer trust that their core data wasn’t compromised, and closed their doors.

Wouldn’t it be great if there was a tool that could help prevent this sort of thing happening? Well, now there is — enter Credulous.

Problems with AWS credentials

There are two problems with IAM user accounts in AWS that we want to address:

• credentials are persistent (in other words, they don’t expire automatically)
• most tools suggest that you keep your credentials in a file on your local hard disk, so that you don’t have to type them in every time. Of course, these files have to be unencrypted so that the tools can read them.

Many very clever people have come up with some fiendishly clever workarounds, such as implementing devilishly complex IAM policies to restrict access in fine-grained and rigorously-defined ways, but getting that right and keeping it that way without severely impacting your developers workflow is nearly impossible. Wrappers that make use of GPG are great, but GPG can be a bit awkward, and those wrappers still leave you with the problem of redistributing credentials when (if) you change them.

Credulous addresses the issues by:

• storing your AWS credentials encrypted on your hard disk, and providing you with a simple and virtually transparent mechanism for retrieving them and making them available to most things which talk to AWS; and
• providing a mechanism for credential rotation combined with secure redistribution so that you can have the new credentials almost immediately on all the systems where you need them.

These problems prompted a few of us to create Credulous. Credulous started as a Hack Day project, which created the first prototype. At our most recent Hack Day, we refined and rewrote it in Go, and we’ve decided to open source the project early because it’s useful even now. It’s available (under the MIT license) and it’s on GitHub now. It’s by no means complete, but we at REA have been using the prototype internally for several months now, and we know that the concepts are sound and the mechanisms work.

How does it work?

First, we use RSA public-key encryption to store your AWS secrets when they’re at rest; this protects you from accidental disclosure in the event that, say, your laptop goes walkabout or gets left on the train after a couple of post-work celebratory ales. We use an RSA key which almost every developer everywhere in the world has — your ssh-rsa public key (if you use DSA keys and refuse to use RSA, credulous is not for you).

Next, we provide a nearly transparent mechanism to retrieve those keys and place them into the runtime environment, interrupted only to prompt you for the passphrase for your SSH private key (you do have a passphrase on your private key, don’t you?)

Third, we make use of github to distribute credentials; the nice thing about this mechanism is that, for a given user, the SSH public keys are instantly available (that’s the whole point of public keys) so we can encrypt the credentials with all the public keys registered for a user, safe in the knowledge that only they can decrypt the credentials, no matter which system they happen to be on at the time.

All this makes the fourth killer feature possible: you can run an automated, external credential rotation engine “somewhere”; it can rotate your AWS credentials on a frequency that you choose, pushing the changes to GitHub, and allowing Credulous to retrieve the latest credentials and integrate them into your runtime environment automatically and seamlessly.

A couple of simple examples

Saving the current AWS credentials into credulous:

localhost$ env | grep AWS
AWS_SECRET_ACCESS_KEY=NfdgUwo86gwPiD4yc0rvb9AcbA8/iUG34ErSkTff8
AWS_ACCESS_KEY_ID=AKIAJIVYYOC3QOND2YJA
localhost$ credulous save
Saving credentials for boffin@examplecorp

Retrieve credentials for IAM user boffin in the AWS account with alias examplecorp:

localhost$ credulous source -u boffin -a examplecorp
Enter passphrase for /home/boffin/.ssh/id_rsa: XXXXXXXXX
export AWS_ACCESS_KEY_ID=AKIAJIVYYOC3QOND2YJA
export AWS_SECRET_ACCESS_KEY=NfdgUwo86gwPiD4yc0rvb9AcbA8/iUG34ErSkTff8

Retrieve credentials and place them into the runtime environment is as simple as

localhost$ eval $( credulous source -u boffin -a examplecorp )

though we thought even that was a bit clunky, so we wrote a tiny shell wrapper for it to make it as easy as

localhost$ cred -u boffin -a examplecorp

That’s all there is to it.

You can help

There’s a lot of work left to do on Credulous — the remote repository work, and the credential rotation are two of the main pieces still not implemented — but we have been using an internal proof-of-concept for a few months now, and we know that it works, so getting the pieces in place is just a matter of time. Perhaps you’d like to help; visit https://github.com/realestate-com-au/credulous if you’d like to know more.