In REA, Amazon Web Services (AWS) is our major development and production environment, and CloudFormation (CF) is one of the best tools we’ve found to manage deployments in AWS. At the time of writing, JSON is still the only template format supported by CloudFormation; but if you search for “Programming in JSON” in your favorite search engine, the results may be very disappointing. Some developers find writing JSON templates hard and have trouble with the data format, especially when the templates are big (you can’t have comments, syntactic strictness, etc).
At REA, we encourage people to explore and find new technologies to solve problems, improve product quality and speed up deployment cycles; this freedom to explore has given us a few choices for addressing this problem.
One possible answer is to use a DSL (domain-specific language) for CloudFormation template development. Recently we discovered cfndsl, a Ruby DSL for template development. Though it gives us the opportunity to develop templates in Ruby, this brings another problem: in our DevOps culture, both operations and developers write templates. Some love to use Ruby, and others love to use JSON. We want to make both choices available and feasible when different approaches want to collaborate on the same project.
It’s fragile to keep both JSON and Ruby templates during development: there’s a non-trivial overhead to update both versions for every single change and that brings a significant risk, as versions drift apart from each other. We also want to minimize the overhead of learning before people can maintain and support the project, and (perhaps most importantly) for people who need to fix issues in production when things go south at 3am.
We can already use the Ruby DSL to generate JSON — but what about the other direction?
One approach we’re using today is to treat the JSON templates as the source of truth and keep only them in source control. Developers can use our new tool cfn2dsl to transform the JSON templates into temporary Ruby DSL code, edit the DSL, and then transform back to JSON using the cfndsl tool. In this way developers can use tools and a language familiar to them to develop CF templates; we’re lowering the bar to maintain and support the project.
Here’s an example workflow using cfn2dsl and cfndsl.
- Bob begins developing a new template. He writes in the Ruby DSL and then, using cfndsl, generates a JSON template.
- Bob commits the JSON template to the project’s git repository. He ignores the DSL file — it’s temporary.
- Alice, an experienced delivery engineer who is comfortable editing JSON directly, pulls the latest version of the repo and modifies the template without using the DSL.
- Bob needs to make some more changes but the JSON version has moved on since he last made changes — so he pulls the changes that Alice committed, and uses our cfn2dsl tool to generate the Ruby DSL from the JSON, which he then edits (and uses cfndsl to regenerate the JSON, which he then commits)
To start using cfn2dsl, please fork cfn2dsl and check the README to start translating your JSON templates into Ruby DSL.