Static assets in an eventually consistent webapp deployment

The Problem

Deploying a high traffic website with zero downtime is a challenge – there’s a natural tradeoff between:

  • Performance and cacheability.
  • Getting updates versions of the application live.

The approach you use to manage your static assets plays a big role in this.

This post explains how we dealt with the challenges in our move from the data centre to a multi region highly available cloud-based architecture.

Continue reading

Lean QA (aka QA Ops)

 

Developers have – with the advent of DevOps – been working more and more in Operations and Infrastructure. Testers however, have not.

Thus far, the testing personnel have been mostly or wholly assigned to application testing work. As SOFTWARE testers, we have only worked on software – and then mostly only on application software.

I pose the questions: What about infrastructure as code? Should that not be explicitly tested?

And: if Testers are meant to be testing the system, why then have they not explicitly been testing the whole system, infrastructure included?

I am going to make a case here for including QA in Operations and Infrastructure, by clarifying how I see the QA fitting in the DevOps world. Continue reading

Create a full-stack serverless web-app with a single file

Serverless is one of the biggest buzzwords of these years, and the implementation of solutions based on such architecture has been boosted by the release of AWS Lambda. This post describes a little experiment I conducted to learn more about this technology. I wanted to define and deploy a fully functional web-app (both back-end and front-end) based on RESTful services with one single file. Continue reading

Hunting Down Memory Issues in Rails

On a Friday a few weeks ago, we deployed a set of minor changes to one of our Rails apps. That evening, our servers started alerting on memory usage (> 95%). Our initial attempts to remedy this situation by reducing the Puma worker count on each EC2 instance didn’t help, and the memory usage remained uncomfortably high through the weekend. On Monday, we popped open NewRelic and had a look in the Ruby VM section. Indeed, both the Ruby heap and memory usage of each web worker process had begun a fairly sharp climb when we deployed on Friday, after being totally flat previously:

Heap Size Growing

However, over the same period of time, the number of objects allocated in each request remained fairly static:

Object Allocations Over the Same Period

If our requests aren’t creating more objects, but there are more and more objects in memory over time, some of them must be escaping garbage collection somehow. Continue reading

What your AutoScaling Groups can learn from rabbits

Rabbits, they are small, the are cute, they are fluffy…. and they are terrible fighters*.

rabbit-705759_640

Which is why evolution has favoured the skittish amongst them.

The rabbit which was overly cautious and ran away and the first hint of danger survived and went on to produce more rabbits. But the rabbit which was more laid back and waited until it was sure it was in peril before trying to flee did not. Of course running away does have a cost, and if you spend all your time running at the slightest sound you’ll expend a lot of energy and have no time to nibble at the grass and gain more**.

So what do rabbits have to do with Auto Scaling Groups (ASGs)?

Continue reading

‘cfn2dsl’ – an open source tool to translate CloudFormation into Ruby DSL

In REA, Amazon Web Services (AWS) is our major development and production environment, and CloudFormation (CF) is one of the best tools we’ve found to manage deployments in AWS. At the time of writing, JSON is still the only template format supported by CloudFormation; but if you search for “Programming in JSON” in your favorite search engine, the results may be very disappointing. Some developers find writing JSON templates hard and have trouble with the data format, especially when the templates are big (you can’t have comments, syntactic strictness, etc).

At REA, we encourage people to explore and find new technologies to solve problems, improve product quality and speed up deployment cycles; this freedom to explore has given us a few choices for addressing this problem.

Continue reading