Hunting Down Memory Issues in Rails

On a Friday a few weeks ago, we deployed a set of minor changes to one of our Rails apps. That evening, our servers started alerting on memory usage (> 95%). Our initial attempts to remedy this situation by reducing the Puma worker count on each EC2 instance didn’t help, and the memory usage remained uncomfortably high through the weekend. On Monday, we popped open NewRelic and had a look in the Ruby VM section. Indeed, both the Ruby heap and memory usage of each web worker process had begun a fairly sharp climb when we deployed on Friday, after being totally flat previously:

Heap Size Growing

However, over the same period of time, the number of objects allocated in each request remained fairly static:

Object Allocations Over the Same Period

If our requests aren’t creating more objects, but there are more and more objects in memory over time, some of them must be escaping garbage collection somehow. Continue reading

What your AutoScaling Groups can learn from rabbits

Rabbits, they are small, the are cute, they are fluffy…. and they are terrible fighters*.

rabbit-705759_640

Which is why evolution has favoured the skittish amongst them.

The rabbit which was overly cautious and ran away and the first hint of danger survived and went on to produce more rabbits. But the rabbit which was more laid back and waited until it was sure it was in peril before trying to flee did not. Of course running away does have a cost, and if you spend all your time running at the slightest sound you’ll expend a lot of energy and have no time to nibble at the grass and gain more**.

So what do rabbits have to do with Auto Scaling Groups (ASGs)?

Continue reading

‘cfn2dsl’ – an open source tool to translate CloudFormation into Ruby DSL

In REA, Amazon Web Services (AWS) is our major development and production environment, and CloudFormation (CF) is one of the best tools we’ve found to manage deployments in AWS. At the time of writing, JSON is still the only template format supported by CloudFormation; but if you search for “Programming in JSON” in your favorite search engine, the results may be very disappointing. Some developers find writing JSON templates hard and have trouble with the data format, especially when the templates are big (you can’t have comments, syntactic strictness, etc).

At REA, we encourage people to explore and find new technologies to solve problems, improve product quality and speed up deployment cycles; this freedom to explore has given us a few choices for addressing this problem.

Continue reading

Using our everyday dev tools for effective Load and Performance testing

Previously at REA we’d had very special tools for Load and Performance testing that were quite expensive, very richly featured but completely disconnected from our every day development tools. The main outcome of this was that we ended up with a couple of engineers who were quite good at L & P testing with our enterprise tools while the majority of engineers found the barriers too great. We have moved to an approach which is far more inclusive and utilises many of the tools our engineers are working with on a daily basis. I’ll talk about how we did this for the most recent project I worked on.

Continue reading

Acquisition, Ownership, and Migration of Legacy Applications

Recently, I presented a talk at the DevOpsDays conference in Brisbane, Australia on how a team of people can utilise the best of DevOPS to deal with legacy applications. Specifically how migration, ownership, maintenance and handover of legacy applications can be handled.

The below video is a tale of such a journey that my team undertook recently.

DevOpsDays Brisbane 2014 – Mujtaba Hussain – Acquisition, Ownership, and Migration of Legacy Applications from devopsdays on Vimeo.

devops, and associated thoughts!

Here at REA, we have been firm believers in accountability of work, from the inception to delivery! To cement this idea, we instituted the concept of devops here some months ago, were a dedicated OPS person would co locate with the development teams for particular projects and be just like another developer. Conversely, a developer from one of the teams would colocate with the Operations team and experience being a systems administrator. The reasons were to get members of both teams aware of the fine grained issues and complexities that develop within the processes of development and/or deployment. We frequently rotated the personnel through this process to better spread the lessons learnt. Being an observer of this process, I have come to a few conclusions.

1. One person cannot be a ‘devops’ person.

It is not possible to have one person in a position where he/she is expected to do deployment and development while all others are expected to do only one of the above. The responsibility, and thats what it is, has to be shared across the team. It is the responsibility of the entire team to ensure that that the delivered product works in production, and not just one person.

2. It is a mentality, not a position.

It should not be marketed as position to slot one person in, or even rotate people through. A software developer who writes the code should be as concerned about his/her code working in production as he/she is in development. Definitely operations are more experienced with deployment and maintenance of products in production like environments, but that does not mean a developer should not base his software decisions before being aware of implications of deploying it in production, or be unaware of the status of his code after it passes quality assurance. The thinking behind devops is the critical bit of software development.

3. Operations need to be consulted early and often.

Operations are somehow considered as the team to call in last, when you have your product ready or almost ready. That in my humble opinion, is a fallacy. They need to be involved in every inception, every retrospective and every card kick off as any other team member. Their thoughts are most invaluable as they will look at a piece of work, not as whether it can be done, but whether it should be done based on the present state of production; whether there are some inherent dependencies that need to be resolved. Not only will this ensure that software developers will start writing software with production in mind, rather than their own laptop, but also the fact that this will spread the knowledge of state of production to people who might not have it to begin with.

devops is not a magical word that resolves our issues of deployment, dependencies etc, but it does get us started on the road of understanding that we are working to deploy to a production box, not our own laptops; and it is the responsibility of everyone involved to make sure that happens, and not just one lone person.