On a Friday a few weeks ago, we deployed a set of minor changes to one of our Rails apps. That evening, our servers started alerting on memory usage (> 95%). Our initial attempts to remedy this situation by reducing the Puma worker count on each EC2 instance didn’t help, and the memory usage remained uncomfortably high through the weekend. On Monday, we popped open NewRelic and had a look in the Ruby VM section. Indeed, both the Ruby heap and memory usage of each web worker process had begun a fairly sharp climb when we deployed on Friday, after being totally flat previously:
However, over the same period of time, the number of objects allocated in each request remained fairly static:
If our requests aren’t creating more objects, but there are more and more objects in memory over time, some of them must be escaping garbage collection somehow. Continue reading
Rabbits, they are small, the are cute, they are fluffy…. and they are terrible fighters*.
Which is why evolution has favoured the skittish amongst them.
The rabbit which was overly cautious and ran away and the first hint of danger survived and went on to produce more rabbits. But the rabbit which was more laid back and waited until it was sure it was in peril before trying to flee did not. Of course running away does have a cost, and if you spend all your time running at the slightest sound you’ll expend a lot of energy and have no time to nibble at the grass and gain more**.
So what do rabbits have to do with Auto Scaling Groups (ASGs)?
realestate.com.au Android app security fix
Earlier this week REA Group released a security update to its realestate.com.au Android app. This corrected how usernames and passwords are sent from the app to its servers when users sign in. This vulnerability was never present in the iOS version of the application.
Let’s face it, writing software is hard. And frankly we humans suck at it. We need all the help we can get. Our industry has developed many tools and techniques over the years to provide “safety rails”, from the invention of the macro assembler through to sophisticated integration and automated testing frameworks. But somewhere along the way the idea of static analysis went out of favour.
I’m here to convince you that static analysis tools still have a place in modern software engineering.
This was my first DevOpsDays event, I had only recently started looking into the world of Ops. I was unsure how much I could get out of the two days, or what exactly I was getting into.
DevOpsDays is a rapidly growing, annual conference acutely focused on Developer Operations and Delivery Engineering. So what is DevOps? Nobody really knows yet, not even the engineers supposedly belonging to this group can concisely explain what it is. The scope of the word has grown unwieldy, despite its state of infancy. In fact, the crowd seem to hate the very use of the word. Everyone pauses and takes time to punctuate each mention of the phrase DevOps with a facetious “air-quotes” gesture. Continue reading
I sometimes get the feeling that Google is watching me.
I don’t mean the way it emails me three hours before I have to catch a plane, or how it recommends news articles I actually would like to read, or how it does that thing where ads follow me all around the internet. That’s not paranoia. We all know they’re watching us for that stuff.
No, I’m talking about REA Hack Day projects.