On a Friday a few weeks ago, we deployed a set of minor changes to one of our Rails apps. That evening, our servers started alerting on memory usage (> 95%). Our initial attempts to remedy this situation by reducing the Puma worker count on each EC2 instance didn’t help, and the memory usage remained uncomfortably high through the weekend. On Monday, we popped open NewRelic and had a look in the Ruby VM section. Indeed, both the Ruby heap and memory usage of each web worker process had begun a fairly sharp climb when we deployed on Friday, after being totally flat previously:
However, over the same period of time, the number of objects allocated in each request remained fairly static:
If our requests aren’t creating more objects, but there are more and more objects in memory over time, some of them must be escaping garbage collection somehow. Continue reading
Let’s face it, writing software is hard. And frankly we humans suck at it. We need all the help we can get. Our industry has developed many tools and techniques over the years to provide “safety rails”, from the invention of the macro assembler through to sophisticated integration and automated testing frameworks. But somewhere along the way the idea of static analysis went out of favour.
I’m here to convince you that static analysis tools still have a place in modern software engineering.
In REA, Amazon Web Services (AWS) is our major development and production environment, and CloudFormation (CF) is one of the best tools we’ve found to manage deployments in AWS. At the time of writing, JSON is still the only template format supported by CloudFormation; but if you search for “Programming in JSON” in your favorite search engine, the results may be very disappointing. Some developers find writing JSON templates hard and have trouble with the data format, especially when the templates are big (you can’t have comments, syntactic strictness, etc).
At REA, we encourage people to explore and find new technologies to solve problems, improve product quality and speed up deployment cycles; this freedom to explore has given us a few choices for addressing this problem.
As previously discussed we’re pretty keen on micro services at REA. Our delivery teams are organised around small, autonomous “squads” that get to choose pretty much any language and technology stack they wish to implement their solutions.
This inevitably leads to a fairly broad church of language use. 🙂
Over the last year at realestate.com.au (REA), I worked on two integration projects that involved synchronising data between large, third party applications. We implemented the synchronisation functionality using microservices. Our team, along with many others at REA, chose to use a microservice architecture to avoid the problems associated with the “tightly coupled monolith” anti-pattern, and make services that are easy to maintain, reuse and even rewrite.
Our design used microservices in 3 different roles:
- Stable interfaces – in front of each application we put a service that exposed a RESTful API for the underlying domain objects. This minimised the amount of coupling between the internals of the application and the other services.
- Event feeds – each “change” to the domain objects that we cared about within the third party applications was exposed by an event feed service.
- Synchronisers – the “sync” services ran at regular intervals, reading from the event feeds, then using the stable interfaces to make the appropriate data updates.
Our team decided to move to a micro-service architecture, and we started wondering how we would test all of our integration points with lots of little services without having to rely on integration tests. We felt that testing the interactions between these services quickly become a major headache.
Integration tests typically are slow and brittle, requiring each component to have its own environment to run the tests in. With a micro-service architecture, this becomes even more of a problem. They also have to be ‘all-knowing’ and this makes them difficult to keep from being fragile.
After seeing J. B. Rainsbergers talk “Integrated Tests Are A Scam” we have been thinking on how to get the confidence we need to deploy our software to production without having a tiresome integration test suite that does not give us all the coverage we think it does.