Hunting Down Memory Issues in Rails

On a Friday a few weeks ago, we deployed a set of minor changes to one of our Rails apps. That evening, our servers started alerting on memory usage (> 95%). Our initial attempts to remedy this situation by reducing the Puma worker count on each EC2 instance didn’t help, and the memory usage remained uncomfortably high through the weekend. On Monday, we popped open NewRelic and had a look in the Ruby VM section. Indeed, both the Ruby heap and memory usage of each web worker process had begun a fairly sharp climb when we deployed on Friday, after being totally flat previously:

Heap Size Growing

However, over the same period of time, the number of objects allocated in each request remained fairly static:

Object Allocations Over the Same Period

If our requests aren’t creating more objects, but there are more and more objects in memory over time, some of them must be escaping garbage collection somehow. Continue reading