Getting Shit Done

This post was originally published internally, as an appeal to REA colleagues.

“Getting Shit Done” is the catchphrase on everybody’s lips, and deservedly so!  When we deliver new functionality, our users regroup and flock to us, our customers grudgingly respect us, and our shareholders rejoice.  When the novel concepts invented by our product managers take shape as they watch, their eyes light up with pride and enthusiasm.  Programmers are never happier than when fire and magic fly from their fingertips; products that change people’s lives materialise from thin air, and insurmountable problems melt like butter.  Beer flows freely, parmas are devoured and our managers circulate glowing praise within the company.

We have all felt the opposite too; long months gone by without new features, frustrated and bored developers; product managers forced to nervously adjust their collars and disappoint their superiors, with often dense technical reasons they can barely hope to convey.  New features pop up like mushrooms on our competitors’ sites, and we wonder: why didn’t we do this years ago?

Yet when I hear this phrase “Get Shit Done”, I grimace; my teeth clench and my back involuntarily stiffens.  Why?  There is truly nothing I want more, and it is clearly important; many of our most talented teammates live by it.

Let me go back in time; my first job was in a raw startup. The tech industry was beginning to pick up the pieces from the wreckage of the late 90s; the founders were recent university graduates like me, only a few years my senior. It was an exhilarating experience, and brutal.  At times, whether we got paid or not depended on whether we shipped a feature that week.  We were all close, and sometimes dear teammates were tearfully given notice, only to be brought back into the fold when a sale came through. We came so close to the edge; but we wanted to succeed.  And we coded; we coded like crazy.  We shipped enormous, complex features, at a speed I can scarcely believe now. Sales started to pick up; the company pulled through, and to this day is a great success story. I could never go back to this kind of lifestyle, but the memories are dear to me — what an achievement, built with camaraderie, callused hands and sweat.

There was a price though; this rapidly created codebase was pure legacy from the moment it hit source control.  There were baffling, crippling architectural decisions that not even inexperience could rightly explain.  There were experiments tried and failed, hacks and shims ossified and immortalised in untouchable knots of mystery.  There are to this day legendary Easter eggs and jokes woven into its fabric that may never be extricated.

Within three years, “Getting Shit Done” became a dire difficulty; we were no longer in immediate danger of failure, but extending the software was hard, sometimes impossible.  We had to say no to our bosses for the simplest of requests, for reasons that were hard to articulate.  The wheels slowed, customers complained; it was time to take stock and reinvest in our infrastructure.  Was it worth it?  Yes! A hundred times over, yes! We built a successful product out of nothing, and even the most horrid legacy was a small price to pay.  But a great price it was; much of that code still remains, and when I meet their engineers today — who know me only through code comments and commit logs — I will apologise at length, and shout them as many beers as they care to drink.

Fast forward to today, and we find ourselves in a new tech golden age; the 90s long forgotten, and startup culture lionised and worshipped as a glorious path to virtue and riches.  Every personal habit of Zuckerberg, Jobs, and every YC-funded college dropout is scrutinised endlessly, while starry-eyed gold-panners return to San Francisco, failing, suffering and burning out in droves.

It should give us pause when established companies ape the language and habits of these foolhardy ingenues.

This brings us to REA. We have been around a long time for a web-focused company, and have weathered many storms. The closest we have recently come to a ship-or-die moment was circa 2008, when we commissioned a US company to repurpose their gargantuan listing search product for us. Most of us now know it as our most painful, unmovable legacy system, but there was a time when it made the difference between survival and stagnation.  REA could have withered and faded, but this hefty monolith helped us steer the ship around and become industry leaders and stock market darlings.  It still hurts, but we have the luxury of hurting in great fortune and prosperity.

We must recognise though, that as an established company with a large body of standing software assets, that these moments are vanishingly rare. Overwhelmingly what stops us from delivering software in a fast, reliable cadence is not that we take the time to mend our assets and produce good work, but the accumulated technical debt and shortcuts of past years. Put another way, what prevents us from “Getting Shit Done” is the last time we got shit done.

We are fortunate to have product managers who understand this deeply, and are committed to removing roadblocks, and not simply ploughing through them. In recent times, we have made great strides. To list a few:

  • Cloud based infrastructure and disposable “cattle” environments have given us flexibility in deployment and tooling, and freedom from the creeping sticky decay of special “pet” environments.
  • Moved staging environments to the cloud, where teams need not fight for shared resources.
  • Moved production to the cloud where we can continuously deploy, roll forward and back with ease.
  • Split large systems into smaller ones that can be quickly redeployed, without company-wide code freezes.
  • Looked towards Functional Programming, and Scala, as a way of writing software that is less error-prone, more understandable, testable and robust in the face of change.
  • Isolated data stores and minimised fragile run-time service dependencies.

Sometimes, this investment results in frustration; why aren’t things getting done faster? In fact though, we are already reaping the rewards; each new service is more efficiently released than the last, and from what we can tell decays more slowly.  Had we simply taken a startup-style release-at-all-costs mentality, we would actually have released less, and incurred much higher business, technical and personal costs.

By and large we have a healthy attitude to the balance required here.  It is imperative, though, that we remain vigilant, and actively conscious of this balance. Consider: recently we took a Hack Day project, restoring a long-dormant feature, and took it to production.  It was an excellent result; a feature missed by many users was back; some irritating legacy was retired, and we drew great praise from around the company, even from our CEO herself!  We took an idea, ran with it and shipped it – no delays, no bullshit. Win!

A careful observer would notice a cost though; the delivery timeframes of the teams whose members were sidetracked slipped noticeably, and operations folk were derailed from other tasks to support the introduction of the new feature.  Was it worth it?  I would say it probably was, but we should be concerned that nobody talked about the tradeoff.  The folk who implicitly bore the cost of the feature’s imprint were not acknowledged with the same vigour, or at all; only the act of releasing a new feature drew acclaim.

Even if this was well judged, what about the next time?  Will the heady glamour of shipping overwhelm all other concerns, no matter their weight? The truth is that regular, reliable shipping for a company of our age and size cannot just be focused on the pointy part of the spear — the tiny part of the iceberg that glistens in the sun and is pooped on by cormorants.  We have to know that a reliable cadence can only be sustained by taking the whole lifecycle into account, insisting upon quality, expanding our maintenance liability only with care, and retiring obsolete components.

By all means, let’s Get Shit Done! Let’s continue blazing a trail as one of the best tech companies in Australia, and attracting the best people. Let’s keep releasing great new features as fast as we can, and keep our users hungry for more.  Be careful, though, to know what that truly means for a company of our size and position; know that shortcuts aren’t always the shortest path, and that burning everything we’ve learnt on the discredited altar of Silicon Valley’s worst excesses isn’t necessarily going to get us there.

  • epiineg1

    One of my favourite realisations is that you can finish the first 95% of any task/project/bug within the 50% of the time you think you should allocate to it, and you should, not will, should spend the next 50% of your time to the last 5% of the task. And the last 5% is as important as the first 95%. But its boring and usually doesn’t fall under the umbrella of the “shit” you set out to do.

    Achieving “goals” is very important. But if you choose to do things quick instead of right, then get ready to achieve the same goal many times. And right takes time.

    • Tei

      What motivate myself to do that 5% with the same enthusiasm the first 95% is that users get a lot of mileage of small shit that may takes minutes to do. These small humble features, small usability changes, is what make software feel like made of magic, and not like swiming in a pool filled with honey. Connected to this is horror stories of users doing manually task that are simple to automatize. Nothing creep more programmers than people doing shit manually.

  • Simon

    Beautifully written, Ken. Reminds me a lot of this text by Bastiat which you might find interesting – http://bastiat.org/en/twisatwins.html

    For those who can’t be bothered clicking the link, here’s an edited version of the first paragraph:

    “In the department of economy, an act, a habit, an institution, a law, gives birth not only to an effect, but to a series of effects. Of these effects, the first only is immediate; it manifests itself simultaneously with its cause – it is seen. The others unfold in succession – they are not seen…it almost always happens that when the immediate consequence is favourable, the ultimate consequences are fatal, and the converse.”

  • Murray

    When I visited ronjeffries.com just now, the random article that popped up was http://ronjeffries.com/xprog/blog/quality-speed-tradeoff-youre-kidding-yourself/ which seems to be saying quite similar things to your post.

    • Ken Scambler

      “Low quality has an immediate and visible negative impact on delivering “fast enough””
      “As far as I can tell, the sole impact of reducing quality is to make things take longer.”
      “To a first approximation, then, looking at a random project, if we want to speed up, we must increase quality.”

      Love it! Thanks for the link.

  • Pingback: Catalin Istratoiu()

  • J

    In my experience and opinion realestate.com.au don’t necessarily post the correct information then refuse to reply to emails. I would never use their service.