How big is your data?

One of the major problems with the term big data, is that “big” is relative. If you’re at a conference or reading articles about big data, you never get informed how much data you need to be dealing with in order to consider using the tools they are talking about. You’re probably getting all excited about using new tools, buzzwords flying everywhere and you’re thinking “I should learn all these things, my data is huge!”. Here’s a tip, it’s probably not that big.

Tools like hadoop, and EMR which makes hadoop easier, came about because scaling vertically is expensive and scaling horizontally is was hard. So they sought a way to make the latter easier. The problem with making scaling easier is that people don’t try to optimise first. That can be fine if you’ve got more money than time, you always have to make a trade off in those situations. If it takes you a month to optimise something and you’re strapped for time, just throw another 10 servers at it, things will be fine right? Well maybe for 6 months. Continue reading

Hack it Forward: Time to Give Back

In October 2014 we ran our 17th Hack Day event at REA, this time focusing on social issues and hacking on outcomes to benefit the wider community. We partnered primarily with 4 charitable organisations:

In addition, participants were encouraged to collaborate with other community groups or charities on a project of their choice. 10 teams were formed in total with a focus on delivering real value to people in need.

How did we go? See for yourself:

We blogged the entire two day experience via Storify:

The Ops Dojo

No doubt that the term Devops is probably one of the most worn-out concepts in IT these days, maybe only slightly overtaken by the emerging buzzword of the moment: docker, docker, docker. But from the vast area and topics that Devops can cover, the one that attracts me more lately is how to build the Operations culture in an organization: bringing down the walls, sense of ownership, capability to operate a service/system, etc.

In REA we are quite proud of the Devops culture in the organization and we work really hard towards creating highly autonomous cross-functional teams that can operate efficiently end to end. This includes the ownership and operation of the services and products the team builds and maintains. This vision of Operations as a capability/responsibility in a team, rather than a separate team or static role of the engineers, has been providing really good results since we started championing it a few years ago, therefore the willingness to commit to it. There is probably not a single way to boost this capability. Sometimes we achieve it  by embedding an Ops specialist in the team, whose mission and passion is to enable and train the rest of the team, but which in general means that we are committed to invest in raising the technical capability and awareness of all our engineers and providing them with the level of access to our systems/data/services to do their job in an efficient manner.

In this post I am going to focus in one of the ways that we are trying to build/improve that Operations capability across the company based on running a set of community driven activities that we have called the Ops Dojo. The idea behind the concept is not new for REA, we already had several ad-hoc training sessions, plenty of brownbags and a strong culture of sharing what we learn across the organization. The Ops Dojo is just an initiative to have even more of that and to build a community that follows it up and makes it happen regularly. The Ops Dojo is just one form of the guild initiatives which are on the rise at REA as a mechanism to share knowledge and passion and today covers such diverse topics as Security, Public speaking, Delivery Engineering, Agile, Happiness, etc… Continue reading

A microservices implementation retrospective


Over the last year at (REA), I worked on two integration projects that involved synchronising data between large, third party applications. We implemented the synchronisation functionality using microservices. Our team, along with many others at REA, chose to use a microservice architecture to avoid the problems associated with the “tightly coupled monolith” anti-pattern, and make services that are easy to maintain, reuse and even rewrite.

Our design used microservices in 3 different roles:

  1. Stable interfaces – in front of each application we put a service that exposed a RESTful API for the underlying domain objects. This minimised the amount of coupling between the internals of the application and the other services.
  2. Event feeds – each “change” to the domain objects that we cared about within the third party applications was exposed by an event feed service.
  3. Synchronisers – the “sync” services ran at regular intervals, reading from the event feeds, then using the stable interfaces to make the appropriate data updates.

Integration Microservices Design

Continue reading

One to One Feedback at REA

Continuous improvement and change is one of the constants at REA. Hence it comes to no surprise that we also think about ways on how to further improve our culture. We have many people who are passionate about implementing new ways of working together. They come up with ideas and experiment on them within their teams. This is a great way to pivot and see which of these ideas work, how they work best and which ones don’t work at all. One of these cultural experiments was the so-called “Speedback”, which evolved from an experiment in one team to a ritual in many teams. Read about our Speedback sessions @REA in Ilya & Greg’s post.

– Victoria Schiffer

One to one speed feedback sessions help to build trust in a team, and contribute to creating an environment of learning. Trust is important because it enables the team to have quality conversations, for example about contentious topics. An environment of learning enables teammates to grow in areas relevant to their personal development goals. Helping individuals achieve mastery is a great way to keep motivation high. Below are the key properties of the feedback sessions we run in some teams at REA. Continue reading

I’m a programmer, not a writer

Hi, I have a very direct, blunt style of writing.

My name is Stephen and I’m a Devop working at REA with about 7 years experience writing Python.

Communication is difficult. I don’t think I’m alone in this but it’s certainly something I’ve battled with over the years. Nothing is more daunting to me than a “clear communication skills” requirement in a job posting.

I even nearly failed TEE English! My final year of highschool I got 20% in my English exam in the first semester. That was a disheartening point in time. The good news is I learnt from that failure and put in a lot of effort to actually pass (barely!) in the second semester.

The great news is form isn’t as important as intent. This is why we can have such internet phenomena as doge (much internet, so hilarious) and dolan. It’s why we can have conversations over twitter and facebook comments.

Writing a blog post is daunting, especially if there is a chance many people will read it. All kinds of insecurities rise to the surface. Especially in the face of the great internet force of grammar Nazis.

I’m not quite at the point of my own public blog (where would the content come from?!) but I have an internal company blog I write to, I started using Twitter (that’s a micro blog right?) and I’m writing this.

Step 1: Put out content
Step 2: ???
Step 3: Profit

Communication is hard, but that shouldn’t stop us.