Do we need QAs/testers?

 

“Do we (still) need QAs?”

… and flavours thereof, is a question I have been hearing for years now.

The same has been asked of Tech Leads, Operations Engineers, “Front End” devs, “Back End” devs, Security, Iteration Managers/Scrum Masters, Business Analysts, etc. Anything that is not a full-stack dev. The #NoOps conversation is interesting research material.

It seems to me that this question stems from a misinterpretation of agile and lean startup materials.

Idealistic hopes of cutting costs, removing waste (and blindly classifying some roles as such), delivering faster, etc., have caused some to think that role specialisations can all be generalised into being “Engineers” and that we can all self-organise and just do it all.

Continue reading

Using Elasticsearch field collapsing to group related search results

Realestate “Developer Projects” are an important feature offered by REA to our customers (i.e. realestate agencies and property developers). An agency is able to use projects to group related listings so that they are displayed together as a group in our search results.  To implement this feature, we have chosen to use Elasticsearch’s support for  “field collapsing”.

The remainder of this article includes a summary of the problem, the approaches considered, and a bit more detail on the chosen approach.
Continue reading

DevOpsGirls becomes a community

DevOpsGirls started a year ago with a bunch of enthusiastic REA people who wanted to make a difference in diversity in Tech, particularly with women who are interested in DevOps.

After running two highly engaging bootcamps we have been discussing a lot about what’s next for this initiative. REA has been an amazing incubator of the idea and has provided everything that we needed so far to kick-start the movement, but among the organisers we had a feeling that if we wanted to reach the next level we needed bigger support from other individuals and organisations. That’s difficult to do if it’s attached to a single sponsor. With the great response we have had from the industry and the beautiful experience of having external coaches joining the previous boot camp has help us determine the next step.

And so — I am super happy to announce that DevOpsGirls will become part of DevOps Australia – an organisation well know for running conferences like DevOps Down Under! Thanks to Matt Jones for the support and we really believe this is a step in the right direction and would help DevOpsGirls operate in a more effective way.

Continue reading

How We Optimise Apache Spark Jobs

Here in Consumer Insights we have been operating Big Data processing jobs using Apache Spark for more than 2 years. Spark empowers our daily batch jobs which extract insights from consumer behaviors from tens of millions of users who visit our sites. This blog covers our usage of Spark and aims to provide some useful insights for optimizing Spark applications based on our experience.

Continue reading

Game of Lambdas

Recently we launched a recommendation engine, which was built using AWS Serverless technology. The journey of implementing this solution turned out to be an interesting one on a number of levels. Since its deployment into production, we thought it would be a good idea to share some of our lessons.

Bucket of Data

Essentially the system transforms a very large dataset into smaller ones that are used to create audiences or data segments which are used for hyper targeted EDMs.

To get from the initial state to the final state, the data is transformed over several stages using 8 Lambdas. Continue reading

Static assets in an eventually consistent webapp deployment

The Problem

Deploying a high traffic website with zero downtime is a challenge – there’s a natural tradeoff between:

  • Performance and cacheability.
  • Getting updates versions of the application live.

The approach you use to manage your static assets plays a big role in this.

This post explains how we dealt with the challenges in our move from the data centre to a multi region highly available cloud-based architecture.

Continue reading