Splunk’ing Netsuite

Splunk’ing Netsuite is the act of aggregating logs from Netsuite into Splunk

– Roi

So what is this blog post about? Allow me to introduce the problem – we have Netsuite. We use it extensively in REA. We also have Splunk which is used, even more extensively, I reckon, throughout the company. We would like to have our Netsuite execution logs in Splunk. Sounds simple? well, as it turns out – that’s not that simple.

Continue reading

What good software looks like at REA

Introduction

Over the years we have created a lot of software at REA ranging from internal tools used by our customer experience team, to mobile apps used by millions of users each month, to data processing engines that crunch hundreds of gigabytes of data. While each piece of software has different functional and non-functional requirements we still have consistent expectations of that software as well as alignment on architectural principles. By taking this general consensus floating in the ether and capturing it using succinct language within high-level categories we have formed a foundation for discussion to ensure we can keep producing good software regardless of team, technology, and timeframe. Additionally, we routinely use this framework to assess all new and existing software to ensure our software continues to meet expectations and, where concerns are detected, to drive change.

The categories – overview

When considering the qualities of good software we use the following lenses:

Continue reading

Use the decorator pattern for clean I/O boundaries

There’s a pattern that I keep recommending to teams over and over again, because it helps separate concerns around I/O; sending and receiving things over a network, interacting with AWS, saving and loading things from data stores. It’s an old idea; you’ll find it filed under “Decorator” in your Gang of Four book.

For our purposes here, this is a compositional pattern that allows us to stack onion-rings of concern around an I/O boundary; strings on the wire, transfer protocols (eg HTTP) , formats (eg JSON), business concepts. There is no code in existence that should be concerned about more than one of these at once; they can be structured as stackable layers, and other modules can pick which level of detail they wish to talk to.

Continue reading

Making Scala and functional programming more accessible in REA

In REA, we have a list of highly sought after internal courses available on various topics including AWS, Docker and Elasticsearch. On 7th and 8th May 2018, we added Scala to the list, with over 20 people attending!

How did this all start?

Scala and functional programming have always had some mystique surrounding it. There is this idea that it is too hard or not pragmatic enough. Existing courses are very technical and many people including myself have struggled to complete them.

Continue reading

Do we need QAs/testers?

 

“Do we (still) need QAs?”

… and flavours thereof, is a question I have been hearing for years now.

The same has been asked of Tech Leads, Operations Engineers, “Front End” devs, “Back End” devs, Security, Iteration Managers/Scrum Masters, Business Analysts, etc. Anything that is not a full-stack dev. The #NoOps conversation is interesting research material.

It seems to me that this question stems from a misinterpretation of agile and lean startup materials.

Idealistic hopes of cutting costs, removing waste (and blindly classifying some roles as such), delivering faster, etc., have caused some to think that role specialisations can all be generalised into being “Engineers” and that we can all self-organise and just do it all.

Continue reading

Using Elasticsearch field collapsing to group related search results

Realestate “Developer Projects” are an important feature offered by REA to our customers (i.e. realestate agencies and property developers). An agency is able to use projects to group related listings so that they are displayed together as a group in our search results.  To implement this feature, we have chosen to use Elasticsearch’s support for  “field collapsing”.

The remainder of this article includes a summary of the problem, the approaches considered, and a bit more detail on the chosen approach.
Continue reading