HomeAbout UsNews, company reports and blogBlogTechFinding your way in the world of microservices – Treasuremap!

Finding your way in the world of microservices – Treasuremap!

Article by Supriya Joshi
Published on Read in 4 mins

“Microservice” is becoming a buzzword in the industry today. Developers all over are experimenting and learning more and more about it. It’s a cool concept and we’ve started working with it at REA. There were several challenges that we faced with microservices and one of the key ones was tracking and documenting them.

After lots of deliberation, discussions and debates we could summarise our dilemma and needs as:

Wouldn’t it be cool to have a living, breathing tracking system for all the microservices that exist in our ecosystem which will emit all the key information that we would like to know and have?

This was a fantastic idea but then there was a lot of ambiguity. What exactly do we mean by “key information”? Who wants to know about the microservices being implemented? And most importantly, how can we make this data available without having to maintain a roster or actually having a complex system to do this?

The process that followed was more of discovery and self analysis. We had to answer a few key questions:

Why do we need the information?

  • To understand what business processes are already implemented
  • To understand the relationships between microservices so we know what will break if we change something
  • To find out which microservices/applications/services are used currently to satisfy a business process
  • To find out which version of the microservice is running in production
  • To understand whether a microservice/application/service is used actively in production or is it obsolete
  • Which team owns the services/applications
  • Where to find the code for the artefacts

What is the problem that we are trying to solve?

  • Technical documentation of any kind is challenging
  • It is one dimensional and hard to maintain
  • It’s a point in time truth and mostly does not contain information that changes often and is sought after constantly

Who are the people or the stakeholders for this?

  • All the technical staff, business analysts, product owners or anyone who wants to know anything about the system.

How do we want to do it?

  • We wanted to build a few microservices to do this of course 🙂 The idea is to build a self-discoverable set of microservices which would have the capability of giving away information when it’s asked for.

Where will this information come from? Data can be divided into two parts:

  • That which is a “rumour”
  • That which is a “fact”

Rumours are more point in time and collected from static information entered by people. Hence they are rumors. This is information given in the “README” files in the Git repository. It can be information given by business analysts about what they think is being implemented. It is a time line of information which can be collected from different sources like “community pages put up by teams”, “SharePoint portal documents”, etc.

Facts are information collected from systems. Information like “versions” deployed on production, GitHub issues which are open still, when did the last deployment happen, when was the last time the microservice was invoked in production, change log information between versions deployed, any information which can be collected from the production logs, etc.

Our implementation, treasuremap, looked a bit like this:

The idea is to create different microservices which can pull data from different sources such as GitHub (or any version control system), live services and production systems, logs from live services (Splunk, logstash, New Relic, etc. if available) and store this metadata in a file store.

The rumour-mill microservice then ingests all the data from various sources into a graph database. Using a graph based database like Neo4j gives us the facility to search the data seamlessly and also provide a browser based search client with good data visualisation. Since rumour-mill itself is another microservice, we can then build a custom UI to extract data in a more consumable format as per the requirements of various consumers. That custom UI will be the treasuremap. This treasuremap can be built in whichever way you want the data to be represented (can be fancy graphs or plain old text shown on a web page! or an rss feed)

The whole design is scalable in a way that we can add custom microservices (such as one to integrate with pact results) in order to pull the data and aggregate it and present it in a meaningful manner.

We are still analysing and experimenting with this idea and trying to find out more sources of useful and meaningful data. Currently we have a crude implementation of the treasuremap with data mostly handcrafted in rumour-mill. The design is still an idea with  a nascent implementation.

Do let us know what you think about this idea and implementation and what kind of information would you like from your microservices.

 

More from the blog

From Teaching to Software Development – A Year On

Category: Career stories
Published on

A vision of the future is a bet we are prepared to make

Category: Tech
Published on

One year at REA: Lessons, Growth, and Gratitude

Category: Life at REA
Published on

Introducing Argonaut – Part Three.

Category: Tech
Published on

Introducing Argonaut – Part Two.

Category: Tech
Published on

Introducing Argonaut – Part One.

Category: Tech
Published on

View all articles