Real-time dashboard for Agent Admin

The Customer Platform team at REA is in charge of developing and maintaining the Agent Admin application, which is the application real estate agents use to upload properties. Agent Admin is a Ruby on Rails application that in order to be deployed goes through a deployment pipeline on an (almost) daily basis. We use Atlassian Bamboo as our CI server.

Deployment Pipeline

The first build triggered after a commit in the Agent Admin (aka AA) codebase generates a package and determines  the version number of the package to be released. In this stage, unit tests, integration tests and a small set of functional tests will be executed. If the build is successful, the package gets promoted to the next step.

The second stage is about certifying that the package built in the previous step passes a more extended set of end-to-end tests. If the package goes through this step, we can execute exploratory tests inside our test environment where the package gets deployed.

After approval, we can deploy to Staging, our production-like environment.

Last step, we can deploy on-demand to production.

Therefore, we’ve got 4 different stages where we can have different versions of the package.

Why did we need a dashboard?

Working on the project for a few months, we developed:

  • the need to have a way of displaying the different versions in all the different steps;
  • the need to monitor the status of our builds and the status of staging/production in real-time;
  • the need to alert us when the build turns into red.

For these reasons, and also having a big screen available visible to the whole team, we decided to develop a dashboard.

Features in the dashboard

That is how the dashboard looks:

Agent Admin dashboard

Agent Admin dashboard

In the dashboard, the term ‘ALPHA‘ is associated to the build triggered by our commits,  ‘RC‘ is used for the release candidate version as it goes through the set of extended end-to-end tests, ‘STG‘ stands for Staging and ‘PROD’ for production. ‘DOMAIN‘ is associated to the build of our internal gem representing the domain model, which is fed into the Agent Admin application.

The colors of each row gives us a feedback around the status of the current build (for ‘DOMAIN’, ‘ALPHA’ and RC’) and the status of the application deployed (‘STG’ and ‘PROD’). A green background means that the current build is successful / application is up and running; a red background (see below) means that the latest build failed / application is down.

Red build

Red build

Notice also the ‘spinner’ image that is used to give us information whether the current build is building in real-time.

The number in white is the version number. The ALPHA number is the version number associated to the latest successful commit build; the RC number is the version number  associated to the latest successful certification build; the STG number is the version deployed in staging and the PROD number is the version deployed in production.

It’s also possible to click on the RC number and the STG number to view the JIRA tickets (see below) developed between that version and the current version deployed in production.

JIRA tickets

JIRA tickets

For instance, looking at the screenshot above, we know there is only one task (code: CQA-81) that is going to be deployed from Staging to Production.

Technical implementation – Event-driven architecture

The dashboard is powered by Node.js (server-side) and is using Web Sockets (specifically the ‘socket.io‘ module) to offer real-time features.

Below you can view a basic diagram of the architecture of the system developed:

Architecture

Architecture

The app is using a NPM module called ‘bamboo-api‘, created by Sebastiano Armeli (@sebarmeli) that retrieves data from the Bamboo JSON API to gather information about the status and the version numbers for the ‘Alpha’ and ‘RC’ layers. Regarding the ‘STG’ and ‘PROD’ rows, the app requests and scrapes specific internal URLs to get the versions of the application deployed (using the request module).

An intermediate layer (agentadmin.js) is used to offer an abstraction around the usage of the ‘bamboo-api‘ and ‘request‘ modules.

The controller is where the event-driven architecture shines. All the different results from the Bamboo API/ private URLs are wrapped in events that are emitted from there.

On the client (still in JavaScript of course) we listen for the events and we update the UI according to the data carried by the event.

The mespeak.js library has been used to make the app alert us when the commit build breaks 🙂

To gather the JIRA tickets between a certain version and the production version, we’ve been looking at the Git commits between two versions, as our Git commits follows a specific pattern (containing the JIRA ticket number).

"JIRA-<number>: <message>"

Code samples

Asynchronousity is the core concept and probably one of the interesting features in Node. Callbacks are usually passed in methods and they are executed once the function returns.

Let’s have a look at some code samples. In the action associated to the request that displays the dashboard, we can see how we get some of the values from the Bamboo API in parallel:

async.parallel({
    alpha_number: function(callback){
      agentAdmin.getAlphaVersionNumber(function(result){
        logger.info("ALPHA" + result);
        socket.emit('alpha_number', (!result || result === "Unreachable endpoint") ? "DOWN" : result);
        callback();
      });
    },
    rc_number: function(callback){
      agentAdmin.getRCVersionNumber(function(result, cert_number){
        socket.emit('rc_number', (!result || result === "Unreachable endpoint") ? "DOWN" : result);
        socket.emit('rc_cert_number', (!cert_number || cert_number === "Unreachable endpoint") ? "DOWN" :  cert_number);
        callback();
      });
    },

    // Other stuff
});

We used the ‘async‘ module, which is a well-know NPM module to manage asynchronous behavior through promises. It provides a nice API to avoid nesting callbacks. As you can see in the code above, both the ‘alpha_number‘ and ‘rc_number‘ functions accept a callback as an argument. We emit a specific event when we get the results. In the above code, we emit two events called ‘alpha_number‘ and ‘rc_number‘.

In the implementation of the methods, you can see how we deal with the agentAdmin object, and one of the methods used is ‘getAlphaVersionNumber‘. Find below the implementation of the method:

getAlphaVersionNumber: function (callback) {
  bamboo.getLatestSuccessfulBuildNumber("CP-REGRESSION", "", function(result){
    callback(result);
  });
},

Again a callback invoked when the ‘getLatestSuccessfulBuildNumber‘ returns.

Regarding the real-time feature, we decided to ping the Bamboo API and the URLs giving information around Staging and Production every 4s (polling). Moreover, Socket.io keeps the connection open between server-side and client-side.

var io = socketio.listen(server);
io.sockets.on('connection', function(socket){
  assignValues(socket);
  setTimeout(assignValues(socket, true), 4000);
});

From the code above, you can see we opened a connection and in the callback we keep pulling the data.

On client (again JavaScript), we connect to the open socket and we listen on specific events:

var socket = io.connect(window.location.hostname);

socket.on('alpha_number', function (data) {
  applyNumber('#dev .value', data);
});

In this snippet, we are listening on the ‘alpha_number‘ event which is triggered (emitted) by a callback you saw in one of the previous snippets.

The tool is quite experimental, but allowed us to improve our productivity and our development/QA workflow. It also allowed us to experiment with Node and showed how you can develop an application fully in JavaScript.