Splunk’ing Netsuite

Splunk’ing Netsuite is the act of aggregating logs from Netsuite into Splunk

– Roi

So what is this blog post about? Allow me to introduce the problem – we have Netsuite. We use it extensively in REA. We also have Splunk which is used, even more extensively, I reckon, throughout the company. We would like to have our Netsuite execution logs in Splunk. Sounds simple? well, as it turns out – that’s not that simple.

Netsuite is an accounting software. It is deployed and maintained in the cloud, i.e., not in our AWS accounts. It holds Customer information and their Invoice and Payment data, among other things. It’s extensively customised by us to integrate with our internal systems and services through small pieces of code called RESTlets, which are basically JavaScript scripts running inside Netsuite’s Node engine.

NOTE: There are other types of scripts and integrations but the main one we use and usually care about are the RESTlets.

Like most micro-services we run in REA, ours write logs which are indexed by Splunk and for most services we also add tracing information using the Transaction Id pattern.

With Netsuite, that’s almost always not the case (until now) and it’s mainly because Netsuite has its own logging system called Execution Logs which does not play well with the rest of our platform.

The situation is even worse as logs are accessible basically only when you are logged into Netsuite. Some of us have (or can easily get) Netsuite SB2 access (SB2 is Netsuite’s equivalent for Staging) however, Production access is rarer and either way – we need to be logged in to look at logs.

Some other issues with the default Netsuite logs are shown in the image above and are described here:

  • Log timestamps are divided into a date part and a time part and the latter has only minute accuracy
  • Logs are deployment-dependant, which means each RESTlet has its own logs that need to be accessed. In other words – if a distributed transaction makes use of more than one RESTlet script, both need to be consulted for their logs independently
  • Netsuite RESTlet has limitations, so integrating with Transaction ID HTTP headers is not possible and Transaction IDs do not exist in logs making it difficult to identify which log entry belongs to a specific transaction/request
  • Netsuite’s logging function is string-based while most of our services have switched to JSON-based logging

Considering all these limitations and the need to have better integration with Netsuite through our logs, the following solution was eventually implemented:

1. A new Netsuite Record (i.e. Table) was created for holding logging information in a more suitable way. The record holds relevant information such as a millisecond timestamp value and an associated Transaction Id value among other logging details:

2. A new shared logging library was introduced which uses this new record for persisting log entries and provides an easy interface for logging that is familiar and based on JSON objects. This library also recognises Transaction Id values passed in as part of the request’s payload.

3. A new RESTlet was created for reading log entries and exposing them as a simple paginated feed:

4. A new service was deployed into our cluster for periodically polling the new RESTlet endpoint (using a cursor in S3) and outputting the log entries to STDOUT. We already have a Splunk forwarder configured to capture and index all output generated by running tasks, so this service was fairly simple.

Here’s a complete and simplified architectural view of the solution:

And here’s an example of how to use the JSON logging shared library:

With this architecture, we are now able to trace requests/transactions from source systems into Netsuite from within Splunk.

As a final note, there are still some small caveats to this approach. Mainly:

  • The timestamp coming from Netsuite’s RESTlets is different than the one which appears as Splunk’s log timestamp. This is because at the moment I am uncertain as to how to force Splunk to read the timestamp from a specific field (if that’s even possible).
  • RESTlets needs to be migrated to use the new JSON Logger library and to be accessed through Netsuite Adapter to take advantage of the integration described above. The former is needed for persisting the logs into the right record and the latter to allow Transaction ID integration.