Testing interactions with web services without integration tests in Ruby

Our team decided to move to a micro-service architecture, and we started wondering how we would test all of our integration points with lots of little services without having to rely on integration tests. We felt that testing the interactions between these services quickly become a major headache.

Integration tests typically are slow and brittle, requiring each component to have its own environment to run the tests in. With a micro-service architecture, this becomes even more of a problem. They also have to be ‘all-knowing’ and this makes them difficult to keep from being fragile.

After seeing J. B. Rainsbergers talk “Integrated Tests Are A Scam” we have been thinking on how to get the confidence we need to deploy our software to production without having a tiresome integration test suite that does not give us all the coverage we think it does.

Pact is a ruby gem that allows you to define a pact between service consumers and providers. It provides a DSL for service consumers to define the request they will make to a service producer and the response they expect back. This expectation is used in the consumer specs to provide a mock producer, and is also played back in the producer specs to ensure the producer actually does provide the response the consumer expects.

This allows you to test both sides of an integration point using fast unit-style tests.

Example Pact use

Source for this example can be found here: https://github.com/uglyog/example_pact

Given we have a client that needs to make a HTTP GET request to a sinatra webapp, and requires a response in JSON format. The client would look something like:

require 'httparty'
require 'uri'
require 'json'

class Client

  def load_producer_json
    response = HTTParty.get(URI::encode('http://localhost:8081/producer.json?valid_date=' + Time.now.httpdate))
    if response.success?
      JSON.parse(response.body)
    end
  end

end

and the provider:

require 'sinatra/base'
require 'json'

class Producer < Sinatra::Base   get '/producer.json', :provides => 'json' do
    valid_time = Time.parse(params[:valid_date])
    JSON.pretty_generate({
      :test => 'NO',
      :valid_date => DateTime.now,
      :count => 1000
    })
  end

end

This provider expects a valid_date parameter in HTTP date format, and then returns some simple JSON back.

Running the client with the following rake task against the producer works nicely:

desc 'Run the client'
task :run_client => :init do
  require 'client'
  require 'ap'
  ap Client.new.load_producer_json
end
$ rake run_client
http://localhost:8081/producer.json?valid_date=Thu,%2015%20Aug%202013%2003:15:15%20GMT
{
          "test" => "NO",
    "valid_date" => "2013-08-15T13:31:39+10:00",
         "count" => 1000
}

Now lets get the client to actually use the data it gets back from the provider. Here is the updated client method that uses the returned data:

  def process_data
    data = load_producer_json
    ap data
    value = data['count'] / 100
    date = Time.parse(data['date'])
    puts value
    puts date
    [value, date]
  end

This divides the returned count by 100 and parses the returned date with the ruby Time class. Doesn’t do much, but enough that a change in the JSON will cause it to fail.

Let’s add a spec to test this client:

require 'spec_helper'
require 'client'

describe Client do

  let(:json_data) do
    {
      "test" => "NO",
      "date" => "2013-08-16T15:31:20+10:00",
      "count" => 100
    }
  end
  let(:response) { double('Response', :success? => true, :body => json_data.to_json) }

  it 'can process the json payload from the producer' do
    HTTParty.stub(:get).and_return(response)
    expect(subject.process_data).to eql([1, Time.parse(json_data['date'])])
  end

end

Now we can run this spec and see it pass:

$ rake spec
/Users/ronald/.rvm/rubies/ruby-1.9.3-p448/bin/ruby -S rspec ./spec/client_spec.rb

Client
http://localhost:8081/producer.json?valid_date=Fri,%2016%20Aug%202013%2005:44:41%20GMT
{
     "test" => "NO",
     "date" => "2013-08-16T15:31:20+10:00",
    "count" => 100
}
1
2013-08-16 15:31:20 +1000
  can process the json payload from the producer

Finished in 0.00409 seconds
1 example, 0 failures

Yay, all good because everything passed. Well, not quite. There is a problem with this integration point. The provider returns a ‘valid_date’ while the consumer is trying to use ‘date’, which will blow up when run for real even with the tests all passing. In the test the response was stubbed so there is no easy way to keep it in sync with with what the actual provider is returning. Here is where Pact comes in.

Pact to the rescue

Lets setup Pact in the consumer. Pact lets the consumers define the expectations for the integration point. We will add some pact config to the spec helper that defines a consumer (called “My Consumer”) and a provider (“My Producer”) that runs on port 8081.

require 'ap'

require 'pact'
require 'pact/consumer/rspec'

$:.unshift 'lib'

Pact.service_consumer 'My Consumer' do

  has_pact_with "My Producer" do
    mock_service :my_provider do
      port 8081
    end
  end

end

We can now add a pact section to the spec for the client.

  describe 'pact with producer', :pact => true do

    let(:date) { Time.now.httpdate }

    before do
      my_producer.
        given("producer is in a sane state").
          upon_receiving("a request for producer json").
            with({
                method: :get,
                path: '/producer.json',
                query: URI::encode('valid_date=' + date)
            }).
            will_respond_with({
              status: 200,
              headers: { 'Content-Type' => 'application/json' },
              body: json_data
            })
    end

    it 'can process the json payload from the producer' do
      expect(subject.process_data).to eql([1, Time.parse(json_data['date'])])
    end

  end

This spec defines an expectation that the provider will receive the request for it’s JSON and provides the same response as the earlier spec. The main difference is that Pact creates a mock service running on the correct port that will provide this response, instead of stubbing the HTTP client.

Running this spec tests the consumer behaviour against the mock service. It also generates a ‘pact file’ as output, which can be used to validate our assumptions on the provider side.

$ rake spec
/Users/ronald/.rvm/rubies/ruby-1.9.3-p448/bin/ruby -S rspec ./spec/client_spec.rb

Client
http://localhost:8081/producer.json?valid_date=Fri,%2016%20Aug%202013%2006:09:44%20GMT
{
  "test"  => "NO",
  "date"  => "2013-08-16T15:31:20+10:00",
  "count" => 100
}
1
2013-08-16 15:31:20 +1000
  can process the json payload from the producer
  pact with producer
http://localhost:8081/producer.json?valid_date=Fri,%2016%20Aug%202013%2006:09:44%20GMT
{
  "test"  => "NO",
  "date"  => "2013-08-16T15:31:20+10:00",
  "count" => 100
}
1
2013-08-16 15:31:20 +1000
    can process the json payload from the producer

Generated pact file (spec/pacts/my_consumer-my_producer.json):

{
  "producer": {
    "name": "My Producer"
  },
  "consumer": {
    "name": "My Consumer"
  },
  "interactions": [
    {
      "description": "a request for producer json",
      "request": {
        "method": "get",
        "path": "/producer.json",
        "query": "valid_date=Fri,%2016%20Aug%202013%2006:09:44%20GMT"
      },
      "response": {
        "status": 200,
        "headers": {
          "Content-Type": "application/json"
        },
        "body": {
          "test": "NO",
          "date": "2013-08-16T15:31:20+10:00",
          "count": 100
        }
      },
      "producer_state": "producer is in a sane state"
    }
  ],
  "metadata": {
    "date": "2013-08-16T16:09:44+10:00",
    "pact_gem": {
      "version": "0.1.23"
    }
  }
}

Provider Setup

Now lets setup the tests for our provider. Pact has a rake task to verify the provider against the generated pact file. It can get the pact file from any URL (like the last successful CI build), but we are just going to use the local one. Here is what we add to the provider Rakefile:

require 'pact'
require 'pact/verification_task'

Pact::VerificationTask.new(:local) do | pact |
  pact.uri 'spec/pacts/my_consumer-my_producer.json', support_file: './spec/pacts/pact_helper'
end

The pact_helper (spec/pacts/pact_helper.rb) needs to tell Pact which Rack app it needs to test against so we edit it as follows:

Pact.provider_states_for 'My Consumer' do
  provider_state "producer is in a sane state" do
    set_up do
      # Create a thing here using your factory of choice
    end
  end
end

require 'pact/provider/rspec'
Pact.service_provider "My Producer" do
  app { Producer.new }
end

Now if we run our pact verification task, it should fail.

$ rake pact:verify:local

Pact in spec/pacts/my_consumer-my_producer.json
  Given producer is in a sane state
    a request for producer json to /producer.json
      returns a response which
        has status code 200
        has a matching body (FAILED - 1)
        includes headers
          "Content-Type" with value "application/json" (FAILED - 2)

Failures:

  1) Pact in spec/pacts/my_consumer-my_producer.json Given producer is in a sane state a request for producer json to /producer.json returns a response which has a matching body
     Failure/Error: expect(parse_entity_from_response(last_response)).to match_term response['body']
       {
         "date"  => {
           :expected => "2013-08-16T15:31:20+10:00",
           :actual   => nil
         },
         "count" => {
           :expected => 100,
           :actual   => 1000
         }
       }
.
.
.

This failed because the expectation stated that the provider should return a value for the date, but did not return anything (it returned the value as expected_date). Also, the count was expected to be 100 but the provider actually returned 1000 (this is an error with the expectation).

Looks like we need to update the producer to return ‘date’ instead of ‘valid_date’, and we also need to update the client expectation to return 1000 for the count and the correct content type (we expected "application/json" but actually got "application/json;charset=utf-8").

Here is the updated consumer spec with the corrections:

require 'spec_helper'
require 'client'

describe Client do

  let(:json_data) do
    {
      "test" => "NO",
      "date" => "2013-08-16T15:31:20+10:00",
      "count" => 1000
    }
  end
  let(:response) { double('Response', :success? => true, :body => json_data.to_json) }

  it 'can process the json payload from the producer' do
    HTTParty.stub(:get).and_return(response)
    expect(subject.process_data).to eql([10, Time.parse(json_data['date'])])
  end

  describe 'pact with producer', :pact => true do

    let(:date) { Time.now.httpdate }

    before do
      my_producer.
        given("producer is in a sane state").
          upon_receiving("a request for producer json").
            with({
                method: :get,
                path: '/producer.json',
                query: URI::encode('valid_date=' + date)
            }).
            will_respond_with({
              status: 200,
              headers: { 'Content-Type' => 'application/json;charset=utf-8' },
              body: json_data
            })
    end

    it 'can process the json payload from the producer' do
      expect(subject.process_data).to eql([10, Time.parse(json_data['date'])])
    end

  end

end

and the corrected provider:

require 'sinatra/base'
require 'json'

class Producer  'json' do
    valid_time = Time.parse(params[:valid_date])
    JSON.pretty_generate({
      :test => 'NO',
      :date => "2013-08-16T15:31:20+10:00",
      :count => 1000
    })
  end

end

Running the client spec to regenerate the pact file followed by the pact verify task shows that the consumer and provider are correctly integrated.

$ rake spec
/Users/ronald/.rvm/rubies/ruby-1.9.3-p448/bin/ruby -S rspec ./spec/client_spec.rb

Client
http://localhost:8081/producer.json?valid_date=Sat,%2014%20Sep%202013%2003:57:02%20GMT
{
  "test"  => "NO",
  "date"  => "2013-08-16T15:31:20+10:00",
  "count" => 1000
}
10
2013-08-16 15:31:20 +1000
  can process the json payload from the producer
  pact with producer
http://localhost:8081/producer.json?valid_date=Sat,%2014%20Sep%202013%2003:57:02%20GMT
{
  "test"  => "NO",
  "date"  => "2013-08-16T15:31:20+10:00",
  "count" => 1000
}
10
2013-08-16 15:31:20 +1000
    can process the json payload from the producer

Finished in 0.1297 seconds
2 examples, 0 failures
$ rake pact:verify:local

Pact in spec/pacts/my_consumer-my_producer.json
  Given producer is in a sane state
    a request for producer json to /producer.json
      returns a response which
        has status code 200
        has a matching body
        includes headers
          "Content-Type" with value "application/json;charset=utf-8"

Finished in 0.04557 seconds
3 examples, 0 failures

We now have fast unit-like tests on each side of the integration point instead of tedious integration tests.