by Adam Brett

Spies, Integrations, and Contracts

At a recent guild meeting with some of my colleagues, we had an interesting discussion on the use of spies. After some wrangling over what a spy is, and the different types of test double in general, the discussion turned to where and how people were using them. One of the developers raised the issue of testing expressjs middleware. Specifically using a spy to ensure that next() is called so that the middleware under test calls the next middleware in the chain.

At first glance, this seems like a reasonable approach, your middleware has logic, therefore it needs a unit test, but good test practice is to test behaviour, and not implementation. This means we should test that the middleware behaves as we expect it to, not that it's implemented how we think it is. I've written about my dislike of spies like this before. Testing the implementation has many drawbacks, not least of which is preventing safe refactoring, because if you change the implementation, but not the behaviour, your tests will break. Here's how the developer suggested we test this:

it('should call next to run the next middleware', function () {
  let sut = require('middleware/foo');
  let next = sinon.spy();

  sut({}, {}, next);

  next.called.should.equal(true);
});

Here's the problem: If express is updated so that calling next() is no-longer required, and therefore is no-longer passed to the middleware, this test will still pass, even though our application will likely break. This is of course, a facetious example, but the concepts remain valid. If we were to refactor our middleware to use our new way of running middleware, this test would break, even though the middleware functions perfectly. Tests like this prevent us from refactoring the way that middleware is triggered without changing tests too.

Changing tests when you refactor implementation, without changing behaviour, is extremely dangerous. If you change code and tests at the same time you lose the confidence that the behaviour of your application hasn't changed.

Integration

The simplest way to overcome this is by not writing a unit test for your middleware, and instead write an integration test. A middlware cannot function on its own, only in the context of an express application, so test the behaviour in the context of an express application.

Here's an example of our middleware:

function foo(req, res, next) {
  if (req.url == '/protected') {
    res.redirect('/login');
  }

  next();
}

Now to write an integration test for this:

describe('middleware/foo', function () {
  const express = require('express');
  const request = require('supertest');
  const middleware = require('middleware/foo');

  it('should redirect when the page is /protected', function (done) {
    let app = express();
    app.use(middleware);

    request(app)
      .get('/protected')
      .expect(302)
      .expect('Location', '/login')
      .end(done);
  });
});

This should work for testing that our middleware behaves as we expect, but it doesn't check that it calls the next middleware in the chain, so now we need another test for that.

it('should call the next middleware in the chain', function (done) {
  let app = express();
  var called = false;

  app.use(middleware);
  app.use(function (req, res, next) {
    called = true;
  });

  request(app)
    .get('/protected')
    .expect(302)
    .expect('Location', '/login')
    .end(function () {
      called.should.equal(true);
      done();
    });
});

This kind of works for us, but seems quite verbose, I wouldn't like to do this a lot, and although this is slightly better than the original (as it doesn't use any doubles) it's still not as good as it could be. Instead, let's try testing that the next middleware in our real app is called instead, because that's what we really want to happen, and the behaviour we really want to test.

Integrated

Building on our tests above, an integrated test is a type of integration test that calls the entire app stack, rather than testing the combination of two small parts of it. Technically, the above tests are also integrated tests. They just test a dummy application, instead of your real one. In the PHP world, you've probably written integrated tests if you've used your frameworks own TestCase class, instead of PHPUnit\Framework\TestCase directly. It's basically a cut down version of your full stack that runs in memory, instead of spinning up real webservers, browsers, and databases like your end-to-end tests would.

Here's the next middleware in the chain:

function bar(req, res, next) {
  res.setHeader('X-Bar', 'baz');
  next();
}

Now using it in our actual application:

const express = require('express');
const app = require('app');

app.use(require('middleware/foo'));
app.use(require('middleware/bar'));

And to test:

describe('app', function () {
  const request = require('supertest');
  const app = require('app');

  it('should redirect to /login when the page is /protected', function (done) {
    request(app)
      .get('/protected')
      .expect(302)
      .expect('Location', '/login')
      .end(done);
  });

  it('should set an X-Bar header with value baz', function () {
    request(app)
      .get('/hello')
      .expect('X-Bar', 'baz')
      .end(done);
  });
});

We're now testing that our application behaves as we expect it to, without making any assertions on implementation. If our application were this simple, we could probably stop there without ever running in to any problems.

But there is a problem with this, and it's called the Integrated Test Fallacy.

Integrated Test Fallacy

There are two major problems with integrated tests. The first, is that they're slow, and the second, is that you can never write enough to cover your code adequately.

As a result, you end up writing just enough to cover your primary paths, and some errors. Then whenever a customer picks up a bug, you write more tests to cover that bug, but you can never cover everything.

Whilst this is perfectly fine for integrated tests that are used as Customer Tests (which, like our end-to-end tests, check the presence of functionality), it is not a good thing when we use them as Programmer Tests to prove the correctness of our application.

To demonstrate this with our example from above, we're not testing what happens if we refactor our application to set the header before doing the redirect, it would look like this:

const express = require('express');
const app = require('app');

app.use(require('middleware/bar'));
app.use(require('middleware/foo'));

Such a simple change, but our tests don't ensure that bar calls next, only that foo does. This means we need another set of integrated tests to make sure that we can reverse the order. We can't just import our app for that one either, because that would mean changing the behaviour of our actual application. Instead, we'll have to bootstrap it all in the test.

Now multiply that by 5, 10, 15 middlewares, and how many routes?

You will never write enough integrated tests for this to work.

Collaborators & Contracts

Now we're back to the beginning. If we were to simply use a spy in a unit test, we could have avoided all of this! But we still haven't solved the refactoring problem.

As with integrated tests, there is another type of integration test, a collaborator test, and is what my colleague was originally calling a unit test (it's not). This is more like the true definition of an integration test. A collaborator test is an integration test that uses a test double to simulate the behaviour, just like in our first example.

it('should call next to run the next middleware', function () {
  let sut = require('middleware/foo');
  let next = sinon.spy();

  sut({}, {}, next);

  next.called.should.equal(true);
});

Here's the secret, though: For every double that you put in a collaborator test, you have to write another type of unit test, a contract test, that ensures the behaviour of your double is checked against the real-life implementation.

The benefit of this is that if you don't know how to write a unit test for the collaborator you've doubled, you don't understand its interface well enough to be writing a double for it. You need to go away and learn it better.

By differentiating your tests this way, and sticking to the rule of creating a contract test for every double in your collaborator tests, you get to negate many of the negative aspect of using doubles.

If you refactor your original code (for example, no longer requiring the call to next in middleware), you would expect your contract tests to fail, but not any other type of test.

Once you refactor your contract tests, that will force you to refactor your doubles as well, and when you do that you would expect your collaborator tests to still pass, but you can be confident that all of your doubles are up-to-date with the real-world implementation and you haven't had a change in behaviour. For our example, it would look something like this:

describe('express/middleware.next', fuction () {
  const request = require('supertest');
  const express = require('express');

  it('should call next()', function (done) {
    const app = express();

    var calls = 0;

    app.use(function (req, res, next) {
      calls++;
      next();
    });

    app.use(function (req, res, next) {
      calls++
      next();
    });

    request(app)
      .get('/')
      .expect(200)
      .end(function () {
        calls.should.equal(2);
        done();
      });
  });
});

Whilst the example of next() in express is a trivial one, it applies to all components of our application that integrate with others, and if this test ever starts to fail, we know we need to update our doubles in our collaborators.

Conclusion

I think this quote from J. B. Rainsberger sums it up better than I can:

Strong integration tests, consisting of collaboration tests (clients using test doubles in place of collaborating services) and contract tests (showing that service implementations correctly behave the way clients expect) can provide the same level of confidence as integrated tests at a lower total cost of maintenance.

For exclusive content, including screen-casts, videos, and early beta access to my projects, subscribe to my email list below.


I love discussion, but not blog comments. If you want to comment on what's written above, head over to twitter.