Testing the behavior

Posted on August 13, 2011

All planned features for libinotify-kqueue are implemented, and now I am working on tests.

Tests are especially important for a library that emulates an existing behavior. So I have decided to cover the library with behavioral tests, not with the unit tests.

How to test the behaviour of such an entity like inotify? To answer this question, we have to figure out what it provides at first.

So, with inotify we can:

  1. Start a watch;
  2. Modify a watch;
  3. Register notifications from watch(es);
  4. Remove a watch.

Great. But if we want to register notifications from the file system, we should produce the activity on the file system at first. So we need to have two entities: one produces activity (a producer) and one registers it (a consumer).

Next, we have to ensure that the registered events are the same events that we have expected. So the scheme how the producer and the consumer communicate might look like the following:

  1. Producer tells to consumer, which events are expected to appear;
  2. Consumer starts registering events;
  3. Producer starts generating a file system activity;
  4. Consumer register all the expected events OR reaches the timeout;
  5. Consumer specifies to producer, which events have left unregistered;
  6. Producer processes this info and passes or failes the appropriate test case.

The communication between the producer and consumer can be also generalized like this:

  1. Producer tells to consumer, what to do;
  2. Consumer executing the action;
  3. Producer waits while the consumer works (producer can do its own work here too);
  4. Consumer tells the results to the producer;
  5. Producer passes or failes the test case.

Besides monitoring for expected events, action is also can be:

  • Starting a watch or updating flags of an existing watch;
  • Stopping a watch.

These actions give us the additional functionality, quite suitable for covering sophicated scenarions.

The test suite have to run on both Linux and NetBSD. All tests have to pass on Linux, some of them are expected to fail on NetBSD. I will write some tests knowing that they will fail on NetBSD intentionally.

The test suite should also have entities for logging and tracking passed and failed tests.

Considering all these requirements, I have developed my own tiny BDD framework. Its last version can be viewed here (and then, the more recent version will be available in the master branch).

It was an interesting experience to make it run on both Linux and NetBSD. I have represented both consumer and producer by threads (probably it is possible to run them in a single thread, but in my vision it would require to build an FSM and with the FSM it would be harder to understand the flow of tests), and the hardest part (as always) were in its synchronization. Pthread barriers fit perferctly here, and it worked fine on Linux but didn’t work well on NetBSD. So I had to implement my own primitives atop of mutexes and condition variables for it.

Currently I am covering inotify with the tests on Linux. There are 2 tests with 16 test cases in total now, but it is just a beggining and I expect to get 50-70 test cases in total. Anyway, the first test already crashes my library to death :) But I will resolve all these issues only after writing all the tests.

Test suite running on Debian GNU/Linux, larger version , 10K