[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

3. Tutorial: Basic Unit Testing

This tutorial will use the JUnit Test Infected article as a starting point. We will be creating a library to represent money, libmoney, that allows conversions between different currency types. The development style will be “test a little, code a little”, with unit test writing preceding coding. This constantly gives us insights into module usage, and also makes sure we are constantly thinking about how to test our code.


[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

3.1 How to Write a Test

Test writing using Check is very simple. The file in which the checks are defined must include ‘check.h’ as so:

 
#include <check.h>

The basic unit test looks as follows:

 
START_TEST (test_name)
{
  /* unit test code */
}
END_TEST

The START_TEST/END_TEST pair are macros that setup basic structures to permit testing. It is a mistake to leave off the END_TEST marker; doing so produces all sorts of strange errors when the check is compiled.


[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

3.2 Setting Up the Money Build

Since we are creating a library to handle money, we will first create an interface in ‘money.h’, an implementation in ‘money.c’, and a place to store our unit tests, ‘check_money.c’. We want to integrate these core files into our build system, and will need some additional structure. To manage everything we’ll use Autoconf, Automake, and friends (collectively known as Autotools) for this example. Note that one could do something similar with ordinary Makefiles, or any other builc system. It is in the authors’ opinion that it is generally easier to use Autotools than bare Makefiles, and they provide built-in support for running tests.

Note that this is not the place to explain how Autotools works. If you need help understanding what’s going on beyond the explanations here, the best place to start is probably Alexandre Duret-Lutz’s excellent Autotools tutorial.

The examples in this section are part of the Check distribution; you don’t need to spend time cutting and pasting or (worse) retyping them. Locate the Check documentation on your system and look in the ‘example’ directory. The standard directory for GNU/Linux distributions should be ‘/usr/share/doc/check/example’. This directory contains the final version reached the end of the tutorial. If you want to follow along, create backups of ‘money.h’, ‘money.c’, and ‘check_money.c’, and then delete the originals.

We set up a directory structure as follows:

 
.
|-- Makefile.am
|-- README
|-- configure.ac
|-- src
|   |-- Makefile.am
|   |-- main.c
|   |-- money.c
|   `-- money.h
`-- tests
    |-- Makefile.am
    `-- check_money.c

Note that this is the output of tree, a great directory visualization tool. The top-level ‘Makefile.am’ is simple; it merely tells Automake how to process sub-directories:

 
SUBDIRS = src . tests

Note that tests comes last, because the code should be testing an already compiled library. ‘configure.ac’ is standard Autoconf boilerplate, as specified by the Autotools tutorial and as suggested by autoscan. The AM_PATH_CHECK() is the only line particular to Check see section AM_PATH_CHECK.

src/Makefile.am’ builds ‘libmoney’ as a Libtool archive, and links it to an application simply called main. The application’s behavior is not important to this tutorial; what’s important is that none of the functions we want to unit test appear in ‘main.c’; this probably means that the only function in ‘main.c’ should be main() itself. In order to test the whole application, unit testing is not appropriate: you should use a system testing tool like Autotest. If you really want to test main() using Check, rename it to something like _myproject_main() and write a wrapper around it.

The primary build instructions for our unit tests are in ‘tests/Makefile.am’:

 
## Process this file with automake to produce Makefile.in

TESTS = check_money
check_PROGRAMS = check_money
check_money_SOURCES = check_money.c $(top_builddir)/src/money.h
check_money_CFLAGS = @CHECK_CFLAGS@
check_money_LDADD = $(top_builddir)/src/libmoney.la @CHECK_LIBS@

TESTS tells Automake which test programs to run for make check. Similarly, the check_ prefix in check_PROGRAMS actually comes from Automake; it says to build these programs only when make check is run. (Recall that Automake’s check target is the origin of Check’s name.) The check_money test is a program that we will build from ‘tests/check_money.c’, linking it against both ‘src/libmoney.la’ and the installed ‘libcheck.la’ on our system. The appropriate compiler and linker flags for using Check are found in @CHECK_CFLAGS@ and @CHECK_LIBS@, values defined by the AM_PATH_CHECK macro.

Now that all this infrastructure is out of the way, we can get on with development. ‘src/money.h’ should only contain standard C header boilerplate:

 
#ifndef MONEY_H
#define MONEY_H

#endif /* MONEY_H */

src/money.c’ should be empty, and ‘tests/check_money.c’ should only contain an empty main() function:

 
int
main (void)
{
  return 0;
}

Create the GNU Build System for the project and then build ‘main’ and ‘libmoney.la’ as follows:

 
$ autoreconf --install
$ ./configure
$ make

(autoreconf determines which commands are needed in order for configure to be created or brought up to date. Previously one would use a script called autogen.sh or bootstrap, but that practice is unnecessary now.)

Now build and run the check_money test with make check. If all goes well, make should report that our tests passed. No surprise, because there aren’t any tests to fail. If you have problems, make sure to see AM_PATH_CHECK.

This was tested on the isadora distribution of Linux Mint GNU/Linux in November 2012, using Autoconf 2.65, Automake 1.11.1, and Libtool 2.2.6b. Please report any problems to check-devel AT lists.sourceforge.net.


[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

3.3 Test a Little, Code a Little

The Test Infected article starts out with a Money class, and so will we. Of course, we can’t do classes with C, but we don’t really need to. The Test Infected approach to writing code says that we should write the unit test before we write the code, and in this case, we will be even more dogmatic and doctrinaire than the authors of Test Infected (who clearly don’t really get this stuff, only being some of the originators of the Patterns approach to software development and OO design).

Here are the changes to ‘check_money.c’ for our first unit test:

 
--- tests/check_money.1.c	2013-09-20 23:37:40.000000000 -0400
+++ tests/check_money.2.c	2013-09-20 23:37:40.000000000 -0400
@@ -1,5 +1,18 @@
+#include <check.h>
+#include "../src/money.h"
+
+START_TEST (test_money_create)
+{
+  Money *m;
+  m = money_create (5, "USD");
+  ck_assert_int_eq (money_amount (m), 5);
+  ck_assert_str_eq (money_currency (m), "USD");
+  money_free (m);
+}
+END_TEST
+
 int
 main (void)
 {
   return 0;
 }

A unit test should just chug along and complete. If it exits early, or is signaled, it will fail with a generic error message. (Note: it is conceivable that you expect an early exit, or a signal and there is functionality in Check to specifically assert that we should expect a signal or an early exit.) If we want to get some information about what failed, we need to use some calls that will point out a failure. Two such calls are ck_assert_int_eq (used to determine if two integers are equal) and ck_assert_str_eq (used to determine if two null terminated strings are equal). Both of these functions (actually macros) will signal an error if their arguments are not equal.

An alternative to using ck_assert_int_eq and ck_assert_str_eq is to write the expression under test directly using ck_assert. This takes one Boolean argument which must be True for the check to pass. The second test could be rewritten as follows:

 
ck_assert(strcmp (money_currency (m), "USD") == 0);

ck_assert will find and report failures, but will not print any user supplied message in the unit test result. To print a user defined message along with any failures found, use ck_assert_msg. The first argument is a Boolean argument. The remaining arguments support varargs and accept printf-style format strings and arguments. This is especially useful while debugging. For example, the second test could be rewritten as:

 
ck_assert_msg(strcmp (money_currency (m), "USD") == 0,
         "Was expecting a currency of USD, but found %s", money_currency (m));

If the Boolean argument is too complicated to elegantly express within ck_assert(), there are the alternate functions ck_abort() and ck_abort_msg() that unconditionally fail. The second test inside test_money_create above could be rewritten as follows:

 
if (strcmp (money_currency (m), "USD") != 0) 
  {
    ck_abort_msg ("Currency not set correctly on creation");
  }

For your convenience ck_assert, which does not accept a user supplied message, substitutes a suitable message for you. (This is also equivalent to passing a NULL message to ck_assert_msg). So you could also write a test as follows:

 
ck_assert (money_amount (m) == 5);

This is equivalent to:

 
ck_assert_msg (money_amount (m) == 5, NULL);

which will print the file, line number, and the message "Assertion 'money_amount (m) == 5' failed" if money_amount (m) != 5.

When we try to compile and run the test suite now using make check, we get a whole host of compilation errors. It may seem a bit strange to deliberately write code that won’t compile, but notice what we are doing: in creating the unit test, we are also defining requirements for the money interface. Compilation errors are, in a way, unit test failures of their own, telling us that the implementation does not match the specification. If all we do is edit the sources so that the unit test compiles, we are actually making progress, guided by the unit tests, so that’s what we will now do.

We will patch our header ‘money.h’ as follows:

 
--- src/money.1.h	2013-09-20 23:37:40.000000000 -0400
+++ src/money.2.h	2013-09-20 23:37:40.000000000 -0400
@@ -1,4 +1,11 @@
 #ifndef MONEY_H
 #define MONEY_H
 
+typedef struct Money Money;
+
+Money *money_create (int amount, char *currency);
+int money_amount (Money * m);
+char *money_currency (Money * m);
+void money_free (Money * m);
+
 #endif /* MONEY_H */

Our code compiles now, and again passes all of the tests. However, once we try to use the functions in libmoney in the main() of check_money, we’ll run into more problems, as they haven’t actually been implemented yet.


[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

3.4 Creating a Suite

To run unit tests with Check, we must create some test cases, aggregate them into a suite, and run them with a suite runner. That’s a bit of overhead, but it is mostly one-off. Here’s a diff for the new version of ‘check_money.c’. Note that we include stdlib.h to get the definitions of EXIT_SUCCESS and EXIT_FAILURE.

 
--- tests/check_money.2.c	2013-09-20 23:37:40.000000000 -0400
+++ tests/check_money.3.c	2013-09-20 23:37:40.000000000 -0400
@@ -1,18 +1,38 @@
+#include <stdlib.h>
 #include <check.h>
 #include "../src/money.h"
 
 START_TEST (test_money_create)
 {
   Money *m;
   m = money_create (5, "USD");
   ck_assert_int_eq (money_amount (m), 5);
   ck_assert_str_eq (money_currency (m), "USD");
   money_free (m);
 }
 END_TEST
 
+Suite *
+money_suite (void)
+{
+  Suite *s = suite_create ("Money");
+
+  /* Core test case */
+  TCase *tc_core = tcase_create ("Core");
+  tcase_add_test (tc_core, test_money_create);
+  suite_add_tcase (s, tc_core);
+
+  return s;
+}
+
 int
 main (void)
 {
-  return 0;
+  int number_failed;
+  Suite *s = money_suite ();
+  SRunner *sr = srunner_create (s);
+  srunner_run_all (sr, CK_NORMAL);
+  number_failed = srunner_ntests_failed (sr);
+  srunner_free (sr);
+  return (number_failed == 0) ? EXIT_SUCCESS : EXIT_FAILURE;
 }

Most of the money_suite() code should be self-explanatory. We are creating a suite, creating a test case, adding the test case to the suite, and adding the unit test we created above to the test case. Why separate this off into a separate function, rather than inline it in main()? Because any new tests will get added in money_suite(), but nothing will need to change in main() for the rest of this example, so main will stay relatively clean and simple.

Unit tests are internally defined as static functions. This means that the code to add unit tests to test cases must be in the same compilation unit as the unit tests themselves. This provides another reason to put the creation of the test suite in a separate function: you may later want to keep one source file per suite; defining a uniquely named suite creation function allows you later to define a header file giving prototypes for all the suite creation functions, and encapsulate the details of where and how unit tests are defined behind those functions. See the test program defined for Check itself for an example of this strategy.

The code in main() bears some explanation. We are creating a suite runner object of type SRunner from the Suite we created in money_suite(). We then run the suite, using the CK_NORMAL flag to specify that we should print a summary of the run, and list any failures that may have occurred. We capture the number of failures that occurred during the run, and use that to decide how to return. The check target created by Automake uses the return value to decide whether the tests passed or failed.

Now that the tests are actually being run by check_money, we encounter linker errors again we try out make check. Try it for yourself and see. The reason is that the ‘money.c’ implementation of the ‘money.h’ interface hasn’t been created yet. Let’s go with the fastest solution possible and implement stubs for each of the functions in money.c. Here is the diff:

 
--- src/money.1.c	2013-09-20 23:37:40.000000000 -0400
+++ src/money.3.c	2013-09-20 23:37:40.000000000 -0400
@@ -0,0 +1,26 @@
+#include <stdlib.h>
+#include "money.h"
+
+Money *
+money_create (int amount, char *currency)
+{
+  return NULL;
+}
+
+int
+money_amount (Money * m)
+{
+  return 0;
+}
+
+char *
+money_currency (Money * m)
+{
+  return NULL;
+}
+
+void
+money_free (Money * m)
+{
+  return;
+}

Note that we #include <stdlib.h> to get the definition of NULL. Now, the code compiles and links when we run make check, but our unit test fails. Still, this is progress, and we can focus on making the test pass.


[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

3.5 SRunner Output

The functions to run tests in an SRunner are defined as follows:

 
void srunner_run_all (SRunner * sr, enum print_output print_mode);

void srunner_run (SRunner *sr, const char *sname, const char *tcname,
                  enum print_output print_mode);

Those functions do two things:

  1. They run all of the unit tests for the selected test cases defined for the selected suites in the SRunner, and collect the results in the SRunner. The determination of the selected test cases and suites depends on the specific function used.

    srunner_run_all will run all the defined test cases of all defined suites except if the environment variables CK_RUN_CASE or CK_RUN_SUITE are defined. If defined, those variables shall contain the name of a test suite or a test case, defining in that way the selected suite/test case.

    srunner_run will run the suite/case selected by the sname and tcname parameters. A value of NULL in some of those parameters means “any suite/case”.

  2. They print the results according to the print_mode specified.

For SRunners that have already been run, there is also a separate printing function defined as follows:

 
void srunner_print (SRunner *sr, enum print_output print_mode);

The enumeration values of print_output defined in Check that parameter print_mode can assume are as follows:

CK_SILENT

Specifies that no output is to be generated. If you use this flag, you either need to programmatically examine the SRunner object, print separately, or use test logging (see section Test Logging.)

CK_MINIMAL

Only a summary of the test run will be printed (number run, passed, failed, errors).

CK_NORMAL

Prints the summary of the run, and prints one message per failed test.

CK_VERBOSE

Prints the summary, and one message per test (passed or failed)

CK_ENV

Gets the print mode from the environment variable CK_VERBOSITY, which can have the values "silent", "minimal", "normal", "verbose". If the variable is not found or the value is not recognized, the print mode is set to CK_NORMAL.

CK_SUBUNIT

Prints running progress through the subunit test runner protocol. See ’subunit support’ under the Advanced Features section for more information.

With the CK_NORMAL flag specified in our main(), let’s rerun make check now. As before, we get the following satisfying output:

 
Running suite(s): Money
0%: Checks: 1, Failures: 1, Errors: 0
check_money.c:9:F:Core:test_money_create:0: Assertion 'money_amount (m)==5' failed: 
money_amount (m)==0, 5==5
FAIL: check_money
=====================================================
1 of 1 test failed
Please report to check-devel AT lists.sourceforge.net
=====================================================

The first number in the summary line tells us that 0% of our tests passed, and the rest of the line tells us that there was one check in total, and of those checks, one failure and zero errors. The next line tells us exactly where that failure occurred, and what kind of failure it was (P for pass, F for failure, E for error).

After that we have some higher level output generated by Automake: the check_money program failed, and the bug-report address given in ‘configure.ac’ is printed.

Let’s implement the money_amount function, so that it will pass its tests. We first have to create a Money structure to hold the amount, and then implement the function to return the correct amount:

 
--- src/money.3.c	2013-09-20 23:37:40.000000000 -0400
+++ src/money.4.c	2013-09-20 23:37:40.000000000 -0400
@@ -1,26 +1,31 @@
 #include <stdlib.h>
 #include "money.h"
 
+struct Money
+{
+  int amount;
+};
+
 Money *
 money_create (int amount, char *currency)
 {
   return NULL;
 }
 
 int
 money_amount (Money * m)
 {
-  return 0;
+  return m->amount;
 }
 
 char *
 money_currency (Money * m)
 {
   return NULL;
 }
 
 void
 money_free (Money * m)
 {
   return;
 }

We will now rerun make check and… what’s this? The output is now as follows:

 
Running suite(s): Money
0%: Checks: 1, Failures: 0, Errors: 1
check_money.c:5:E:Core:test_money_create:0: (after this point) 
Received signal 11 (Segmentation fault)

What does this mean? Note that we now have an error, rather than a failure. This means that our unit test either exited early, or was signaled. Next note that the failure message says “after this point”; This means that somewhere after the point noted (‘check_money.c’, line 5) there was a problem: signal 11 (a.k.a. segmentation fault). The last point reached is set on entry to the unit test, and after every call to the ck_assert(), ck_abort(), ck_assert_int_*(), ck_assert_str_*(), or the special function mark_point(). For example, if we wrote some test code as follows:

 
stuff_that_works ();
mark_point ();
stuff_that_dies ();

then the point returned will be that marked by mark_point().

The reason our test failed so horribly is that we haven’t implemented money_create() to create any Money. We’ll go ahead and implement that, the symmetric money_free(), and money_currency() too, in order to make our unit test pass again, here is a diff:

 
--- src/money.4.c	2013-09-20 23:37:40.000000000 -0400
+++ src/money.5.c	2013-09-20 23:37:40.000000000 -0400
@@ -1,31 +1,41 @@
 #include <stdlib.h>
 #include "money.h"
 
 struct Money
 {
   int amount;
+  char *currency;
 };
 
 Money *
 money_create (int amount, char *currency)
 {
-  return NULL;
+  Money *m = malloc (sizeof (Money));
+  if (m == NULL)
+    {
+      return NULL;
+    }
+
+  m->amount = amount;
+  m->currency = currency;
+  return m;
 }
 
 int
 money_amount (Money * m)
 {
   return m->amount;
 }
 
 char *
 money_currency (Money * m)
 {
-  return NULL;
+  return m->currency;
 }
 
 void
 money_free (Money * m)
 {
+  free (m);
   return;
 }

[ << ] [ >> ]           [Top] [Contents] [Index] [ ? ]

This document was generated by brarcher on January 20, 2014 using texi2html 1.82.