In my new job I have opportunity to work a little bit more with Erlang code and Erlang related tools than earlier. That forced me a little to learn and use Common Test library which is the testing library in Erlang world. When after that I needed to work back it hit me hard how much stuff I am missing in Elixir. In this article I will try to present Common Test library from the viewpoint of long standing Elixir guy and present how it compares to the ExUnit.

Unit testing vs integration testing#

The main difference between these two is their intended usage. In it's core ExUnit is intended as a library for unit testing (no surprise there, as it is literally in it's name), on the other hand Common Test is more about integration testing and looking on the system as a whole. So the discussion there would not be fair, as it isn't apples to apples, but for second let's check what makes each of them great for their intended uses:

ExUnit for unit testing:

  • Simple way to setup and teardown tests via setup and setup_all macros
  • Built in testing of documentation examples via doctest
  • Immutability of test environment (at least that is what you should try to achieve)
  • Very simple way of running different test modules in parallel (tests within single module are always sync)
  • Each test run in separate process, which prevents sharing data between processes via process dictionary
  • Grouping tests into describe blocks
  • In overall simplicity of the library as a whole

Common Test for integration testing:

  • Persistent test results - logs, results, "generated garbage" is stored between runs in the Common Test, which allows to track what and when happened, and if needed trace the bug in the logs/leftovers
  • Generating "private directory" per suite
  • Grouping tests while allowing single test to be reused in different groups
  • Extensive control about order and parallelization of tests within groups
  • Built in very extensive and detailed reports (HTML and JUnit)
  • Distributed testing
  • Define dependencies between tests
  • Set of helpers for working with different network protocols (SSH, Telnet, SNMP, FTP, Erlang RPC, netconf)
  • Built in support for data fixtures
  • Built in support for complex configuration of test runs via test specifications
  • Built in logging (we will cover that later)

So as you can see, the Common Test is much broader and complex, but at the same time it provides a lot of utilities that are really helpful while writing integration tests, that can span multiple applications.

Writing Common Test suite#

While CT provides enormous power in hand of the experienced tester/developer it's API isn't the top of the class (which happen a lot in Erlang tools, whoever wanted to integrate them with anything then they will know what I mean). But in fact writing simple test suite in CT is fairly easy:

-module(example_SUITE). % The naming convention (with uppercase _SUITE) Erlang
                        % convention which allow ct to find test suites.
                        % Something like ExUnit _test.exs naming convention

-export([all/0]).

-export([test_function_name/1]).

all() ->
    [test_function_name].

test_function_name(_Config) ->
    2 = 1 + 1.

It is pretty easy and understandable format, but with due CT power it quickly can grow to be "less" readable. Taming that power can be problematic and can cause few headaches for newcomers.

Reporting#

Another part, and the one that I miss the most in ExUnit, is reporting in Common Test. This alone is in my humble opinion THE best part of the library. To present you how this work assume that we have above test suite which we will run with ct_run -dir .. This will compile and run our tests (obviously), but in addition to that it will generate hell lot of cruft files, some JQuery (yes, this is still used in 2019), some CSS files, some HTML and other.

Just check this out. This is example report generated by the Common Test. As you can see it contains a lot of information in quite readable format. Not only it contains informations about current run, but all previous runs as well, which is really handy when tracking regressions.

But can we store even more informations there? Yes, as CT includes simple logging facility it is completely possible to log your own informations during tests, for example, lets modify our test to log some informations:

test_function_name(_Config) ->
    ct:log("Example message"),
    2 = 1 + 1.

Now when we run tests again, then we will see more informations (even coloured) in our test log:

Common Test log "Example message" on green background

Additionally there is support Surefire XML output (commonly known as JUnit XML) via hook that is distributed with Common Test. This is very useful, as many CI services (for sure Jenkins, Circle CI, and GitLab CI) support such files to provide better CI results than raw stdout.

Comparison to ExUnit#

ExUnit is great tool, which is very flexible while being relatively simple. Unfortunately that simplicity makes it very lacking when comparing with "more established" solutions like Common Test which shines when we "grow out" of the ExUnit tests. It provides a lot great tools that really help with finding bugs (for example when I was joining the project and I was trying to run tests locally I have some issues due to misconfiguration, it was much easier to send tarball of the test results instead of sending wall of text captured from stdout).

Don't get me wrong, ExUnit is great and powerful, and for sure you should use it. It is great evolution over EUnit, but again, it would be worth of having built in Surefire XML output due to it's popularity and support in many CI services.

Future#

But is everything lost and we need to fall back to the Erlang code (which can be problematic for many people) to use Common Test? Not exactly, Comcast (yes, that Comcast) created simple wrapper for Elixir (unfortunately seems pretty dead to me) and I have started Commoner library that is meant to provide API more familiar and easier to use:

defmodule Commoner.ExampleSuite do
  use CommonTest.Suite

  test "foo bar" do
    assert (1 + 1) in [2]
  end

  test "bar baz" do
    CommonTest.log("Hello")
  end
end

I am very interested about your feelings about Common Test usage in Elixir and about proposed API for Commoner. You can ping me:

∎