When an assert isn't assertive enough (or using the wrong macro in an EUnit test)

Full article

I stumbled on an unexpected surprise in Erlang today, while I was cleaning up some EUnit tests. Not a tax refund surprise or a surprise birthday though... more like a bag full of crap on your front doorstep surprise. πŸ’©πŸ’©πŸ’©

I found a test that passed even though it was completely and obviously wrong and should've failed by any reasonable expectation. It took me a few moments to realize what was wrong with it. First though, let me backup a few steps.

Flavors of EUnit

When it comes to EUnit tests, there are a few different ways you can set things up.

  • Test functions: The name ends in _test() and the test macro is something like ?assert(), ?assertEqual(), ?assertMatch(), etc.

  • Test generator functions: The name ends in _test_() and the test generator macro is something like ?_assert(), ?_assertEqual(), etc. The macro with the underscore is effectively a function that calls the previous macros like this, which are then executed.
    fun() -> ?assertEqual(ExpectedValue, ActualValue) end.
    If a list of test generators are specified, then each is run separately, and any failing tests are reported separately.

Lies.. all lies!

So, back to the problem I noticed. In the following example, the test with the PASS.. WTF? comment after it passed when it obviously shouldn't... splitting "Grant Winney" does not result in ["Um", "What?"]. In a production codebase, this is really bad and unexpected behavior. A simple typo or copy/paste error results in tests that should fail just.. passing. Sometimes I hate Erlang. So what's happening?


-spec split_name(string()) -> [string()].
split_name(FullName) ->
    string:split(FullName, " ", all).


% Test Function - should pass
split_name_test() ->
    ?assertEqual(["Grant", "Winney"], split_name("Grant Winney")).         % PASS

% Test Generator Function with a single Test Generator - should pass 
split_name_simple_generator_test_() ->
    ?_assertEqual(["Grant", "Winney"], split_name("Grant Winney")).        % PASS

% Test Generator Function with a set of Test Generators - should pass
split_name_complex_generator_test_() ->
        ?_assertEqual(["Kenny", "Chesney"], split_name("Kenny Chesney")),  % PASS
        ?_assertEqual(["Bob", "Dilan"], split_name("Bob Dilan")),          % PASS
        ?_assertEqual([""], split_name(""))                                % PASS

% Test Function with wrong macro - should fail (doesn't)
split_name_invalid_test() ->
    ?_assertEqual(["Um", "What?"], split_name("Grant Winney")).            % PASS.. WTF?

% Test Generator Function, single Test Generator with wrong macro - should fail
% "result from generator names:split_name_simple_generator_invalid_test_/0 is not a test"
split_name_simple_generator_invalid_test_() ->
    ?assertEqual(["Grant", "Winney"], split_name("Grant Winney")).         % FAILS.. yay!

% Test Generator Function, set of Test Generators with wrong macro - should fail
%  result: first failing test is reported; any after it do not run
split_name_complex_generator_invalid_test_() ->
        ?assertEqual({"Kenny", "Chesney"}, split_name("Kenny Chesney")),   % FAILS
        ?assertEqual({"B", "D"}, split_name("Bob Dilan")),                 % doesn't run
        ?assertEqual({"UmNo"}, split_name(""))                             % doesn't run

What happened?

It helps to start out by opening the OTP codebase and checking out how the Test Generator macros like ?_assertEqual are defined in eunit.hrl, as well as the basic ?assertEqual macros in assert.hrl. Here's those functions for convenience... but first, a couple things to note.

  • The _test() macro wraps asserts in a function, like I mentioned earlier.
  • The ?LINE and ?MODULE macros are predefined macros that return the current line number and module name, respectively.
-define(assertEqual(Expect, Expr),
        ((fun () ->
            __X = (Expect),
            case (Expr) of
                __X -> ok;
                __V -> erlang:error({assertEqual,
                                     [{module, ?MODULE},
                                      {line, ?LINE},
                                      {expression, (??Expr)},
                                      {expected, __X},
                                      {value, __V}]})

-define(_test(Expr), {?LINE, fun () -> (Expr) end}).

-define(_assertEqual(Expect, Expr), ?_test(?assertEqual(Expect, Expr))).

Okay, keep that in mind for a minute and check this out. If you create a test without an assert macro, the result of the test is whatever the result of the test function is. In the case of the last example below, the "result" is just a pointer to the function itself, something like Fun<erl_eval.20.99386804>... it doesn't actually execute it. So technically the test "passes".

good_math_test() ->
    2 + 2.             % Pass
bad_math_test() ->
    1 / 0.             % Fail, error:badarith
womp_womp_test() ->
    fun() -> 1/0 end.  % Pass, since it returns a function but doesn't execute it

So putting those two pieces together, hopefully now it's easier to see why that other test passed when it should've failed. Here it is again, with output of the ?assertEqual added in for good measure.


-spec split_name(string()) -> [string()].
split_name(FullName) ->
    string:split(FullName, " ", all).

split_name_invalid_test() ->
    TestResult = ?_assertEqual(["Um", "What?"], split_name("Grant Winney")),
    ?debugFmt("RESULT? ~p", [TestResult]),

And here's the result. The macro generates a function, but the Test Function won't actually execute it.

> eunit:test(names).
names.erl:13:<0.79.0>: RESULT? {12,#Fun<names.0.98814677>}
  Test passed.

So how do we fix it?

I don't know. I'm working on a script that I'll post here, which will tear through a module and fix tests.. or at least report them so you can fix them.

It's a real problem in my opinion that the EUnit tool isn't capable of reporting a mismatch like this, where a Test Function uses a macro reserved for Test Generators and Fixtures. There's absolutely no indication of a problem, just a silent "pass". πŸ‘Ž

Learn more

If you want to learn more about testing, check out this, this, this or this. There aren't a lot of great resources out there for Erlang.

If somehow that witty and quirky site isn't enough, here's a link to the official docs and may God have mercy on your soul. πŸ™„


Grant Winney

I write when I've got something to share - a personal project, a solution to a difficult problem, or just an idea. We learn by doing and sharing. We've all got something to contribute.

Comments / Reactions