Shake
A Simple Lua Test Engine

Introduction

Many Lua modules and applications use a simple pattern for internal tests. They have a script (usually called test.lua) that exercises the module API using API calls and assert() calls to verify the results. It is also common practice in those modules to use print() to output information about the test progress and to use Lua comments to inform what is being tested to whoever reads the source. Although widespread this approach may be too crude for those wanting to test more than one module at a time or to have a more detailed view of the results.

Shake assumes that the tests have been implemented for the above scenario but offers a transparent test engine for those wanting the execution of the tests in batch mode or those wanting to get an overview of the various results. The Shake engine can inform not only the number of tests, failures and errors found in a group of test scripts, but it can infer information about the test context using the output and comments associated with the assert() calls.

The main characteristic of Shake is being transparent, which means that the module authors and test writers do not have to be aware that tests are going to be run with Shake. As long as the tests call assert() Shake can obtain quite a lot of information from the source and run time execution. This is done through the pre-processing of the tests source code using Leg (and therefore LPeg) to replace every call to assert() with one that can extract information about the expressions, values and operators involved in the assertion.

Another common alternative to testing is the one adopted by test frameworks like lunit and luaunit, which offer a quite sophisticated API for those test scripts. For those wanting to use a xUnit style framework these two Lua modules are very useful but in this case the test authors have to be aware that the tests are going to be executed by a xUnit framework.

One of the strong points of the xUnit framework style is being able to report failures in a much more detailed view than a simple assertion. In Lua, a call to assert() is basically equivalent to an if that displays nice error messages (provided in the assert() call). Unfortunately, an assertion just checks if an expression is true or false and report that as an error in the second case.

This makes the test scripts much simpler, but also means that you have to provide decent error messages for every assertion and that once one assertion fails, your test stops running. Another disvantage of a simple assert() call is that it is does not provide much information about what caused the error. For example, in an assertion like:

assert(items == 10, "too many items")

If the assertion does not fail then you can be certain that you have 10 items, but if it fails then all you can say is that items is not the number 10. If you want to know what is the current value of items in that assertion you would have to add it to the error message.

The Shake engine offers an alternative way to obtain that information by preprocessing the test script before it is executed and constructing a Lua table representation of the test execution. Once the tests are run, we can iterate over this results table and display a lot more information about the test execution than is usually available. In the above example Shake would not only be able to inform that the assertion failed, but also the value of items and even that the asserted expression involved a non terminal called "items" which was supposed to be equal to the number 10.

Since a results table is usually quite long and have a lot of information that may be useful for some users but not others, Shake offers the separation of a test runner and a test reporter. A test runner is responsible for determining which test scripts are going to be executed and accumulating the results in the runner results table. A test reporter uses the results table and displays information about the results in some way. Shake offers two built in runners, one to be used in the command line and another to be used as a web application. Shake also offers two built in reporters, one that outputs a summary of the results as a text message and another that displays a much more complete report using HTML tables and CSS to format the results.

For those wanting more control on the test execution and reporting, the Shake module (the engine itself) can be used as an standard module and offers an API for test execution and for accessing the results. For more details see the examples and the API reference.

Installation

The Shake source distribution contains two components. The first one is a Lua module and command line runner, the second is a CGILua application runner.

The easiest way to install the Shake module and command line runner is using LuaRocks:

luarocks install shake

If you prefer to install Shake manually and you are using Unix then you can install the command line runner and the Shake module using the command below (where the optional PREFIX parameter assumes /usr/local as default):

make PREFIX=prefix_path install

After you have the Shake engine installed you can optionally install the CGILua application using a similar command (note that the CGILua application cannot be installed by using LuaRocks):

make KEPLER_PREFIX=prefix_path install_app

Finally, if you are using Kepler in Windows you would have to copy the Shake source files to your KEPLER_BIN, KEPLER_LUA (and CGILUA_APPS if you want to use the Shake CGILua application) directories as the example below shows:

Kepler Base
    /apps
    /cgilua
        /shake		-- from /src/apps/shake
    /bin
        shake			-- from /src/bin
        shake.bat		-- from /src/bin
    /lua
        shake.lua		-- from /src/shake
        /shake
            stir.lua		-- from /src/shake

Running Shake

Once installed Shake can be run as a command line command or as a CGILua web application.

Command line runner

The Shake command line runner is called shake and uses the following syntax:

shake [options] [filename]

This command runs a test file called filename (assuming test.lua by default) in the current directory or any of its subdirectories if the -r option is given.

The command options are:

-h, --help
Prints this help message
-r, --recursive
Recursively scans subdirectories for test.lua files and runs them
-v, --version
Prints the Shake version

Check the examples page for an output of a Shake run.

CGILua application runner

If you have installed the Shake CGILua application in your CGILUA_APPS directory, Shake assumes that the tests to be run will be in the SHAKE_TESTS directory. If you don't define SHAKE_TESTS in your CGILua configuration file, Shake assumes the default of KEPLER_CONF.."/tests".

The SHAKE_TESTS directory should contain one directory for each module that you want to test. When you access the URL

/app.lua/shake

Shake will look into those directories for a test.lua file and execute it if present. After all the test scripts are executed Shake will report a summary of the results for all the tests executed.

This report is an HTML table that also offers a link to a more detailed view to the results of a single module test run. Once you click on that link Shake will show the full results of the test execution, including every assertion involved, even the succesfull ones

Reference

For those interested in more control over the runner and the report, Shake offers the following functions (check the examples for more information):

shake.runner()
Returns a Shake runner that offers the functions:
runner.test(filename)
Executes a test suite and stores the results in the runner.results table.
runner.summary()
Outputs a summary of the results from the runner.results table.
shake.isterminal(expression)
Returns a boolean value indicating if the expression refers to a terminal symbol.

Valid XHTML 1.0!

$Id: manual.html,v 1.6 2007/12/21 22:55:20 carregal Exp $