REBOL 3.0 Testing and Test Suite

Carl Sassenrath
CTO, REBOL Technologies
Updated 10-Apr-2008
REBOL 3 Home Page

Overview

Goal

Our main goal is to build a reliable way to certify the proper operation of REBOL for each new version.

Primarily we want to know that all datatypes operate correctly, that all functions produce correct results, that complex expressions evaluate as they should, and that the system is reliable and stable over time.

This goal is non-trivial because REBOL is an extensive language, allowing substantially greater depth and range of expressions than most other programming languages. However, each area we test provides incremental improvement. Adding tests, even if a little at a time, is better than not adding tests at all.

Areas of Testing

Testing R3 can be divided into a number of sections:

	Datatypes	testing the 56 built-in datatypes of REBOL. Includes load and mold serialization as well as testing all combinations of polymorphic action functions (add, subtract, insert, remove, etc. - 68 total.)
	Natives	testing the 132 native functions and all their refinements.
	Mezzanines	testing the mezzanine functions and all their refinements.
	Series	testing the most common series combinations. For example, using different index positions.
	I/O	testing files, ports, networking with open, close, read, write, and other such functions.
	Graphics	rendering and windowing code. Will require a method to automatically confirm results.
	Events	event handling, port waits, etc. This is a special area because it requires user input and confirmation (but can be automated to some degree).
	Memory	testing the language's stack, memory management, and garbage collection.

Each of these areas will require test files to specify how the tests are to be done, and what data values and parameters to use.

Test Approach

In general, each test must be kept as simple as possible in a way that adequately tests the required component, element, or property.

Keeping tests simple also enables us to quickly isolate specific problems, for example a problem with adding two decimal values. At the same time it gives us a fighting chance at analyzing and locating the source of the bug at its lower levels.

What is a Test Vector?

The basic unit of testing is a test vector. A test vector is nothing more than a block of code, that when evaluated, produces a logical result.

For example, this test vector checks that LOAD converts a string to an integer:

[integer? load "0"]

These test that the integer is actually zero:

[0 = load "0"]
[zero? load "0"]

Note that although similar, these tests are not identical. Each tests something a bit different.

This is typically the case in testing. A wide range of possibilities must be enumerated. Here are just a few lines from an actual test suite:

[integer? load "0"]
[integer? load "+0"]
[integer? load "-0"]
[integer? load "1"]
[integer? load "+1"]
[integer? load "-1"]
[integer? load "000"]
[integer? load "0001"]
[integer? load "2"]
[integer? load "-2"]

The important point is that each test vector is minimal.

Generating Tests with Reflection

What is the best way to create these test vectors? We could code them by hand, but it could take a lot of effort to both think-up all the combinations and type-in a few thousand vectors, just for the integer datatype alone.

Instead, it is better for us to generate all the necessary combinations using REBOL itself. This is easy to do in REBOL because it is a highly reflective language -- it is able to process itself.

To generate reflective tests, two sources are needed: a source for code variation and a source for data variation. The code variations are specified within the language implementation itself.

For example, the complete set of datatype action functions (polymorphic functions) can be found:

>> system/catalog/actions
== [add subtract multiply divide remainder power ...

For each of these the arguments are known from the reflective action:

>> spec-of :add
== [
    "Returns the result of adding two values."
    value1 [scalar! date!]
    value2
]

Using this information, it is fairly easy to design a method to test all combinations of the function.

The next requirement is that we provide a range of data values to test with. Picking the best data values requires a bit of analysis.

Using integers as an example, these values are obviously useful:

-1 0 1

In addition, picking a couple large values may be worthwhile:

1234567890123456789 -1234567890123456789

However, it can be debated whether other values provide much benefit:

2 -2 12 -12 1234 -1234

In fact, since every value is tested with every other value, the number of tests grows exponentially with each data value. This fact forces us to keep our test data limited to a reasonable set of values.

However, for now, let's go ahead and add whatever data we think makes sense. We can review it if necessary later and remove useless values.

"Manual" Direct Tests

Although reflective tests generate a large number of test vector combinations, certain patterns may not lend themselves well to that technique. For example, the test generator may not produce reasonable tests for special functions like random or efficient tests for things like file I/O.

Therefore, every test data file can include a set of special direct tests that are used as-is, not generated.

For example, the pair! datatype may include extra tests such as these that test the x and y refinements of a value:

[p: 1x2 p/x = 1]
[p/y = 2]
[p/x: 3 p = 3x2]
[p/y: 4 p = 3x4]
[5 = p/x: 5]
[6 = p/y: 6]

Format of a Test File

Here is an example test file:

REBOL [
    Title: "Pair Tests"
    File: %pair.r
]

spec: [
    ; This is a test of this datatype (can also be 'native, etc.)
    type: pair!

    ; If you want to watch this test run:
    verbose: off

    ; These strings should load/mold:
    load-args: [
        "0x0" "1x2" "10x20"
        "-0x0" "-1x2" "-10x20"
        "0x-0" "1x-2" "10x-20"
        "-0x-0" "-1x-2" "-10x-20"
        "123456x123456" "-123456x123456"
        "123456x-123456" "-123456x-123456"
    ]

    ; These should fail to load:
    no-load-args: [
        "1x" "1x2x3"
    ]

    ; A common set of test values:
    test-args: [
        0x0 1x0 0x1 1x1 -1x0 0x-1 -1x-1 1000x2000
    ]

    ; Other datatypes to test with specific action groups:
    with-types: [scalar!]  ; test with all other scalars

    ; Do not auto-test this action:
    not-actions: [random reflect]
]

test: [
    ; Manual test vectors go here. Each vector is a block.
    ; These tests are odd cases, or ones that require special
    ; testing, or are being verified from bug reports.
    ; Each vector returns TRUE. Use ERROR? for ones that should error out.
    [p: 1x2 p/x = 1]
    [p/y = 2]
    [p/x: 3 p = 3x2]
    [p/y: 4 p = 3x4]
    [5 = p/x: 5]
    [6 = p/y: 6]
    [10x20 = as-pair 10 20]
]

checks: [
    load #{14EDC79DE0D7931E918330E721AAB43F1A8D61E8}
    noload #{42E5508F274F679211B5813F8B8F217B1364C6B8}
    actions #{EC284EC4712D23CA05BFAEB0D549B1085E6DAA31}
    direct #{D858E95AD5F52596E08E2D1D6D3F37133661B9CD}
]

The file is divided into these sections:

	spec	an object that specifies the data and other parameters used to generate reflective tests. Fields of the object are explained in the comments above.
	test	manual tests, run as-is.
	checks	a set of hashes used to detect changes in test results. These are automatically written to the file by the test engine.

Test Engine

The file rebtest.r is the test engine. The design has been kept simple and clean, allowing any intermediate REBOL user to understand its operation.

Configuration

A few configuration options can be controlled at the top of the file:

;-- Config -----------------------

; VERBOSE: higher numbers, more output:
;   * 0 for quiet
;   * 20 for progress reports
;   * 40 to print each test (after)
;   * 60 to print each test (before)
verbose: 20

; REHASH: update hash tables in datatype files:
rehash: off

; SHOW-ERROR: show test errors (but, many may be correct)
show-error: off

; DATATYPE-DIR: location of datatype tests:
;   Default: %datatypes/
datatype-dir: %datatypes/

; ONLY-TYPES: for selective testing of specific types:
;   Example: [pair!]
only-types: []

; EXCLUDE-TYPES: to ignore a specific type:
;   Example: [tuple! pair!]
exclude-types: []

The rehash option is important. It controls whether the checks portion of a datatype is overwritten with new hash checksums. This should be done whenever the test results have been verified to be correct.

Log Files

For each test file that is run, a log file is written. The log file has the same name as the test file. For example:

integer.r

will output the log file:

integer.log

The log file is written as a valid REBOL file and contain lists of test vectors for each section of the tests. The log file can be evaluated by REBOL (using a very simple teset driver script that evaluates each vector block) to confirm that all of the tests still succeed. This is one way to detect a regression in REBOL (a new bug).

Adding Tests

We can use your help in improving this test suite. Tests can be added or edited by modifying the existing test files or by creating new ones.

Currently, all test files are stored on the R3-Alpha AltME world. Once R3 is working well enough, and once this test approach has had the chance to be refined and proven, we will move the tests to DevBase for greater ease of public access.

In addition, we will be adding a built-in test function to R3 that will quickly download the current tests, run them, and display possible failures. The idea is that if we make it easy enough to run the tests, many developers will do so. This will be especially helpful over a wide array of target platforms.