What I mean by Reflective Testing

A few readers have asked for more detail about reflective testing (mentioned here.)

Let me first say that "reflective testing" sounds complicated, but it's not. In fact, you already have the power, you just may not know it. (I'm tempted to add "young Skywalker" to that, sorry.)

Here are the steps...

First, you know about the help function:

>> help union
USAGE:
        UNION set1 set2 /case /skip size

DESCRIPTION:
        Returns the union of two data sets.
        UNION is a native value.

ARGUMENTS:
        set1 -- first set (block! string! bitset!)
        set2 -- second set (block! string! bitset!)

REFINEMENTS:
        /case -- Use case-sensitive comparison (any)
        /skip -- Treat the series as records of fixed size (any)
                size --  (integer!)

How does help know all that info? It asks REBOL. Do this:

>> probe spec-of :union
[
    "Returns the union of two data sets."
    set1 [block! string! bitset!] "first set"
    set2 [block! string! bitset!] "second set"
    /case "Use case-sensitive comparison"
    /skip "Treat the series as records of fixed size"
    size [integer!]
]

That block is the specification of the function. It comes from the function itself, not from some other data source. That's why we call it reflective.

BTW, if you are using R2, do this first to make the above work:

spec-of: :third

Ok, so you know all the arguments needed to test the union function.

The arguments are:

>> probe third spec-of :union
[block! string! bitset!]

>> probe fifth spec-of :union
[block! string! bitset!]

Now, for each type, figure out what data you want to test with. Here we will start with blocks:

block: [[] [a] [b] [a b] [a b c]]

And, you are ready for the test loop. Give this a try.

foreach arg1 block [
    foreach arg2 block [
        test: reduce ['union arg1 arg2]
        probe test
        do test
    ]
]

You should see the test run and output:

[union [] []]
[union [] [a]]
[union [] [b]]
...

Ok, so now make a test script. Start by making the above into a simple function:

test-func: func [
    "Test a function with all values of a specific type"
    func-name [word!]
    data1
    data2
    /local test
][
    if not all [data1 data2] [exit]

    foreach arg1 data1 [
        foreach arg2 data2 [
            test: reduce [func-name arg1 arg2]
            probe test
            result: try test
            unless error? result [
                print ["result:" mold result]
            ]
        ]
    ]
]

Make the setup into a function:

test-function: func [
    "Test a function with all combinations of values."
    func-name [word!]
    data [block!]
    /locals spec types1 types2
][
    ; Skip to first arg word:
    spec: find spec-of get func-name word!

    ; Get the arg types:
    spec: find spec block!
    types1: first spec
    spec: find next spec block!
    types2: first spec

    ; Try all the types in combination:
    foreach type1 types1 [
        foreach type2 types2 [
            test-func func-name
                select data type1
                select data type2
        ]
    ]
]

Make a block of values of different types:

data: [
    string! ["" "a" "b" "ab" "bc" "abab"]
    block!  [[] [a] [b] [a b] [b c] [a b a b]]
]

And, you are ready to test:

>> test-function 'union data
[union [] []]
result: []
[union [] [a]]
result: [a]
[union [] [b]]
result: [b]
[union [] [a b]]
result: [a b]
[union [] [b c]]
result: [b c]
[union [] [a b a b]]
result: [a b]
[union [a] []]
result: [a]
[union [a] [a]]
result: [a]
...

That created and ran 144 tests.

Now try testing multiple functions:

foreach name [union intersect difference exclude] [
    test-function name data
]

That's 576 tests!

Of course, you now need to look over the results to be sure they look valid. But, that's a lot easier than typing in 576 tests. You just saved yourself days of agony.

So, that's the general method. There are many enhancements you can make, such as testing functions that have a variable number of arguments or testing refinements as well.

PS: I tested this code under R3 and it ran fine. Minor adjustments may be needed for R2.

5 Comments

Updated 10-Mar-2024   -   Copyright Carl Sassenrath   -   WWW.REBOL.COM   -   Edit   -   Blogger Source Code