Comments on: What I mean by Reflective Testing
REBOL Technologies

Comments on: What I mean by Reflective Testing

Carl Sassenrath, CTO
REBOL Technologies
3-Apr-2008 0:07 GMT

Article #0358
Main page || Index || Prior Article [0357] || Next Article [0359] || 5 Comments || Send feedback

A few readers have asked for more detail about reflective testing (mentioned here.)

Let me first say that "reflective testing" sounds complicated, but it's not. In fact, you already have the power, you just may not know it. (I'm tempted to add "young Skywalker" to that, sorry.)

Here are the steps...

First, you know about the help function:

>> help union
        UNION set1 set2 /case /skip size

        Returns the union of two data sets.
        UNION is a native value.

        set1 -- first set (block! string! bitset!)
        set2 -- second set (block! string! bitset!)

        /case -- Use case-sensitive comparison (any)
        /skip -- Treat the series as records of fixed size (any)
                size --  (integer!)

How does help know all that info? It asks REBOL. Do this:

>> probe spec-of :union
    "Returns the union of two data sets."
    set1 [block! string! bitset!] "first set"
    set2 [block! string! bitset!] "second set"
    /case "Use case-sensitive comparison"
    /skip "Treat the series as records of fixed size"
    size [integer!]

That block is the specification of the function. It comes from the function itself, not from some other data source. That's why we call it reflective.

BTW, if you are using R2, do this first to make the above work:

spec-of: :third

Ok, so you know all the arguments needed to test the union function.

The arguments are:

>> probe third spec-of :union
[block! string! bitset!]

>> probe fifth spec-of :union
[block! string! bitset!]

Now, for each type, figure out what data you want to test with. Here we will start with blocks:

block: [[] [a] [b] [a b] [a b c]]

And, you are ready for the test loop. Give this a try.

foreach arg1 block [
    foreach arg2 block [
        test: reduce ['union arg1 arg2]
        probe test
        do test

You should see the test run and output:

[union [] []]
[union [] [a]]
[union [] [b]]

Ok, so now make a test script. Start by making the above into a simple function:

test-func: func [
    "Test a function with all values of a specific type"
    func-name [word!]
    /local test
    if not all [data1 data2] [exit]

    foreach arg1 data1 [
        foreach arg2 data2 [
            test: reduce [func-name arg1 arg2]
            probe test
            result: try test
            unless error? result [
                print ["result:" mold result]

Make the setup into a function:

test-function: func [
    "Test a function with all combinations of values."
    func-name [word!]
    data [block!]
    /locals spec types1 types2
    ; Skip to first arg word:
    spec: find spec-of get func-name word!

    ; Get the arg types:
    spec: find spec block!
    types1: first spec
    spec: find next spec block!
    types2: first spec

    ; Try all the types in combination:
    foreach type1 types1 [
        foreach type2 types2 [
            test-func func-name
                select data type1
                select data type2

Make a block of values of different types:

data: [
    string! ["" "a" "b" "ab" "bc" "abab"]
    block!  [[] [a] [b] [a b] [b c] [a b a b]]

And, you are ready to test:

>> test-function 'union data
[union [] []]
result: []
[union [] [a]]
result: [a]
[union [] [b]]
result: [b]
[union [] [a b]]
result: [a b]
[union [] [b c]]
result: [b c]
[union [] [a b a b]]
result: [a b]
[union [a] []]
result: [a]
[union [a] [a]]
result: [a]

That created and ran 144 tests.

Now try testing multiple functions:

foreach name [union intersect difference exclude] [
    test-function name data

That's 576 tests!

Of course, you now need to look over the results to be sure they look valid. But, that's a lot easier than typing in 576 tests. You just saved yourself days of agony.

So, that's the general method. There are many enhancements you can make, such as testing functions that have a variable number of arguments or testing refinements as well.

PS: I tested this code under R3 and it ran fine. Minor adjustments may be needed for R2.



Book Siberia
3-Apr-2008 19:55:48
Very nice example!

Definitely shows the power of reflective testing.

But now I am curious:

Can you show examples of how to do reflective testing on refinements? how to incorporate variable number of arguments for reflective testing?

Also, I am very curious - what other enhancements were you thinking of when you state that "there are many enhancements you can make..."?

For example? ... (please?)

Jerry Tsai
4-Apr-2008 0:48:35
I really cannot check all these results again and again. The results should be saved as a file. When a new REBOL release comes out, do the reflective testing again, and save the results as a file again. Now a simple diff program can tell us what was right is wrong, or vise versa. ...
Carl Sassenrath
4-Apr-2008 12:40:34
Book: yes, we can do some more examples and talk more about the method... and it will come online very soon, so you can see the actual code (which is only a page or so, and fairly easy to read.)

Jerry: the plan is that each test writes a log file. You verify the log file, but only once. Each main section of the log contains a hash key of both the test and the result. That hash is saved. Next time you run the test, it compares the hash. If it fails, you then diff the log files to find the specific test(s) that failed.

Duane Lachney
10-Apr-2008 3:44:29
It would be interesting if you could enhance your planned reflective test harness release so that it runs on top of rebol lightweight services.

The ability to conduct a suite of tests that are distributed amongst each downloaded copy of the harness would be a powerful mechanism that could showcase REBOL's capabilities.

That way you could parcel out hundreds of millions of tests and make quick work of the testing. You could build in redundancy by having the same tests distributed to the degree of coverage desired.

On the REBOL homefront end, as the test results come pouring in, the results could be aggegated into a central database (sort of like devbase?).

Once the test results are in, someone could peruse the results and flag whatever exceptions are found.

As new tests are thought up and previous tests need to be re-done (due to fixes and software updates), the tests can be placed in a queue for retesting - in this way you could systematically run through every reflective test you can think of.

Another possibility: users of the harness could suggest (and upload) new test suggestions [ complete with suggested data ], which could then be considered for addition to the queue.

Any plans for something like this in the near to intermediate future?

Anton Rolls
27-May-2008 3:51:05
There appears to be a small error in this line:
>> probe fifth spec-of :union
should be sixth

Post a Comment:

You can post a comment here. Keep it on-topic.


Blog id:



 Note: HTML tags allowed for: b i u li ol ul font p br pre tt blockquote

This is a technical blog related to the above topic. We reserve the right to remove comments that are off-topic, irrelevant links, advertisements, spams, personal attacks, politics, religion, etc.

Updated 8-Jun-2023   -   Copyright Carl Sassenrath   -   WWW.REBOL.COM   -   Edit   -   Blogger Source Code