red-system-quick-test.txt

Red/System Quick Test - A testing framework 

  Author: Peter W A Wood
  Date: 29/02/2012
  Version 0.8.0
	Home: <a href="http://www.red-lang.org">red-lang.org</a>

===Introduction
Quick-Test is a small testing framework for Red/System. It is called quick-test as it was put together quickly to support the development of the "boot-strapped" version of Red/System. Hopefully, it also runs tests quite quickly and, perhaps more importantly, is a quick way to write meaningful tests.

As a result, quick-test is very fussy about where files are and it is not particularly fault tolerant. These were traded-off for an earlier implementation.

There are actually two aspects to Quick-Test. The first is quick-test.reds, a handful of Red/System functions which can be included in a Red/System program to write unit tests. This can easily be used on its own to test Red/System code and is not at all fussy about where files reside.

A temporary addition has been made to this part of Quick-Test. It is quick-unit-test.r which is written in REBOL and used for unit testing the bootstrapping Red compiler.

The second aspect is that Quick-Test is a full-test suite for Red/System itself and includes the files used to test Red/System.

There are many improvements that could be made to quick-test. It is expected that quick-test will be replaced at the time when Red/System is re-written so they probably won't get made.

===Types of Tests
Quick-Test supports three types of tests - tests of Red/System code, tests of Red/System programs and tests of the Red/System compiler behaviour. Temporarily, Quick-Test also supports REBOL unit tests for the Red compiler.

The first type of tests are written in Red/System, the other three in REBOL.

===Quick-Test Components
Quick-Test consists of the following:

  quick-test.r - a REBOL script that supports testing compiler and executable output.
  quick-test.reds - a set of Red/System functions that supports writing tests in Red/System.
  quick-unit-test.r - a set of REBOL functions that supports writing tests in REBOL.
  
(The above are stored in the Red/quick-test/ directory.)
  
  run-all.r - a script(s) which runs a set of Red/System tests.
            - run-all.r in Red/red-system/tests/ runs the Red/System test suite
            - run-all.r in Red/red/tests/ will run the Red test suite*
            - run-all.r in Red/ will run all the tests*
  run-test.r - a script which will run an individual test.
             - stored in Red/quick-test/
  
* to be written
  
===Directory Structure
Quick-Test uses the following directory structure:

  Red/
    quick-test/                     ;; quick-test components
  red/
    tests/
      run-all.r                     ;; runs all Red tests
      source/
        compiler/                   ;; red compiler unit tests
        runtime/                    ;; runtime tests
  red-system/
    tests/                          ;; the main test directory
      run-all.r                     ;; runs all Red/System tests
      run-test.r                    ;; runs an individual Red/System test     
      source/                       ;; all test sources
        builtin/                    ;; tests for builtin functions (eg print)
        compiler/                   ;; tests of the compiler
        run-time/                   ;; tests of the run time library
        units/                      ;; base language tests (eg datatype tests)
      runnable/                     ;; the test executables
                                    ;; automatically created by Quick-Test
                                    ;; listed in .gitignore

===Running Tests
The tests are designed to be run from the command line or a REBOL console session. This must be a REBOL/View console session under Windows.

The following examples assume that they have been run from the Red dir

To run all Red tests from the console:
  
  rebol -s red/tests/run-all.r
  
To run all Red tests from the REBOL console:
  
  do %red/tests/run-all.r

To run all Red/System tests from the console:
  rebol -s red-system/tests/run-all.r
  
To run all Red/System tests from the REBOL console:
  do %red-system/tests/run-all.r
  
To run an individual test file from the console:
  rebol -s quick-test/run-test.r "<file>"  
  
To run an individual test file from the REBOL console:
  do/args %quick-test/run-test.r "<file>"
  
  where <file> is the path to the file to be tested .
  
e.g. The command to run the logic-test.reds test file which is in the units directory:
    do/args %quick-test/run-test.r "red-system/tests/source/units/logic-test.reds"
    
(Note: %run-test.r can run both .r and .reds tests).
(Note: When using run-test.r from within a REBOL console session, the file paths to both run-test.r and the test file msut be relative to the directory from which the REBOL session was launched (system/options/path).

===Writing Tests
In terms of writing tests, Quick-Test is quite minimal by design. It provides only a simple assert function and a minimal set of test organisation functions.

---Examples

Example 1: A Test of inc.reds - an imaginary function which adds 1 to an integer

  Red/System [
    Title:   "Tests of inc.reds"
    File: 	 %source/run-time/inc-test.reds
  ]

  #include %../../quick-test/quick-test.reds 
  #include %relativepathto/inc.reds

  ~~~start-file~~~ "inc"                          ;; start test file  
                                                  ;;  initialises totals

  ===start-group=== "increment an variable"       ;; a marker to group tests
                                                  ;;  group name [string!]

    --test-- "inc-test-1"                         ;; start of code for a test
                                                  ;;   test name [string!]
      i: 1
    --assert 2 = inc i                            ;; an assertion
                                                  ;;   expression [logic!]

  ===end-group===                                 ;; end of group marker                                                  

  ~~~end-file~~~                                  ;; finish test - print totals
  
  
Example 2: A test to check the compiler correctly identifies aan unidentified variable:

  change-dir %../                   ;; revert to tests/ dir (from runnable)
                                    ;; .r test scripts are copied to runnable 
                                    ;; before execution

                                    ;; There is no need to include quick-test.r
                                    ;; as it will have been included by either 
                                    ;; %run-all.r or %run-test.r

  ~~~start-file~~~ "comp-err"       ;; start test

  --test-- "compile-error-1"
  
  --compile-this {                  ;; compiles the suplied string
                                    ;; automatically inserts Red/System []
                                    ;; compiler output is collected in
                                    ;;  qt/comp-output
        i := 1;                     
      }     

    --assert none <> find qt/comp-output "*** undefined symbol"
    --assert none <> find qt/comp-output "at:  ["
    --assert none <> find qt/comp-output "i := 1"
    --assert none <> find qt/comp-output "]"
    
    --clean                         ;; tidies up temporary files

  ~~~end-file~~~                          ;; ends test and print totals

Example 3: Test the output of a Red/System programs
  
  change-dir %../                   ;; revert to tests/ directory from runnable/

  ~~~start-file~~~ "output"  

    --test-- "hello"
                                    ;; --compile-and-run compiles and runs 
                                    ;;  a Red/System program. The output is 
                                    ;;  collected qt/output
                                    
    --compile-and-run %source/compiler/hello.reds 
    --assert none <> find qt/output "hello"    
    --assert none <> find qt/output "world"
      
  ~~~end-file~~~   

---Test Dialects

+++quick-test.reds

/note 

The syntax for quick-test.red is the same as that for quick-test.reds

\note

  ***start-run*** <title>
    <title> : title of the test run - c-string!
  
Marks the start of the test run and initialises the run totals.

  ~~~start-file~~~ <title>
    <title> : title of the test file - c-string!
    
Marks the start of an individual test file.

  ===start-group=== <title>
    <title> : title of the test group - c-string!
    
Marks the start of a group of tests. Using groups in a test is optional.

  --test-- <title>
    <title> : title of an inidvidual test - c-string!
    
The start of an individual test. The number of tests in a run and file are counted and reported.

  --assert <assertion>
    <assertion> : a Red/System expression that returns a logic! value
    
An assertion. True = pass. False = fail. The number of assertions, passes and failures are counted and reported for each run and file.

  --assertf~= <value> <value> <tolerance>
    <value> :     a float! value
    <tolerance> : a float! value
    
An assertion that two floating point numbers are approximately equal. The tolerance provided is used both as an absolute and relative tolerance. The calculation of the tolerance is a quite simplistic but has proved sufficient for its purpose.

  --assertf32~= <value> <value> <tolerance>
    <value> :     a float32! value
    <tolerance> : a float32! value
    
AssetF~= for float32! values.

  ===end-group===
  
Marks the end of a group of tests.

  ~~~end-file~~~

Marks the end of a test file.

  ***end-run***
  
Marks the end of a test run.

+++quick-test.r

quick-test.r supports tests with two different levels of output. The more verbose of the two displays totals for each file in a run and highlights failing tests. 

The second, "quiet mode" displays only a summary of each tests in a concise report. The more detailed report is logged to %quick-test.log in the tests/ directory. This is triggered by using the quiet version of the commands in the dialect.

  ***start-run*** <title>
    <title> : title of the test run - string!
  
Marks the start of the test run and initialises the run totals.

 ***start-run-quiet*** <title>
    <title> : title of the test run - string!
  
Marks the start of a "quiet" test run and initialises the run totals.

  ~~~start-file~~~ <title>
    <title> : title of the test file - string!
    
Marks the start of an individual test file.

  ===start-group=== <title>
    <title> : title of the test group - string!
    
Marks the start of a group of tests. Using groups in a test is optional.

  --test-- <title>
    <title> : title of an inidvidual test - string!
    
The start of an individual test. The number of tests in a run and file are counted and reported.

  --compile <file>
    <file> : Red/System source file - file!
    
Compiles a Red/System source file. The output from the compiler is captured in qt/comp-ouput. If the compile is successful, qt/exe will contain the path to the executable file. It will be 'none if the compile failed.

  --compile-this <source code>
    <source code> : Red/System source - string!
    
Compiles a Red/System source supplied as a string in the test script. A Red/System header is automatically inserted if one is not supplied. The output from the compiler is captured in qt/comp-ouput. If the compile is successful, qt/exe will contain the path to the executable file. It will be 'none if the compile failed.

  --compile-and-run <file>
    <file> : Red/System source file - file!
    
Compiles a Red/System source file and runs it. The output from the compiler is captured in qt/comp-ouput. If the compile is successful, qt/exe will contain the path to the executable file. It will be 'none if the compile failed. The output produced by the executable is captured in qt/output. (It will contain "Compilation failed" in the case of a compilation error.)

  --compile-and-run-this <source code>
    <file> : Red/System source - string!
    
Compiles a Red/System source supplied as a string in the test script and runs it. A Red/System header is automatically inserted if one is not supplied. The output from the compiler is captured in qt/comp-ouput. If the compile is succesful, qt/exe will contain the path to the executable file. It will be 'none if the compile failed. The output produced by the executable is captured in qt/output. (It will contain "Compilation failed" in the case of a compilation error.)

  --compile-run-print <file>
  <file> : Red/System source file - file!
  
Compiles, runs and prints the output of a Red/System program supplied as a source file. It is designed to run test programs written using the quick-test.reds test framework.

  --add-to-run-totals
  
Is used after a --compile-and-run or --compile-run-print to collect the totals from a test using the quick-test.reds framework.

  --run <file>
    <file> : an executable program - file!
    
Runs an executable program. The output is captured in qt/output.

  --run-red-test-quiet <file>
    <file> : a REBOL script file - file!
    
Runs a quick-test.red test written in REBOL in quiet mode.

  --run-script <file>
    <file> : a REBOL script - file!
    
Runs a REBOL script.

  --run-script-quiet <file>
    <file> : a REBOL script - file!
    
Runs a REBOL script in quiet mode.

  --run-test-file <file>
  <file> : a quick-test.reds test program
 
Compiles, runs and prints the output of a Red/System test program written using quick-test.reds. It automatically adds the totals from the test to the run-totals.

  --run-test-file-quiet <file>
  <file> : a quick-test.reds test program
 
Compiles, runs and prints the output of a Red/System test program written using quick-test.reds in quiet mode. It automatically adds the totals from the test to the run-totals.

  --assert <assertion>
    <assertion> : a Red/System expression that returns a logic! value
    
An assertion. True = pass. False = fail. The number of assertions, passes and failures are counted and reported for each run and file.

  --assert-msg? <message>
    <message> : an expected compiler messsage - string!
    
Checks if the compiler has produced the expected message.

  --assert-printed? <phrase>
    <phrase> : expected runtime output - string!
    
Checks if the runtime output included the expected phrase.

  --clean
  
Cleans up the temporary files created by --compile-this and --compile-and-run-this. Can be safely used, but is not necessary, after --compile, --compile-and-run and --run.
  
  ===end-group===
  
Marks the end of a group of tests.

  ~~~end-file~~~

Marks the end of a test file.

  ***end-run***
  
Marks the end of a test run.

  ***end-run-quiet***
  
Marks the end of a quiet-mode test run.
  


###

REBOL []

do/args %makedoc2.r 'load-only
doc: scan-doc read file: system/options/script
set [title out] gen-html/options doc [(options)]

file: last split-path file
replace file ".txt" ".html"
file2: copy file
insert find file2 "." "-light"
replace out "$DARK$"  file
replace out "$LIGHT$" file2

write file out
replace out "dark.css" "light.css"
write file2 out