Automated testing on emulated targets
AmForth has a suite of tests that can run on emulated targets easily. This test suite is based on the standard Forth testing framework and includes the standard Forth test suite as well (some of the tests of the standard suite are excluded via preprocessing, look for lines with #ifdef DONT_SKIP for details). The test suite files are in $(AMFORTH)/tests directory.
To run the test suite, the target has to be built with WANT_IGNORECASE option set. This is because the standard test suite is written in all-caps and AmForth words are all in lowercase. This option allows words to match regardless of the case.
The make tests command runs the full test suite assuming a QEMU target compiled with WANT_IGNORECASE. For example, to run the test suite on the arm/mcu/qemu target, following steps would be used.
WANT_IGNORECASE=1 make allmake tests
The command analyses the output of the test run and summarizes the test results. An awk script is used to parse the test output. Awk is usually preinstalled on the host OS, if not use the OS package manager to install it.
Any test failures will be emitted in the output showing the test that failed and the incorrect output it produced. The command emits a final test summary showing the number of tests passed and failed and whether the whole suite completed. Summary of a successful test run looks like this
qemu-system-arm: terminating on signal 15 from pid 17281 (<unknown process>)
FINISHED: Y, PASS: 635, FAIL: 0
The termination warning is there because the test command must kill the QEMU process at the end, otherwise it would not quit. The test command gives the QEMU process predefined amount of time (e.g. 5 seconds) to run through the test suite and then kills it. It detects from the test output whether the whole test suite ran in that time or not. The timeout command is used to control the QEMU process. This command is native on Linux, on other OSes install coreutils to get it.
AmForth CI runs the full test suite on every new commit pushed to GitHub. It runs it twice, once in normal build configuration and once with WANT_ITC set. When WANT_ITC option is set the build prefers the core/words ITC version of words over the native assembler version if both exist (normally it is the other way around). This makes sure both versions are exposed to the test suite in the CI tests. This whole process is also repeated for each CPU architecture. It uses the QEMU -M virt target for each.
Manual testing on physical boards
Testing on physical boards is currently a manual process, but follows broadly the same method as the automated testing above. With a physical board attached via a serial connection
make tests-mcu
will execute a similar test of tests that can target both the RAM and flash of the device.
Not all tests are suitable when targeting flash.
Repeatedly running the test suite in flash with altered starting positions also provides a robustness test of the device’s flash framework implementation.
|S|10216|TESTING CORE WORDS
|S|10219|TESTING BASIC ASSUMPTIONS
|S|10227|TESTING BOOLEANS: INVERT AND OR XOR
...
|S|11638|TESTING compare, cscan, cskip
FINISHED: Y, PASS: 9024, FAIL: 0, UNKNOWN: 0
DP progress in flash whilst running tests
|O| 97|TYPE DP 0000A454
|O| 819|TYPE DP 0000B2E4
|O| 1541|TYPE DP 0000C174
|O| 2262|TYPE DP 0000D000
|O| 2984|TYPE DP 0000DE90
|O| 3706|TYPE DP 0000ED20
|O| 4428|TYPE DP 0000FBB0
|O| 5149|TYPE DP 00010A3C
|O| 5871|TYPE DP 000118CC
|O| 6593|TYPE DP 0001275C
|O| 7315|TYPE DP 000135EC
|O| 8036|TYPE DP 00014478
|O| 8758|TYPE DP 00015308
|O| 9480|TYPE DP 00016198
|O|10202|TYPE DP 00017028
|O|10923|TYPE DP 00017EB4