Writing tests

Description: In this tutorial you will learn how to write tests for your CRAM functions and features.

Previous Tutorial: Implementing failure handling for the TurtleSim

The library we are going to use to write our unit tests is called lisp-unit, it is one of the simplest ones out there for Common Lisp. There are more elaborate ones that can do more things but in most cases lisp-unit is going to be more than enough for you.

For more information on the library check it's documentation page and the GitHub repo.

Let's create a new ASDF system for our test suite, it should be in the root of our ROS package, right next to cram-my-beginner-tutorial.asd. Call it cram-my-beginner-tutorial-tests.asd and put the following barebones code inside:

(defsystem cram-my-beginner-tutorial-tests
  :depends-on (cram-my-beginner-tutorial 
               lisp-unit))

In addition to creating a new ASDF system, we are going to create a new directory within our ROS package, called tests, where the test files will go. We need a package.lisp file for our new ASDF system, so let's create it and put it in the tests directory. Put the following code into it:

(defpackage :cram-my-beginner-tutorial-tests
  (:nicknames :tut-tests)
  (:use :common-lisp :lisp-unit))

In this tutorial we are going to test the function relative-angle-to from simple-plans.lisp. Therefore, we are going to create a file in our test suite called simple-plans-tests.lisp. Name the files with test functions the same as the original files, just with -tests as a suffix. So, let's create simple-plans-tests.lisp inside of tests directory and put the following inside:

(in-package :tut-tests)
 
;;; Turtle world is like this:
;;;  Y
;;;  ^
;;;  |
;;;  |
;;;  o-----> X
;;; Identity orientation is looking right in the direction of X axis.
 
(define-test relative-angle-to-dont-turn
  ;; turtle is in origin with identity orientation, wants to look at (3 0 0)
  (let* ((goal (cl-transforms:make-3d-vector 3 0 0))
         (pose-msg (roslisp:make-message 'turtlesim-msg:pose))
         (angle (tut::relative-angle-to goal pose-msg)))
    (assert-number-equal angle 0.0)))

So, we are testing the relative-angle-to function, which returns an angle, with which the turtle has to turn, in order to look towards a goal point.

What does the test checks the following situation: if the turtle is standing in the origin of the coordinate frame of the Turtlesim world, looking in the identity orientation, i.e., to the right in the direction of the X axis, what would be the angle with which it would have to turn to look at coordinate (3 0). As (3 0) lies on the X axis, and the turtle is already looking in that direction, it would have to move with the angle of 0.0. And that is what the test asserts.

Let us add just another test case: if the turtle is standing at the origin with identity orientation and wants to look at (-1 0), what would be the angle? Let us create a new test function for this, such that your simple-plans-tests.lisp would look as following:

(in-package :tut-tests)
 
;;; Turtle world is like this:
;;;  Y
;;;  ^
;;;  |
;;;  |
;;;  o-----> X
;;; Identity orientation is looking right in the direction of X axis.
 
(define-test relative-angle-to-dont-turn
  ;; turtle is in origin with identity orientation, wants to look at (3 0 0)
  (let* ((goal (cl-transforms:make-3d-vector 3 0 0))
         (pose-msg (roslisp:make-message 'turtlesim-msg:pose))
         (angle (tut::relative-angle-to goal pose-msg)))
    (assert-number-equal angle 0.0)))
 
(define-test relative-angle-to-look-back
  ;; turtle is in origin, wants to look at (-1 0 0)
  (let* ((goal (cl-transforms:make-3d-vector -1 0 0))
         (pose-msg (roslisp:make-message 'turtlesim-msg:pose))
         (angle (tut::relative-angle-to goal pose-msg)))
    (assert-number-equal angle pi)))

You can see that we needed to use the cl-transforms library for the test, as well as roslisp and turtlesim-msg. So let us add these to the .asd file of our test suite. We should also add the new files that we have created, package.lisp and simple-plans-tests.lisp to the ASDF system. It should now look something like this:

(defsystem cram-my-beginner-tutorial-tests
  :depends-on (cram-my-beginner-tutorial 
               lisp-unit
               cl-transforms
               roslisp
               turtlesim-msg)
  :components ((:module "tests"
                :components
                ((:file "package")
                 (:file "simple-plans-tests" :depends-on ("package"))))))

Now let us load the test package in REPL and switch into its namespace. You might need to restart your Emacs at this point, because the new ASDF system might not be visible to Emacs without restarting.

CL-USER> (ros-load:load-system "cram_my_beginner_tutorial" :cram-my-beginner-tutorial-tests)
CL-USER> (in-package :tut-tests)

To run the tests written with the lisp-unit library, we call the run-tests function:

TUT-TESTS> (lisp-unit:run-tests)
 
Unit Test Summary
 | 2 assertions total
 | 2 passed
 | 0 failed
 | 0 execution errors
 | 0 missing tests
 
#<TEST-RESULTS-DB Total(2) Passed(2) Failed(0) Errors(0)>

We can also specify which test specifically we want to run:

TUT-TESTS> (lisp-unit:run-tests '(relative-angle-to-dont-turn))
Unit Test Summary
 | 1 assertions total
 | 1 passed
 | 0 failed
 | 0 execution errors
 | 0 missing tests
 
#<TEST-RESULTS-DB Total(1) Passed(1) Failed(0) Errors(0)>

If we are in a different package in the REPL than the test suite package, we can also specify the package to test explicitly:

TUT-TESTS> (in-package :cl-user)
#<PACKAGE "COMMON-LISP-USER">
CL-USER> (lisp-unit:run-tests :all :cram-my-beginner-tutorial-tests)
Unit Test Summary
...

As there exist multiple testing libraries in Lisp, the ASDF facility has a unified way of calling the test suites, independent of the library. So let us integrate our test suite with the ASDF interface. For that, we are going to add a :perform tag to our cram-my-beginner-tutorial-tests.asd file, such that it now should look as following:

(defsystem cram-my-beginner-tutorial-tests
  :depends-on (cram-my-beginner-tutorial 
               lisp-unit
               cl-transforms
               roslisp
               turtlesim-msg)
  :components ((:module "tests"
                :components
                ((:file "package")
                 (:file "simple-plans-tests" :depends-on ("package")))))
  :perform (test-op (operation component)
                    (symbol-call :lisp-unit '#:run-tests :all :cram-my-beginner-tutorial-tests)))

And to use the ASDF interface for testing, we simply need to call the asdf:test-system function on our package. We can do that at any point in the REPL. So let us start a completely fresh REPL with nothing loaded in it and try the testing function:

CL-USER> (asdf:test-system :cram-my-beginner-tutorial-tests)
T

You will get some compilation prompts in between and in the end it should return T. Now let us destroy one of our tests and see what happens: change the angle in the first test in simple-plans-tests.lisp from 0.0 to pi:

(define-test relative-angle-to-dont-turn
  ;; turtle is in origin with identity orientation, wants to look at (3 0 0)
  (let* ((goal (cl-transforms:make-3d-vector 3 0 0))
         (pose-msg (roslisp:make-message 'turtlesim-msg:pose))
         (angle (tut::relative-angle-to goal pose-msg)))
    (assert-number-equal angle pi))

Unfortunately, if we rerun the tests now we will still get T as a result, as ASDF expects the testing library to throw an error if the tests fail, and ours does not.

In the future, we might want to tweak the library such that it does throw errors but for now just remember how the integration with the ASDF interfaces works.

So let us see, though, if the test actually failed:

CL-USER> (lisp-unit:run-tests :all :cram-my-beginner-tutorial-tests)
Unit Test Summary
 | 2 assertions total
 | 1 passed
 | 1 failed
 | 0 execution errors
 | 0 missing tests
 
#<TEST-RESULTS-DB Total(2) Passed(1) Failed(1) Errors(0)>

To get more detailed information, set the *print-failures* parameter to T:

CL-USER> (setf lisp-unit:*print-failures* t)
T
CL-USER> (lisp-unit:run-tests :all :cram-my-beginner-tutorial-tests)
 | Failed Form: PI
 | Expected 0.0d0 but saw 3.141592653589793d0
 |
RELATIVE-ANGLE-TO-DONT-TURN: 0 assertions passed, 1 failed.
 
RELATIVE-ANGLE-TO-LOOK-BACK: 1 assertions passed, 0 failed.
 
Unit Test Summary
 | 2 assertions total
 | 1 passed
 | 1 failed
 | 0 execution errors
 | 0 missing tests
 
#<TEST-RESULTS-DB Total(2) Passed(1) Failed(1) Errors(0)>

Now go back and fix the angle to the correct value and see if the tests pass again.

Now let us add more tests for the simple-plans.lisp file into our suite. Let's add some more tests for the relative-angle-to to check if the number are correct with assert-number-equal function. In addition to that assertion, there are also many different types of assertions, so we are going to try the assert-error one. Put the following into your simple-plans-tests.lisp:

(in-package :tut-tests)
 
;;; Turtle world is like this:
;;;  Y
;;;  ^
;;;  |
;;;  |
;;;  o-----> X
;;; Identity orientation is looking right in the direction of X axis.
 
(define-test relative-angle-to-dont-turn
  ;; turtle is in origin with identity orientation, wants to look at (3 0 0)
  (let* ((goal (cl-transforms:make-3d-vector 3 0 0))
         (pose-msg (roslisp:make-message 'turtlesim-msg:pose))
         (angle (tut::relative-angle-to goal pose-msg)))
    (assert-number-equal angle 0.0))
 
  ;; turtle is in (3 0 0) with identity orientation, wants to look at (4 0 0)
  (let* ((goal (cl-transforms:make-3d-vector 4 0 0))
         (pose-msg (roslisp:make-message 'turtlesim-msg:pose
                                         :x 3.0 :y 0.0 :theta 0.0))
         (angle (tut::relative-angle-to goal pose-msg)))
    (assert-number-equal angle 0.0))
 
  ;; turtle is in (0 0 0) looking up in direction of Y, wants to look at (0 1 0)
  (let* ((goal (cl-transforms:make-3d-vector 0 1 0))
         (pose-msg (roslisp:make-message 'turtlesim-msg:pose
                                         :x 0.0 :y 0.0 :theta (/ pi 2)))
         (angle (tut::relative-angle-to goal pose-msg)))
    (assert-number-equal angle 0.0))
 
  ;; turtle is in (1 1 0) looking towards the origin, wants to look at (0 0 0)
  (let* ((goal (cl-transforms:make-3d-vector 0 0 0))
         (pose-msg (roslisp:make-message 'turtlesim-msg:pose
                                         :x 1.0 :y 1.0 :theta (+ pi (/ pi 4))))
         (angle (tut::relative-angle-to goal pose-msg)))
    (assert-number-equal angle 0.0)))
 
(define-test relative-angle-to-look-back
  ;; turtle is in origin, wants to look at (-1 0 0)
  (let* ((goal (cl-transforms:make-3d-vector -1 0 0))
         (pose-msg (roslisp:make-message 'turtlesim-msg:pose))
         (angle (tut::relative-angle-to goal pose-msg)))
    (assert-number-equal angle pi))
 
  ;; turtle is in (1 0 0), wants to look at origin
  (let* ((goal (cl-transforms:make-3d-vector 0 0 0))
         (pose-msg (roslisp:make-message 'turtlesim-msg:pose
                                         :x 1.0 :y 0.0 :theta 0.0))
         (angle (tut::relative-angle-to goal pose-msg)))
    (assert-number-equal angle pi))
 
  ;; turtle is in (-1 -1 0) looking towards origin, wants to look away from origin
  (let* ((goal (cl-transforms:make-3d-vector -2 -2 0))
         (pose-msg (roslisp:make-message 'turtlesim-msg:pose
                                         :x -1.0 :y -1.0 :theta (/ pi 4)))
         (angle (tut::relative-angle-to goal pose-msg)))
      (assert-number-equal angle pi)))
 
(define-test relative-angle-to-type-error
  (assert-error 'type-error 
                (tut::relative-angle-to 
                 (cl-transforms:make-identity-pose)
                 (roslisp:make-message 'turtlesim-msg:pose))))

At this point, all the tests should be successful.

Please beware that if you define a test with a specific name, then delete it from the file but not restart the REPL, the test will still be in the current REPL session. So if you delete a test from a file, also think about restarting your REPL.