Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
tutorials:beginner:testing [2019/07/10 15:06] – created gkazhoyatutorials:beginner:testing [2022/03/15 10:12] (current) – [Integrating into the ASDF testing operation] schimpf
Line 1: Line 1:
 ====== Writing tests ====== ====== Writing tests ======
-(Caution: this tutorial is currently under construction. Please come back tomorrow to see the complete version.) 
  
-**Description:** In this tutorial you will learn how to write unit (and not only) tests for your CRAM functions and features.+**Description:** In this tutorial you will learn how to write tests for your CRAM functions and features.
  
 **Previous Tutorial:** [[tutorials:beginner:failure_handling|Implementing failure handling for the TurtleSim]] **Previous Tutorial:** [[tutorials:beginner:failure_handling|Implementing failure handling for the TurtleSim]]
Line 9: Line 8:
 ===== lisp-unit ===== ===== lisp-unit =====
  
 +The library we are going to use to write our unit tests is called ''lisp-unit'', it is one of the simplest ones out there for Common Lisp. There are more elaborate ones that can do more things but in most cases ''lisp-unit'' is going to be more than enough for you.
 +
 +For more information on the library check it's [[https://github.com/OdonataResearchLLC/lisp-unit/wiki|documentation page]] and the [[https://github.com/OdonataResearchLLC/lisp-unit|GitHub repo]].
 +
 +
 +===== Setting up the infrastructure and our first test =====
 +
 +Let's create a new ASDF system for our test suite, it should be in the root of our ROS package, right next to ''cram-my-beginner-tutorial.asd''. Call it ''cram-my-beginner-tutorial-tests.asd'' and put the following barebones code inside:
  
 <code lisp> <code lisp>
-blabla+(defsystem cram-my-beginner-tutorial-tests 
 +  :depends-on (cram-my-beginner-tutorial 
 +               lisp-unit))
 </code> </code>
  
 +In addition to creating a new ASDF system, we are going to create a new directory within our ROS package, called ''tests'', where the test files will go.
 +We need a ''package.lisp'' file for our new ASDF system, so let's create it and put it in the ''tests'' directory. Put the following code into it:
  
-===== Writing our own test for the beginner tutorial code =====+<code lisp> 
 +(defpackage :cram-my-beginner-tutorial-tests 
 +  (:nicknames :tut-tests) 
 +  (:use :common-lisp :lisp-unit)) 
 +</code>
  
-Let's see how to implement this.+In this tutorial we are going to test the function ''relative-angle-to'' from ''simple-plans.lisp''
 +Therefore, we are going to create a file in our test suite called ''simple-plans-tests.lisp''
 +**Name the files with test functions the same as the original files, just with ''-tests'' as a suffix.** 
 +So, let's create ''simple-plans-tests.lisp'' inside of ''tests'' directory and put the following inside:
  
 <code lisp> <code lisp>
-(define-test ...+(in-package :tut-tests) 
 + 
 +;;; Turtle world is like this: 
 +;;;  Y 
 +;;;  ^ 
 +;;;  | 
 +;;;  | 
 +;;;  o-----> X 
 +;;; Identity orientation is looking right in the direction of X axis. 
 + 
 +(define-test relative-angle-to-dont-turn 
 +  ;; turtle is in origin with identity orientation, wants to look at (3 0 0) 
 +  (let* ((goal (cl-transforms:make-3d-vector 3 0 0)) 
 +         (pose-msg (roslisp:make-message 'turtlesim-msg:pose)) 
 +         (angle (tut::relative-angle-to goal pose-msg))) 
 +    (assert-number-equal angle 0.0)))
 </code> </code>
  
 +So, we are testing the ''relative-angle-to'' function, which returns an angle, with which the turtle has to turn,
 +in order to look towards a goal point.
  
 +What does the test checks the following situation:
 +if the turtle is standing in the origin of the coordinate frame of the Turtlesim world,
 +looking in the identity orientation, i.e., to the right in the direction of the X axis,
 +what would be the angle with which it would have to turn to look at coordinate ''(3 0)''.
 +As ''(3 0)'' lies on the X axis, and the turtle is already looking in that direction, it would have to move with the angle of ''0.0''. And that is what the test asserts.
  
 +Let us add just another test case: if the turtle is standing at the origin with identity orientation
 +and wants to look at ''(-1 0)'', what would be the angle?
 +Let us create a new test function for this, such that your ''simple-plans-tests.lisp'' would look as following:
 +
 +<code lisp>
 +(in-package :tut-tests)
 +
 +;;; Turtle world is like this:
 +;;;  Y
 +;;;  ^
 +;;;  |
 +;;;  |
 +;;;  o-----> X
 +;;; Identity orientation is looking right in the direction of X axis.
 +
 +(define-test relative-angle-to-dont-turn
 +  ;; turtle is in origin with identity orientation, wants to look at (3 0 0)
 +  (let* ((goal (cl-transforms:make-3d-vector 3 0 0))
 +         (pose-msg (roslisp:make-message 'turtlesim-msg:pose))
 +         (angle (tut::relative-angle-to goal pose-msg)))
 +    (assert-number-equal angle 0.0)))
 +
 +(define-test relative-angle-to-look-back
 +  ;; turtle is in origin, wants to look at (-1 0 0)
 +  (let* ((goal (cl-transforms:make-3d-vector -1 0 0))
 +         (pose-msg (roslisp:make-message 'turtlesim-msg:pose))
 +         (angle (tut::relative-angle-to goal pose-msg)))
 +    (assert-number-equal angle pi)))
 +</code>
 +
 +You can see that we needed to use the ''cl-transforms'' library for the test, as well as ''roslisp'' and ''turtlesim-msg''.
 +So let us add these to the ''.asd'' file of our test suite. We should also add the new files that we have created, ''package.lisp'' and ''simple-plans-tests.lisp'' to the ASDF system. It should now look something like this:
 +
 +<code lisp>
 +(defsystem cram-my-beginner-tutorial-tests
 +  :depends-on (cram-my-beginner-tutorial
 +               lisp-unit
 +               cl-transforms
 +               roslisp
 +               turtlesim-msg)
 +  :components ((:module "tests"
 +                :components
 +                ((:file "package")
 +                 (:file "simple-plans-tests" :depends-on ("package"))))))
 +</code>
 +
 +
 +===== Running the test suite =====
 +
 +Now let us load the test package in REPL and switch into its namespace.
 +You might need to restart your Emacs at this point, because the new ASDF system might not be visible to Emacs without restarting.
 +
 +<code lisp>
 +CL-USER> (ros-load:load-system "cram_my_beginner_tutorial" :cram-my-beginner-tutorial-tests)
 +CL-USER> (in-package :tut-tests)
 +</code>
 +
 +To run the tests written with the ''lisp-unit'' library, we call the ''run-tests'' function:
 +
 +<code lisp>
 +TUT-TESTS> (lisp-unit:run-tests)
 +
 +Unit Test Summary
 + | 2 assertions total
 + | 2 passed
 + | 0 failed
 + | 0 execution errors
 + | 0 missing tests
 +
 +#<TEST-RESULTS-DB Total(2) Passed(2) Failed(0) Errors(0)>
 +</code>
 +
 +We can also specify which test specifically we want to run:
 +
 +<code lisp>
 +TUT-TESTS> (lisp-unit:run-tests '(relative-angle-to-dont-turn))
 +Unit Test Summary
 + | 1 assertions total
 + | 1 passed
 + | 0 failed
 + | 0 execution errors
 + | 0 missing tests
 +
 +#<TEST-RESULTS-DB Total(1) Passed(1) Failed(0) Errors(0)>
 +</code>
 +
 +If we are in a different package in the REPL than the test suite package, we can also specify the package to test explicitly:
 +
 +<code lisp>
 +TUT-TESTS> (in-package :cl-user)
 +#<PACKAGE "COMMON-LISP-USER">
 +CL-USER> (lisp-unit:run-tests :all :cram-my-beginner-tutorial-tests)
 +Unit Test Summary
 +...
 +</code>
 +
 +==== Integrating into the ASDF testing operation ====
 +
 +As there exist multiple testing libraries in Lisp, the ASDF facility has a unified way of calling the test suites, independent of the library.
 +So let us integrate our test suite with the ASDF interface.
 +For that, we are going to add a '':perform'' tag to our ''cram-my-beginner-tutorial-tests.asd'' file, such that it now should look as following:
 +
 +<code lisp>
 +(defsystem cram-my-beginner-tutorial-tests
 +  :depends-on (cram-my-beginner-tutorial 
 +               lisp-unit
 +               cl-transforms
 +               roslisp
 +               turtlesim-msg)
 +  :components ((:module "tests"
 +                :components
 +                ((:file "package")
 +                 (:file "simple-plans-tests" :depends-on ("package")))))
 +  :perform (test-op (operation component)
 +                    (symbol-call :lisp-unit '#:run-tests :all :cram-my-beginner-tutorial-tests)))
 +</code>
 +
 +And to use the ASDF interface for testing, we simply need to call the ''asdf:test-system'' function on our package.
 +We can do that at any point in the REPL. So let us start a completely fresh REPL with nothing loaded in it and try the testing function:
 +
 +<code lisp>
 +CL-USER> (asdf:test-system :cram-my-beginner-tutorial-tests)
 +T
 +</code>
 +
 +You will get some compilation prompts in between and in the end it should return T.
 +Now let us destroy one of our tests and see what happens: change the angle in the first test in ''simple-plans-tests.lisp'' from 0.0 to pi:
 +
 +<code lisp>
 +(define-test relative-angle-to-dont-turn
 +  ;; turtle is in origin with identity orientation, wants to look at (3 0 0)
 +  (let* ((goal (cl-transforms:make-3d-vector 3 0 0))
 +         (pose-msg (roslisp:make-message 'turtlesim-msg:pose))
 +         (angle (tut::relative-angle-to goal pose-msg)))
 +    (assert-number-equal angle pi)))
 +</code>
 +
 +Now let us rerun the tests:
 +
 +<code lisp>
 +CL-USER> (asdf:test-system :cram-my-beginner-tutorial-tests)
 +
 +Unit Test Summary
 + | 2 assertions total
 + | 1 passed
 + | 1 failed
 + | 0 execution errors
 + | 0 missing tests
 +
 +T
 +</code>
 +
 +To get more detailed information, set the ''*print-failures*'' parameter of ''lisp-unit'' to T:
 +
 +<code lisp>
 +CL-USER> (setf lisp-unit:*print-failures* t)
 +T
 +CL-USER> (asdf:test-system :cram-my-beginner-tutorial-tests)
 + | Failed Form: PI
 + | Expected 0.0d0 but saw 3.141592653589793d0
 + |
 +RELATIVE-ANGLE-TO-DONT-TURN: 0 assertions passed, 1 failed.
 +
 +RELATIVE-ANGLE-TO-LOOK-BACK: 1 assertions passed, 0 failed.
 +
 +Unit Test Summary
 + | 2 assertions total
 + | 1 passed
 + | 1 failed
 + | 0 execution errors
 + | 0 missing tests
 +
 +#<TEST-RESULTS-DB Total(2) Passed(1) Failed(1) Errors(0)>
 +</code>
 +
 +Now go back and fix the angle to the correct value and see if the tests pass again.
 +
 +
 +===== Adding some more assertions =====
 +
 +Now let us add more tests for the ''simple-plans.lisp'' file into our suite.
 +Let's add some more tests for the ''relative-angle-to'' to check if the number are correct with ''assert-number-equal'' function.
 +In addition to that assertion, there are also many different types of assertions, so we are going to try the ''assert-error'' one.
 +Put the following into your ''simple-plans-tests.lisp'':
 +
 +<code lisp>
 +(in-package :tut-tests)
 +
 +;;; Turtle world is like this:
 +;;;  Y
 +;;;  ^
 +;;;  |
 +;;;  |
 +;;;  o-----> X
 +;;; Identity orientation is looking right in the direction of X axis.
 +
 +(define-test relative-angle-to-dont-turn
 +  ;; turtle is in origin with identity orientation, wants to look at (3 0 0)
 +  (let* ((goal (cl-transforms:make-3d-vector 3 0 0))
 +         (pose-msg (roslisp:make-message 'turtlesim-msg:pose))
 +         (angle (tut::relative-angle-to goal pose-msg)))
 +    (assert-number-equal angle 0.0))
 +
 +  ;; turtle is in (3 0 0) with identity orientation, wants to look at (4 0 0)
 +  (let* ((goal (cl-transforms:make-3d-vector 4 0 0))
 +         (pose-msg (roslisp:make-message 'turtlesim-msg:pose
 +                                         :x 3.0 :y 0.0 :theta 0.0))
 +         (angle (tut::relative-angle-to goal pose-msg)))
 +    (assert-number-equal angle 0.0))
 +
 +  ;; turtle is in (0 0 0) looking up in direction of Y, wants to look at (0 1 0)
 +  (let* ((goal (cl-transforms:make-3d-vector 0 1 0))
 +         (pose-msg (roslisp:make-message 'turtlesim-msg:pose
 +                                         :x 0.0 :y 0.0 :theta (/ pi 2)))
 +         (angle (tut::relative-angle-to goal pose-msg)))
 +    (assert-number-equal angle 0.0))
 +
 +  ;; turtle is in (1 1 0) looking towards the origin, wants to look at (0 0 0)
 +  (let* ((goal (cl-transforms:make-3d-vector 0 0 0))
 +         (pose-msg (roslisp:make-message 'turtlesim-msg:pose
 +                                         :x 1.0 :y 1.0 :theta (+ pi (/ pi 4))))
 +         (angle (tut::relative-angle-to goal pose-msg)))
 +    (assert-number-equal angle 0.0)))
 +
 +(define-test relative-angle-to-look-back
 +  ;; turtle is in origin, wants to look at (-1 0 0)
 +  (let* ((goal (cl-transforms:make-3d-vector -1 0 0))
 +         (pose-msg (roslisp:make-message 'turtlesim-msg:pose))
 +         (angle (tut::relative-angle-to goal pose-msg)))
 +    (assert-number-equal angle pi))
 +
 +  ;; turtle is in (1 0 0), wants to look at origin
 +  (let* ((goal (cl-transforms:make-3d-vector 0 0 0))
 +         (pose-msg (roslisp:make-message 'turtlesim-msg:pose
 +                                         :x 1.0 :y 0.0 :theta 0.0))
 +         (angle (tut::relative-angle-to goal pose-msg)))
 +    (assert-number-equal angle pi))
 +
 +  ;; turtle is in (-1 -1 0) looking towards origin, wants to look away from origin
 +  (let* ((goal (cl-transforms:make-3d-vector -2 -2 0))
 +         (pose-msg (roslisp:make-message 'turtlesim-msg:pose
 +                                         :x -1.0 :y -1.0 :theta (/ pi 4)))
 +         (angle (tut::relative-angle-to goal pose-msg)))
 +      (assert-number-equal angle pi)))
 +
 +(define-test relative-angle-to-type-error
 +  (assert-error 'type-error 
 +                (tut::relative-angle-to 
 +                 (cl-transforms:make-identity-pose)
 +                 (roslisp:make-message 'turtlesim-msg:pose))))
 +</code>
  
 +At this point, all the tests should be successful.
  
 +Please beware that if you define a test with a specific name, then delete it from the file but not restart the REPL, the test will still be in the current REPL session. So if you delete a test from a file, also think about restarting your REPL.