Description: In this tutorial you will learn..

Requirements:

• Unreal Engine version 2.25.4/2.26/2.27.
  • Install Guide
  • Unreal requires a lot of disc space, at least 60 GB for the engine itself, and with several projects even more.
  • Making the Unreal engine takes a lot of computation time. Schedule it adequately.
  • Either:
    1. Check the installation of Unreal by launching the engine under <unreal-path>/Engine/Binaries/Linux/UE4Editor. Since it requires OpenGL or Vulkan, running it from a remote machine can get finicky.
    2. Or use Rider for Unreal Engine
  • Clone the Demo Project:

     git clone --recurse-submodules git@github.com:urobosim/DemoProject.git

  • For further information how to Setup the Demo Project please take a look into this README.

To communicate with the robot in the Unreal Engine from host-machine additional components are required, including the host-side rosbridge, shared ROS messages and descriptions. First, install the following packages for communication via WebSockets, the TF2 buffer, kinematic models and a state publisher.

sudo apt install ros-melodic-rosbridge-server ros-melodic-robot-state-publisher ros-melodic-joint-state-publisher-gui ros-melodic-tf ros-melodic-tf2 ros-melodic-tf2-ros ros-melodic-pr2-arm-kinematics

Further software stacks are needed to control the robot in Unreal via CRAM plans and Giskards motion planner, as well as perception and knowledge inference and logging.

Follow the installation instructions for each of the following software stacks, picking the ones required for your specific purpose. Using separate, overlayed workspaces is recommended.

• Install Giskard, the motion planner.
• Install Knowrob, the long-term memory, especially for NEEMs
• Install CRAM, the noetic branch, please follow the Github-ReadMe.

Now create a catkin workspace for deps from repositories. Use wstool for git imports and rosdep for apt sources.

mkdir -p ~/unreal_project_ws/src
cd ~/unreal_project_ws/src
wstool init
wstool merge https://raw.githubusercontent.com/urobosim/DemoProject/master/rosinstall/unreal_demo_project.rosinstall
wstool up
rosdep update
rosdep install --from-paths --ignore-src . -r

Build the workspace either with the ros-native catkin or with python-catkin-tools, which can be added via apt.

cd ~/unreal_project_ws
catkin_make                                  # or 'catkin build'
source ~/unreal_project_ws/devel/setup.bash  # rather put the top-level workspace in your .bashrc

If the build process wasn't successful, install the missing packages via apt, find missing repositories in the code-iai GitHub group or ask the contributors for help.

Finally, the connection to the ROS network can be established by launching the following nodes. Make sure the workspace is sourced.

#Launch rosbridge for communication between unreal and ROS
roslaunch rosbridge_server rosbridge_websocket.launch 
 
# Launch urobosim world
roslaunch urobosim_ros_config world.launch

Hit the 'Play' button in Unreal. The rosbridge terminal should indicate, that the connection has been established. Also use rostopic list to check, if the controller topics for the robot are published.

Giskard requires the whole_body_controller topics of the robot. To achieve that, make sure the rosbridge to Unreal is running properly.

roslaunch giskardpy giskardpy_pr2_unreal.launch

This launches Giskards trajectory planner. There may occur one timeout error for the state topic, which is fine for now.

Launch RViz by executing rviz. Add the Robot Model to the scene. Add InteractiveMarkers and choose the topic; if /eef_control/update is not available, check the giskard setup again. Use the markers to move the robot and see the robot in Unreal move.

cram-unreal provides

• A :detecting PM and plan to detect objects from Unreal to put them in Bullet
• The service clients to spawn and move objects in a running Unreal world (and stubs to extend to further clients)
• Some unused plans for pouring (feeding-plans)

cram-pr2-unreal-process-modules (aka pr2-unreal-pms) provides

• An extension to pr2-pms as
• The with-unreal-robot macro uses unreal-detection while using the same manipulation pms as pr2-pms:with-real-robot
• Also a few changes to Giskard's constraints, which seemed to be outdated. Giskard won't work without them.

Note: Remove CATKIN_IGNORE in the cram_pr2 packages and cram_external_interfaces/cram_urobosim to build them.

Play the map in Unreal. The PR2 should twitch and wiggle its arms a bit if it's good. If not, check the DemoProject repo ReadMe how to troubleshoot.

Startup roslisp_repl. First load cram_pr2_pick_place_demo, then cram_pr2_unreal_process_modules. Start a rosnode with (roslisp-utilities:startup-ros). To start the setting demo, run

(pr2-unreal-pms:with-unreal-robot
   (demo::setting-demo '(:spoon)))

Run the same launch files as above, but without giskard. Open Emacs, load cram_urobosim and start a node. Try to spawn an object with

(unreal:spawn-object :spoon (cl-tf:make-identity-pose))