Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
tutorials:advanced:unreal [2020/01/13 13:24] – [Usage and Code] Formatting hawkintutorials:advanced:unreal [2020/06/11 11:54] (current) – [Prerequisites] adding the source to the Episode Data/VR data hawkin
Line 5: Line 5:
 This tutorial will introduce you to the ''cram_knowrob_vr (short: kvr)'' package, which uses the data recorded in the Virtual Reality environment using [[http://robcog.org/|RobCog]], extracts information from them using [[http://www.knowrob.org/|KnowRob]] and executes the CRAM high-level-plans based on this data either on the real robot or in the CRAM [[tutorials:advanced:bullet_world|bullet world]]. This tutorial will introduce you to the ''cram_knowrob_vr (short: kvr)'' package, which uses the data recorded in the Virtual Reality environment using [[http://robcog.org/|RobCog]], extracts information from them using [[http://www.knowrob.org/|KnowRob]] and executes the CRAM high-level-plans based on this data either on the real robot or in the CRAM [[tutorials:advanced:bullet_world|bullet world]].
  
 +==== Idea ====
 +We want to essentially teach a robot how to perform every day activities without having to dive deep into code, but rather simply show the robot what we want him to do using Virtual Reality. This would allow robots to learn from humans easily, since this way, the robot can acquire information about where the human was looking for things and where things were placed. Of course, this could also be hard coded, but that would take a lot more time and be prone to failure, since we as humans, often forget to describe very minor things which we automatically take for granted, but which can play a huge role in the success of a task for the robot. E.g. if the cooking pot is not in its designated area, we would automatically check the dishwasher or the sink area. This is something the robot would have to learn first.
 +
 +The advantage of using Virtual Reality for this is also, that we can train the robot on all kinds of different kitchen setups, which can be build within a few minutes, instead of having to move around physical furniture. This would also allow for generalization of the acquired data and would add to the robustness of the pick and place tasks. 
 ==== Prerequisites ==== ==== Prerequisites ====
 This tutorial assumes that you've completed the [[tutorials:intermediate:json_prolog|Using JSON Prolog to communicate with KnowRob]] tutorial and therefore have ROS, CRAM, KnowRob and MongoDB installed. In order to be able to use the kvr package, a few specific changes have to be made. Within the ''knowrob_addons'' the ''knowrob_robcog'' package has to be replaced by this one [[https://github.com/robcog-iai/knowrob_robcog.git This tutorial assumes that you've completed the [[tutorials:intermediate:json_prolog|Using JSON Prolog to communicate with KnowRob]] tutorial and therefore have ROS, CRAM, KnowRob and MongoDB installed. In order to be able to use the kvr package, a few specific changes have to be made. Within the ''knowrob_addons'' the ''knowrob_robcog'' package has to be replaced by this one [[https://github.com/robcog-iai/knowrob_robcog.git
 |knowrob_robcog github]]. If it doesn't build because dependencies are missing, please install them. If it still doesn't build, you can try to pull this fork of the knowrob_addons [[https://github.com/hawkina/knowrob_addons|knowrob_addons github]] instead. |knowrob_robcog github]]. If it doesn't build because dependencies are missing, please install them. If it still doesn't build, you can try to pull this fork of the knowrob_addons [[https://github.com/hawkina/knowrob_addons|knowrob_addons github]] instead.
 The ''knowrob_robcog'' package contains the VR-specific queries, which we need in order to be able to extract the information we need for the plans. The ''knowrob_robcog'' package contains the VR-specific queries, which we need in order to be able to extract the information we need for the plans.
 +
 +Please also download the **episode data** provided at this [[https://seafile.zfn.uni-bremen.de/d/ba524dbadfa748488d4d/|source]]. The download can take about 10 minutes, depending on your internet connection. Unzip the archive and put the **episodes** directory next to your work space. It is it's own workspace, and is ment to be layered under your working workspace. To learn more about workspace layering and on how to set it up, please refer to this [[http://wiki.ros.org/catkin/Tutorials/workspace_overlaying|tutorial]]. 
  
 === Roslaunch === === Roslaunch ===
Line 34: Line 40:
 </code> </code>
  
-To launch all the necessary components, simply execute: +If this is used in simulation and depending on if the PR2 or Boxy robot is supposed to be used, the respective description needs to be loaded first. 
 + 
 +For PR2: 
 +<code lisp> 
 +CL-USER>  (swank:operate-on-system-for-emacs "cram-pr2-description" (quote load-op)) 
 +</code> 
 + 
 +For Boxy: 
 +<code lisp> 
 +CL-USER>  (swank:operate-on-system-for-emacs "cram-boxy-description" (quote load-op)) 
 +</code> 
 + 
 + 
 +To launch all the necessary initializations, simply execute: 
 <code lisp> <code lisp>
 CL-USER> (kvr::init-full-simulation :namedir '("ep1") :urdf-new-kitchen? nil) CL-USER> (kvr::init-full-simulation :namedir '("ep1") :urdf-new-kitchen? nil)
 </code> </code>
  
-This will create a lisp ros node, clean up the belief-state, load the episodes that get passed to the init function as a list of strings in the ''namedir'' key parameter, e.g. in our case "ep1", spawn the semantic map of the episode and the items and initialize the location costmap. This process may take a while, so please have some patience. When the function has finished running through your bullet world should look like this: +This will create a lisp ros node, clean up the belief-state, load the episodes that get passed to the ''init'' function as a list of strings in the ''namedir'' key parameter, e.g. in our case "ep1", spawn the semantic map of the episode and the items and initialize the location costmap. This process may take a while, so please have some patience. (or go grab a coffee meanwhile. It can really take several minutes on Ubuntu 16.04.) When the function has finished running through your bullet world should look like this: 
  
 Now, let's execute the pick and place plan: Now, let's execute the pick and place plan:
Line 72: Line 91:
 == debugging-utils.lisp == == debugging-utils.lisp ==
 Contains a lot of debugging and helper functions which can also visualize all the calculated poses.  Contains a lot of debugging and helper functions which can also visualize all the calculated poses. 
-==== Importing new episode data into MongoDB and KnowRob(Additional information) ==== +==== Importing new episode data into MongoDB and KnowRob (Additional information)==== 
-In order for us to be able to query data for information, we first need to import that data into KnowRob and MongoDB.+In order for us to be able to query episode data for information, we first need to import that data into KnowRob and MongoDB.
 The MongoDB contains all the poses of the objects, camera, hand and furniture of the episode. So the pose is read out from there. KnowRob takes care of everything else. It knows of all the object classes and instances used in the episodes. So in order to be able to use the data, we need to import it into MongoDB and KnowRob. The MongoDB contains all the poses of the objects, camera, hand and furniture of the episode. So the pose is read out from there. KnowRob takes care of everything else. It knows of all the object classes and instances used in the episodes. So in order to be able to use the data, we need to import it into MongoDB and KnowRob.
 +The following description only applies if the data has been recorded using the [[http://robcog.org/ | RobCog]] tool for Unreal Engine, which results in a ''json'' file containing the events and a matching ''SemanticMap.owl'' describing the environment.
 +
 +=== MongoDB (quick with scripts)===
 +There is a script which imports the episode data into the currently running MongoDB instance. Please see [[https://github.com/hawkina/useful_scripts|scripts]] and the provided Readme for reference and usage.
 +
 +=== MongoDB (manuel in-depth) ===
 +This will explain how to import the episode data manually into MongoDB. This is essentially what the script in the description above does automatically. So if the script didn't work (please leave an issue on [[https://github.com/hawkina/useful_scripts| github]] in that case), you can follow this guide.
  
-=== MongoDB === 
 If you record data in the Virtual Reality using [[http://robcog.org/ | RobCog]], you essentially get the following files: If you record data in the Virtual Reality using [[http://robcog.org/ | RobCog]], you essentially get the following files:
 ''RawData_ID.json'' <- this contains all the recorded events that happened, where everything was, who moved what where when...etc. ''RawData_ID.json'' <- this contains all the recorded events that happened, where everything was, who moved what where when...etc.
Line 128: Line 153:
  
 === KnowRob === === KnowRob ===
-KnowRob needs to be able to find the .owl and .bson files. It's important to have a directory for all the Episodes. You can either have a look or download these [[ftp://open-ease-stor.informatik.uni-bremen.de/|episodes]] for reference. +KnowRob needs to be able to find the .owl and .bson files. It's important to have a directory for all the Episodes. You can either have a look or download these [[ftp://open-ease-stor.informatik.uni-bremen.de/|episodes]] for reference. Keep in mind, that depending on which episode data should be loaded, the path within CRAM (the ''*episode-path*'' variable within the ''init.lisp'') might have to be adapted, as well as the parameter which is passed to the ''init-full-simulation'' function.
  
 === Performance === === Performance ===
 +This step is also covered by the [[https://github.com/hawkina/useful_scripts | scripts]] mentioned above, but can also be executed manually. 
 +
 Depending on how many collections your database has, it can get slow when quering for information. One way to make it faster, is to include an index over timestamp for all collections. One way to add this, is to install [[https://www.mongodb.com/products/compass|compass]] for mongodb. Launch it, and connect it to your database. The defautl settings should be fine so just click ''ok'' when it launches. Then go to your collection -> indexes -> create index. Call the new index ''timestamp'', select the field ''timestamp'' and set the type to ''1 (asc)'', click create. Repeat for all the collections. It will improve the query speed greatly. Depending on how many collections your database has, it can get slow when quering for information. One way to make it faster, is to include an index over timestamp for all collections. One way to add this, is to install [[https://www.mongodb.com/products/compass|compass]] for mongodb. Launch it, and connect it to your database. The defautl settings should be fine so just click ''ok'' when it launches. Then go to your collection -> indexes -> create index. Call the new index ''timestamp'', select the field ''timestamp'' and set the type to ''1 (asc)'', click create. Repeat for all the collections. It will improve the query speed greatly.