Reval is an open-source framework to evaluate the performance of Robotics platforms. Currently it supports Husky platform, Turtblebot3. The useres can evalute the performance of a mission for a given gazebo envirnoment (or on their own gazebo envirnment) for different configurations in an automated fashion and log the results. In addition, Reval supports the following metrics to evaluate the quality of a mission:

Evaluation metrics

Metrics	Description
DWA F	# of failed produced path by DWA planner
DWA NP	# of re-planning by DWA planner
DWA IT	# of DWA invalid trajectory
RR	# of rotate recovery excuted
RCU	# of ClearCostMaps recovery executed for unstuck robot
RCL	# of ClearCostMaps layer recovery executed
IRC	# of invalid rotation cmd
ERG	# of error rotating on the goal
DRMS	The square root of the average of the squared horizontal position errors, ; where standard deviation of the delta x and y, . Probability of 65%
2DRMS	Twice the Distance Root Mean Squared (DRMS) of the horizontal position error, . Probability of 95%
CPE	The radius of circle centered at the true position, containing the position estimate with probability of 50%.
ED	Euclidean distance between actual goal location and the location the robot reached.
RNS	robustness in narrow spaces, ; where Ns is the total narrow spaces in the gazebo environment, and passed_Ns is the narrow spaces that the robot successfully crossed.
DT	total distance traveled during a mission
BP	battery percentage. For more details: Gazebo-ROS battery plugin
Col	number of collisions in a mission
MT	time taken to complete a mission
MS	mission success. Example: if the robot successfully reached point A to B

The instructions provided below are for Turtlebot3. Follow the initial setup for Turtlebot3.

Build Type	Status
ROS melodic
ROS noetic

Platform	Status
Husky UGV
TurtleBot3
OceanWATERS

• Ubuntu 20
• ROS Noetic
• Python 3.6+

sudo apt remove ros-noetic-dynamixel-sdk
sudo apt remove ros-noetic-turtlebot3-msgs
sudo apt remove ros-noetic-turtlebot3
git clone -b turtlebot3 https://github.com/softsys4ai/Reval.git
mkdir -p ~/Reval/src/turtlebot3
git clone -b noetic-devel https://github.com/ROBOTIS-GIT/DynamixelSDK.git
git clone -b noetic-devel https://github.com/ROBOTIS-GIT/turtlebot3_msgs.git
git clone -b noetic-devel https://github.com/ROBOTIS-GIT/turtlebot3.git
cd ~/Reval && catkin build

Download the file

wget https://raw.githubusercontent.com/ElectricRCAircraftGuy/eRCaGuy_dotfiles/master/useful_scripts/ros_readbagfile.py

Make it executable

chmod +x ros_readbagfile.py

Ensure you have the ~/bin directory for personal binaries

mkdir -p ~/bin

Move this executable script into that directory as ros_readbagfile, so that it will be available as that command

mv ros_readbagfile.py ~/bin/ros_readbagfile

Create a symlink in ~/bin to this script so you can run it from anywhere:

ln -si "${PWD}/ros_readbagfile.py" ~/bin/ros_readbagfile

If this is the first time ever creating the "~/bin" dir, then log out and log back in to your Ubuntu user account to cause Ubuntu to automatically add your ~/bin dir to your executable PATH.

Re-source your ~/.bahsrc file

source ~/.bashrc

Installing the dependencies

sudo apt install ripgrep
pip install pandas
pip install tqdm
pip install tabulate 

Or cd Reval/ run ./requirements.sh. If you face permission denied, first run chmod +x requirements.sh

cd Reval/

source your new setup.sh file. You need source this setup.sh file everytime you open a new Terminal

source devel/setup.bash

To evaluate the mission run

python reval.py

• To define your custom configuration options: Set Configuration
• To change the goal locations: Update Goal location

Please feel free to contact via email if you have any feedbacks. Thank you for using Reval!