The basic terminology, problem domains, and methods of evaluation are defined normatively in the Challenge Document. The sources are available under doc/norm/ in the fmrbenchmark repository. The two other major forms of documentation are the User's Guide and the API manual.
Get the current release, which corresponds to version 0.0.4 of the fmrbenchmark repository.
(For convenience, this section is included here from the current release of the Challenge Document.)
A team is the basic entity that can compete and, depending on performance, receive rankings in the Challenge. A team must have a name by which it will be referred to during the Challenge.
There are three problem domains. Within a problem domain, a problem instance is a particular workspace, arrangement of obstacles, labeling of the workspace, selection of parameters for the robot dynamics, and a task formula.
Each problem domain may have one or more variants, which concern different manners of implementation or random instance generation. E.g., the traffic of Dubins cars domain has a simulation variant, which relies on Gazebo, and a physical variant, which involves a real testbed of several Kobuki bases as well as an overhead pose tracking system.
A controller is the basic entity that a team submits for the Challenge within a particular problem domain. Each controller is scored and ranked, and the score of a team is simply the maximum score of any controller belonging to that team.