Track: Benchmarking and Competitions
Date and Time: 13 Mar 2024 12:00-13:20
Organisers: Fabio Bonsignorio, Matthew Studley, Diego Torricelli, Pedro Lima
Main questions to be answered: Reproducibility, benchmarking, standards
Additional questions: Objective measures of robustness, reliability and agility
Additional questions 2: TRL roadmap for new products
Workshop Description: The reproducibility of research results and the objective and operational comparison of different methods is still not mainstream in Robotics, AI and Automation Research. However, in particular thanks to the EU Commission significant progress has been done, This will be a gathering of the TG on Benchmarking and Competions. The SotA will be presented and the road ahead discussed. Info about currently completed and starting projects and results willl be shared. The meeting is open to all.
Intended Outcome: Workshop on recent developments in technology or applications, Workshop expected to create a roadmap or white paper, Workshop discussion topics of common interest, success stories, use cases, etc, Workshops engaging non-roboticists to understand needs/goals (policy, commercial, technical), ‘The RIMINI DECLARATION’ A 5 pages documente summarising the actions to be undertaken to achieve mainstream reproducible and measurable robotics research
Approach: Trustable and Dependable Intelligent Robots for Smart Societies Subtitle: Research Competitions, Benchmarking, Societal and Ethical Issues This is a two-parts workshop. This is workshop is a follow up to the first very successful edition we had in 2023 in Odense during ERF 2023. We propose the following: • Before the workshop we summarize in a 5 pages document – the ‘Rimini Declaration Reproducibility of Research and Benchmarking in Robotics’-the insights we gathered from 2023 edition of the workshop • We then gather opinions about this document from the industrial and research community, and from the communities involved in funding, legal standards, etc (as far as this is possible and reasonable) • We will use Mentimeter or a similar tool to promote synthesis of knowledge from these opinions, and build an emerging consensus • We will finalize the document after the workshop • The workshop will consist of a series of short provocations – briefings by panellists on topics linked to this. There will then be a round table discussion of the community consensus to finalise the document . • After the workshop, the final ‘Rimini Declaration’ will be published. Before the workshop we will spread by mailing lists, social networks etc. a number of crucial thought provoking and outright provocative questions and challenges to the community with the objective of raising awareness and stimulating contributions on the topic. Thought provoking video from recent robotics competitions will be shared before and during the event. Hybrid Event Agenda – first 80 mn: – intro + summary of the Declaration (MS): 15 mn – 10 pitches by invited speakers of 5 mn each + spare time: 65 mn (presentations + Q&A) – second 80 mn: Discussion and work on the document Follow up Insight session: the scientific issues still open will be addressed by 7–10 accepted peer-reviewed contributions We are working to coordinate this workshop with the Challenges that will be organized during ERF 2024 to the design of which we are contributing.
Contributors: We have a long list: Matteo Matteucci, Enrica Zereik, Davide Scaramuzza, Agnes Delaborde, Isabel Ferreira, Tony Downs (NIST),Ricardo Sanz, Antidio Viguria Jiménez, Alin Albu-Schäffer , Francesco Ferro