Results of the RERS 2012 Challenge

The RERS 2012 Challenge has ended during ISoLA 2012. On this page, we present the results of all five contestants. In order to provide a more fine-grained view on the results, we offer the possibility to customize the evaluation process, allowing visitors to investigate the specific strengths of the competing approaches.

Results

RankNameScoreTotal answersWrong answers% correct
1Twente2519024375297.9%
2Paris1310414861099.3%
3Vienna84339383696.2%
4Passau58235980100.0%
5Southampton50617051697.7%

List of participants

  • Passau
    Dirk Beyer, Andreas Stahlbauer, Peter Häring
    University of Passau, Germany
  • Twente
    Jaco van de Pol
    University of Twente, Netherlands
  • Vienna
    Markus Schordan, Adrian Prantl
    Technikum Wien, Austria / Lawrence Livermore National Laboratory, USA
  • Southampton
    Jeremy Morse, Lucas Cordeiro, Denis Nicole, Bernd Fischer
    University of Southampton, UK / Federal University of Amazonas, Brazil
    Offline participation only
  • Paris
    Pascal Cuoq, Virgile Prevosto, Boris Yakobowski
    CEA LIST, France
    Offline participation only

Customize evaluation

The following form allows you to customize the evaluation of the results, imposing other rules than the official ones. This is offered due to our idea of providing a profile of the tool landscape, providing a more fine-grained overview on the specific strengths and weaknesses of the contestant's tools.

The following values can be changed:

  • Property types: Restrict evaluation to only reachability or behavioral (LTL) properties, or take both into account.
  • Problem selection: Only include certain problems for the evaluation.
  • Respect confidence: If checked, then the confidence values supplied by the contestants are taken into account. If not checked, than confidence 1 is assumed for all answers.
  • Penalty factor: By default, wrong answers are penalized by subtracting twice the confidence value from the total score. Enter another number to change this factor.
  • Double score for online problems: The score for the online problems 14 through 19 counts twice in the official evaluation. Uncheck to disable.

Property types
 Reachability
 LTL

Problem selection
Original problems:
 Problem 1  Problem 2  Problem 3  Problem 4  Problem 5  Problem 6 
 Problem 7  Problem 8  Problem 9 

Additional problems:
 Problem 12  Problem 13 

Online problems:
 Problem 14  Problem 15  Problem 16  Problem 17  Problem 18  Problem 19 
Respect confidence

Penalty factor

Double score for online problems