BBSR - Benchmarking, Benchmarks, Software, and Reproducibility
Description
The Benchmarking, Benchmarks, Software, and Reproducibility (BBSR) track welcomes submissions that touch on all aspects of benchmarking, reproducibility, and software of genetic and evolutionary computation methods.
Scope:
- Benchmarking methodologies for assessing the performance and behavior of evolutionary algorithms and related optimization techniques,
- Benchmark problems, generators, and toolboxes for evaluating evolutionary computation methods or enabling the training of meta-learning techniques for these,
- Statistical analysis and visualization techniques for understanding problem spaces or the performance and behavior of optimization techniques, including instance space analysis and landscape analysis,
- Reproducibility studies that rigorously replicate published experiments with a substantial shift in confidence in the results of the original study,
- Innovative software and toolboxes for developing, evaluating, deploying, or teaching genetic and evolutionary computation in original and unique ways.
This is a non-exhaustive list, and we invite the authors to get in touch with the track chairs if in doubt about the suitability of their submission to this track.
Requirements for reproducibility studies
To promote reproducible research, the BBSR track encourages submissions to provide a reference to a repository containing any artifacts (software, data, etc) needed to reproduce the results presented in the paper. We strongly encourage authors to follow the guidelines for the ACM reproducibility standard of 'artifacts functional', for which the details can be found here: https://www.acm.org/publications/policies/artifact-review-and-badging-current.
For reproducibility studies, the authors should clearly motivate their decision to replicate the original study. In addition, the reasons for the new findings must be clearly explained in order to ensure a meaningful and distinct contribution from the original study (e.g., different benchmarks, application scenarios, technical or implementation differences). The submission must follow the highest reproducibility standards by providing all implementation details, input data, parameters and hardware specifications. All artifacts must be made available in a public repository upon submission and must remain available after publication. The submission must also follow the usual standards in terms of plagiarism.
Of particular interest are replicability studies, defined as follows by the
ACM: "The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials (different team, different experimental setup)."
Anonymization
We acknowledge that for some of the works fitting this track, it may be difficult to submit in completely anonymized form, e.g., when links to demos, data, or software are required to assess the suitability of the submission for GECCO. Whenever possible, we strongly encourage the authors to make use of anonymous repositories (available on Zenodo and for GitHub repositories, for example). In the ideal case, these repositories will be deanonymized only after the notification. Where it is impossible to anonymize repositories – and only in those cases – the BBSR track allows to link resources that possibly reveal authors’ identity. However, all other elements of the paper shall follow the standard anonymization guidelines. In particular, we require that author names, affiliations, and acknowledgments are suppressed and that, to the maximum extent possible, references to any of the author's own work should be made as if the work belonged to someone else. We strongly recommend the use of the following option:
\documentclass[dvipsnames,format=sigconf,anonymous=true,review=true]{acmart}
Track Chairs
Diederick Vermetten
Sorbonne Université, CNRS, LIP6
Diederick Vermetten is a Post-Doc at Sorbonne Université, where he works with Carola Doerr on the DynaBBO project, focused on dynamic algorithm selection. He is part of the core development team of IOHprofiler, with a focus on the IOHanalyzer. His research interests include benchmarking of optimization heuristics, dynamic algorithm selection and configuration, as well as hyperparameter optimization.Pascal Kerschke
TU Dresden, Germany | webpage
Pascal Kerschke is professor of Big Data Analytics in Transportation at TU Dresden, Germany. His research interests cover various topics in the context of benchmarking, data science, machine learning, and optimization - including automated algorithm selection, Exploratory Landscape Analysis, as well as continuous single- and multi-objective optimization. Moreover, he is the main developer of flacco, co-authored further R-packages such as smoof and moPLOT, co-organized numerous tutorials and workshops in the context of Exploratory Landscape Analysis and/or benchmarking, and is an active member of the Benchmarking Network and the COSEAL group.