1st International Workshop on Recruiting Participants for Empirical Software Engineering (RoPES’22)
- Co-located with ICSE 2022
- Virtual/In Person
- May/June X, 2022
Theme & Goals
Software studies, and the field of software engineering research, benefit from the use of high numbers of quality participants. However, participant recruitment is challenging. Recruiting people from open source projects often involves ethical hurdles like privacy protection 1, or participant fatigue from being over-researched. Using students can be helpful, but also suffers from generalizability to the wider software developer population 2. On top of this, challenges due to pandemic-related health measures have made in-person studies even more difficult to conduct. Finally, the maturity of our community in understanding sampling issues 3, and the associated power calculations, effect size measurements etc. derived from samples can always be improved 4.
There are also new, potentially useful tools for crowd-sourcing research, such as Mechanical Turk, or Prolific, but these tools are typically not used very often, and when they are, require careful study organization to prevent bias and flawed sampling 5. Other communities, such as HCI and psychology, do have extensive knowledge of how to use these tools, but may not have similar expertise requirements of their subjects.
Topics of Interest
ROPES 2021 seeks contributions addressing, but not limited to, the following topics related to recruitment of participants in empirical software engineering:
- strategies for new platforms like Prolific, such as payment and filtering;
- students vs practitioners;
- determining adequate sample sizes;
- ethical considerations of participant involvement (such as GDPR, use of emails);
- equity, diversity and inclusion considerations, such as oversampling women;
- incentive structures;
- building reputation with a community (so they don’t reject study invites);
- challenges with joint studies with industry and academia;
- self-selection bias (the same people always respond);
- privacy and IRB restrictions;
- overfishing problem (returning to the same well);
- personal contacts effective but limited;
- study fadeout effect and how to get people to complete the study;
- use of attention questions;
- challenges related to recruiting the ``right” participants - right number, right background, right demographics, etc.
- budgeting for recruitment costs.
Times will be Eastern Standard Time (EST, UTC-5)
|9:00 AM||Welcome||RoPES organizers|
|10:15 AM||Session 1: Finding Participants|
|11:30 AM||Discussion session|
|1:30PM||Session 2: Keeping Participants|
|3:15 PM||Session 3: Paper Brainstorming||Organizers|
|05:45||End of RoPES 2022|
Workshop papers must follow the ICSE 2021 Format and Submission Guideline, but will use a single blind submission process. All submitted papers will be reviewed on the basis of technical quality, relevance, significance, and clarity by the program committee. All workshop papers should be submitted electronically in PDF format through the EasyChair workshop website. Accepted papers will become part of the workshop proceedings.
|PAPER SUBMISSION||January 14, 2022|
|MAIN TRACK AUTHOR NOTIFICATION||February 18, 2022|
|CAMERA-READY||March 18, 2022|
|DATE OF WORKSHOP||TBD|
|Neil Ernst||U. Victoria||@neilernst|
|Jeffrey C. Carver||U. Alabama||@JeffCarver32|
|Carianne Pretorius||TU Eindhoven||@cari_pretorius|
|Preetha Chatterjee||Drexel U.||@PreethaChatterj|
|Alexander Serebrenik||TU Eindhoven||@aserebrenik|
The OC + the following generous individuals below:
|First name||Last name||Organization||Social Media|
|Kelly||Blincoe||The University of Auckland|
|Anastasia||Danilova||University of Bonn|
|Felipe||Ebert||Eindhoven University of Technology|
|Felipe||Fronchetti||Virginia Commonwealth University|
|Davide||Fucci||HITeC, University of Hamburg|
|Fabian||Gilson||University of Canterbury|
|Daniel||Mendez||Blekinge Institute of Technology, Sweden, and fortiss, Germany|
|Kevin||Moran||College of William & Mary|
|Nicole||Novielli||Dipartimento di Informatica, University of Bari|
|Daniel||Russo||Department of Computer Science, Aalborg University|
|Igor||Scaliante Wiese||Federal University of Technology – Paraná - UTFPR|
|Janet||Siegmund||Chemnitz University of Technology|
|Melina||Vidoni||Australian National University, CECS School of Computing|
|Andy||Zaidman||Delft University of Technology|
Call for Papers
People are invited to submit short, 2 page papers that encompass the following categories and pertain to the workshop themes:
- short position papers describing issues related to participant recruitment;
- summaries of previously published work that involved significant participant recruitment challenges, i.e. an experience report;
- extended abstracts of ongoing work or challenges.
In all cases, papers should be no more than 2 pages. We will handle submissions via Google Form and ask that authors self-host submissions. Accepted papers will then be stored on the workshop home page. All papers will be reviewed by the PC for suitability to the topics, but we hope to accept all on-topic submissions.
The workshop outcome will be an extended workshop summary for a venue such as Software Engineering Notes or IEEE Software.
Please follow the ICSE formatting guidelines but anonymization is not required.
Nicolas E. Gold and Jens Krinke. 2020. Ethical Mining. InProceedings of the 17th International Conference on Mining Software Repositories. ACM. https://doi.org/10.1145/3379597.33874623 ↩
Robert Feldt, Thomas Zimmermann, Gunnar R. Bergersen, Davide Falessi, AndreasJedlitschka, Natalia Juristo, Jürgen Münch, Markku Oivo, Per Runeson, MartinShepperd, Dag I. K. Sjøberg, and Burak Turhan. 2018. Four commentaries on theuse of students and professionals in empirical software engineering experiments.Empirical Software Engineering23, 6 (Nov. 2018), 3801–3820. https://doi.org/10.1007/s10664-018-9655-0 ↩
Sebastian Baltes and Paul Ralph. 2020. Sampling in Software Engineering Research: A Critical Review and Guidelines. CoRRabs/2002.07764 (2020). https://arxiv.org/abs/2002.07764 ↩
Francisco Gomes de Oliveira Neto, Richard Torkar, Robert Feldt, Lucas Gren,Carlo A. Furia, and Ziwei Huang. 2019. Evolution of statistical analysis in empiricalsoftware engineering research: Current state and steps forward. Journal of Systemsand Software 156 (Oct. 2019), 246–267. https://doi.org/10.1016/j.jss.2019.07.002 ↩
Anastasia Danilova, Alena Naiakshina, Stefan Horstmann, and Matthew Smith. 2021. Do you Really Code? Designing and Evaluating Screening Questions forOnline Surveys with Programmers. In2021 IEEE/ACM 43rd International Conferenceon Software Engineering (ICSE). IEEE. https://doi.org/10.1109/icse43902.2021.0005 ↩