1st International Workshop on Recruiting Participants for Empirical Software Engineering (RoPES’22)

  • Co-located with ICSE 2022
  • Virtual/In Person
  • May/June X, 2022

Theme & Goals

Software studies, and the field of software engineering research, benefit from the use of high numbers of quality participants. However, participant recruitment is challenging. Recruiting people from open source projects often involves ethical hurdles like privacy protection 1, or participant fatigue from being over-researched. Using students can be helpful, but also suffers from generalizability to the wider software developer population 2. On top of this, challenges due to pandemic-related health measures have made in-person studies even more difficult to conduct. Finally, the maturity of our community in understanding sampling issues 3, and the associated power calculations, effect size measurements etc. derived from samples can always be improved 4.

There are also new, potentially useful tools for crowd-sourcing research, such as Mechanical Turk, or Prolific, but these tools are typically not used very often, and when they are, require careful study organization to prevent bias and flawed sampling 5. Other communities, such as HCI and psychology, do have extensive knowledge of how to use these tools, but may not have similar expertise requirements of their subjects.

Topics of Interest

ROPES 2021 seeks contributions addressing, but not limited to, the following topics related to recruitment of participants in empirical software engineering:

  • strategies for new platforms like Prolific, such as payment and filtering;
  • students vs practitioners;
  • determining adequate sample sizes;
  • ethical considerations of participant involvement (such as GDPR, use of emails);
  • equity, diversity and inclusion considerations, such as oversampling women;
  • incentive structures;
  • building reputation with a community (so they don’t reject study invites);
  • challenges with joint studies with industry and academia;
  • self-selection bias (the same people always respond);
  • privacy and IRB restrictions;
  • overfishing problem (returning to the same well);
  • personal contacts effective but limited;
  • study fadeout effect and how to get people to complete the study;
  • use of attention questions;
  • challenges related to recruiting the ``right” participants - right number, right background, right demographics, etc.
  • budgeting for recruitment costs.

Keynotes

TBD.

Workshop Program

Times will be Eastern Standard Time (EST, UTC-5)

Time Title Who
9:00 AM Welcome RoPES organizers
9:15 AM Keynote TBD
10:00 AM Break  
10:15 AM Session 1: Finding Participants  
11:30 AM Discussion session  
12:00PM Lunch  
1:30PM Session 2: Keeping Participants  
3:00 PM Break  
3:15 PM Session 3: Paper Brainstorming Organizers
05:45 End of RoPES 2022  

Submission Guidelines

Workshop papers must follow the ICSE 2021 Format and Submission Guideline, but will use a single blind submission process. All submitted papers will be reviewed on the basis of technical quality, relevance, significance, and clarity by the program committee. All workshop papers should be submitted electronically in PDF format through the EasyChair workshop website. Accepted papers will become part of the workshop proceedings.

Important Dates

Event Deadline
PAPER SUBMISSION January 14, 2022
MAIN TRACK AUTHOR NOTIFICATION February 18, 2022
CAMERA-READY March 18, 2022
DATE OF WORKSHOP TBD

Organizing Committee

Name Affil Twitter
Neil Ernst U. Victoria @neilernst
Jeffrey C. Carver U. Alabama @JeffCarver32
Carianne Pretorius TU Eindhoven @cari_pretorius
Preetha Chatterjee Drexel U. @PreethaChatterj
Alexander Serebrenik TU Eindhoven @aserebrenik
Bonita Sharif Nebraska @shbonita
Matthew Smith Bonn @m42smith

Program Committee

The OC + the following generous individuals below:

First name Last name Organization Social Media
Alex Bezzubov JetBrains  
Christian Bird Microsoft  
Kelly Blincoe The University of Auckland  
Fabiano Dalpiaz Utrecht University  
Anastasia Danilova University of Bonn  
Felipe Ebert Eindhoven University of Technology  
Felipe Fronchetti Virginia Commonwealth University  
Davide Fucci HITeC, University of Hamburg  
Fabian Gilson University of Canterbury  
Valentina Lenarduzzi LUT University  
Daniel Mendez Blekinge Institute of Technology, Sweden, and fortiss, Germany  
Kevin Moran College of William & Mary  
Nicole Novielli Dipartimento di Informatica, University of Bari  
Richard Paige McMaster University  
Paul Ralph Dalhousie University  
Martin Robillard McGill University  
Daniel Russo Department of Computer Science, Aalborg University  
Igor Scaliante Wiese Federal University of Technology – Paraná - UTFPR  
Janet Siegmund Chemnitz University of Technology  
Melina Vidoni Australian National University, CECS School of Computing  
Andy Zaidman Delft University of Technology  

Call for Papers

People are invited to submit short, 2 page papers that encompass the following categories and pertain to the workshop themes:

  • short position papers describing issues related to participant recruitment;
  • summaries of previously published work that involved significant participant recruitment challenges, i.e. an experience report;
  • extended abstracts of ongoing work or challenges.

In all cases, papers should be no more than 2 pages. We will handle submissions via Google Form and ask that authors self-host submissions. Accepted papers will then be stored on the workshop home page. All papers will be reviewed by the PC for suitability to the topics, but we hope to accept all on-topic submissions.

The workshop outcome will be an extended workshop summary for a venue such as Software Engineering Notes or IEEE Software.

Format

Please follow the ICSE formatting guidelines but anonymization is not required.

References

  1. Nicolas E. Gold and Jens Krinke. 2020. Ethical Mining. InProceedings of the 17th International Conference on Mining Software Repositories. ACM. https://doi.org/10.1145/3379597.33874623 

  2. Robert Feldt, Thomas Zimmermann, Gunnar R. Bergersen, Davide Falessi, AndreasJedlitschka, Natalia Juristo, Jürgen Münch, Markku Oivo, Per Runeson, MartinShepperd, Dag I. K. Sjøberg, and Burak Turhan. 2018. Four commentaries on theuse of students and professionals in empirical software engineering experiments.Empirical Software Engineering23, 6 (Nov. 2018), 3801–3820. https://doi.org/10.1007/s10664-018-9655-0 

  3. Sebastian Baltes and Paul Ralph. 2020. Sampling in Software Engineering Research: A Critical Review and Guidelines. CoRRabs/2002.07764 (2020). https://arxiv.org/abs/2002.07764 

  4. Francisco Gomes de Oliveira Neto, Richard Torkar, Robert Feldt, Lucas Gren,Carlo A. Furia, and Ziwei Huang. 2019. Evolution of statistical analysis in empiricalsoftware engineering research: Current state and steps forward. Journal of Systemsand Software 156 (Oct. 2019), 246–267. https://doi.org/10.1016/j.jss.2019.07.002 

  5. Anastasia Danilova, Alena Naiakshina, Stefan Horstmann, and Matthew Smith. 2021. Do you Really Code? Designing and Evaluating Screening Questions forOnline Surveys with Programmers. In2021 IEEE/ACM 43rd International Conferenceon Software Engineering (ICSE). IEEE. https://doi.org/10.1109/icse43902.2021.0005