We had great discussion during the second event of the workshop series about reproducibility and open science collocated with the AGILE conference. Based on experience of the first event “Reproducible Geosciences Discussion Forum” at AGILE 2017 held in Wageningen, The Netherlands, we run the “Reproducible Research Publications workshop” at the past AGILE 2018 conference, in Lund (Sweden).

Daniel Nüst, from Münster university, wrote a piece summarizing the workshop results. I would like to reproduce here an extract below that highlights the interesting discussion we, organizers and participants, had during the last part of the workshop. The sentences in bold give an idea of the type of required actions that we must take in preparation for the third workshop in the series at AGILE 2019 in Limassol, Cyprus.

 The discussion showed that the detailed manual reproduction is an important learning experience when starting with reproducible research. Understanding the challenges and pitfalls a reader might have is an important prerequisite and motivation for changing one’s own habits. But it also became clear that a room full of people with different operating systems, knowledge, and software versions will always find a problem unknowable to the author. There are limits to the level of detail a scholarly article can provide regarding technical instructions, so not “everyone” may be able to reproduce, but somebody must be able to – at least the author and even after a few years. […]. It was also stated that the objectives of a reproduction (reproduce a chart, apply method to different data, conduct a review) differ and thus might require different reproduction workflows. The different workflows could cater for anonymity (blind peer review) or formal limitations (e.g. of a submission system). One participants made the interesting suggestion of documenting the workflow visually, in a figure or diagram, to help the reader. The existing practice of double-blind reviews was also seen as posing yet another challenge towards suitable documentation and easily reproducible workflows.

All of these numerous challenges and limitations lead the discussers towards the need for more education (e.g. courses or updated syllabi) and materials (e.g. domain-specific guidelines) for authors, and also a broader discussion on the topic of reproducible research.

Stay tuned for new activities at the Reproducible Research @ AGILE website, http://o2r.info/reproducible-agile/. If you’re interested in learning more about reproducible research and/or attending the next workshop, let us know what you would like to learn or discuss (see contact emails at the previous web site).