Among the top priorities the group singled out on the topic of preservation and reproducible research are responsibility, maintenance, and timeframe (see notes). The summary below captures the main themes of the conversation, highlighting key questions.

Reproducibility presents new challenges to preservation

• Does preservation of computationally reproducible research require a paradigm shift in the field of digital preservation?

Digital preservation is a “series of managed activities, policies, strategies and actions to ensure the accurate rendering of digital content for as long as necessary, regardless of the challenges of media failure and technological change.” A fundamental target of digital preservation is the digital object, and a primary concern is bit-level preservation, “literally preserving the bits forming a digital object.” The idea is to make sure the integrity of the bits is intact over time.

Where computational reproducibility is concerned, as pointed out by participants, the target for preservation is better thought of as a “performance.” In the context of reproducibility, we are interested in preserving the execution of a computational process, often as it relates to specific input and output data. This also requires the preservation of the various components that enable the performance or the process, including the data, the software, the state of the computer, and so on.

Importantly, the nature and goals of different types of “performances” – executions of a computational process – vary potentially affecting what needs to be preserved and how. For example, reproducing computation that performs data analysis for the purpose of verifying a specific scientific claim is different from computation that performs modeling and simulation, with implications for what needs to be preserved. Guidelines about whether it’s enough to preserve the documentation or metadata reproduction no longer works, and what documentation we need to preserve different types of computation, would be helpful. Context is important and guidelines and standards can help steer toward preservation actions that are appropriate for what it is that the computation aims to reproduce.

Participants expressed support for approaches that are aimed at reducing dependencies as a way to facilitate both preservation and reuse. Efforts specifically designed to support the preservation of computational research mentioned at the meeting include emulation (e.g., EaaSI provides shared infrastructure for long term access to emulated hardware, maintaining the contextual space of the software and systems required to reproduce computation) and packaging (e.g., RO-Crate is a lightweight approach to packaging research data with their metadata).

• Are there sufficient opportunities for cross-fertilization between digital preservation experts and those implementing reproducible research policies, workflows, and tools?

It seems likely that a combination of these various approaches and a commitment by various stakeholders will be needed. This is a recurring theme in digital preservation writ large, as noted by the Digital Preservation Coalition, “digital preservation cannot be perceived as solely a concern for archives, libraries, museums and other memory institutions: it is a challenge for all who have an interest in creating, using, acquiring and making accessible, digital materials.” That said, it is often unclear who is responsible for preservation of reproducible research (this issue has come up in prior community conversations, e.g., on publishing, solutions).

• Who should be responsible for preserving reproducible research?

Currently, preservation is usually attended to at the end of the research cycle, but there are relevant touch points along the entire process. While repositories “do” preservation, it is better when all the stakeholders work toward this goal. Preservation is better supported when research is done with an eye toward reproducibility from the beginning, when proper data management is performed throughout, and when curators are involved before the research is shared, among other things.

• How do we develop and support a skilled and professional workforce of archivists and preservation experts to work alongside researchers?

Importantly, individuals assigned to preservation tasks must have the skills to perform those tasks (see previous conversation about training) and supporting resources and infrastructure (i.e., guidelines, standards, policies, established practices, tools).

Gaps in tools and infrastructure

With respect to tools, some may facilitate reproducibly but not lend themselves easily to preservation. For example, container technology is a popular solution for reproducibility but introduces dependencies which can be a challenge for preservation. At this time there is no standardized approach to documenting essential information for preservation such as the operating system and runtime required, to run the container and how to connect to a data source that may be archived elsewhere.  

Participants also expressed reservations about the development and use of commercial tools that enable and facilitate reproducibility. Commercial tools require licenses and are often “black boxes,” presenting challenges to long-term preservation and reuse. One idea might be to encourage commercial toolmakers to incorporate preservation-supporting functionality into their solutions. The recent GitHub integration with Zenodo can be thought of as a positive example of this. An example from the open source community is the EaaSI project’s concerted effort to contribute detailed documentation to Wikidata to ensure its value for the long term. As noted in previous conversations, a clear message on the community’s position about open vs. commercial reproducibility tools – and specifically as it pertains to preservation – can inspire the creation of guidelines for researchers, publishers, and repositories on selecting tools that lend themselves to preservation.

National infrastructure projects for data management and preservation offer benefits such as resources and funding. This kind of investment in repositories is encouraging, but more emphasis is needed on preserving a diversity of materials such as software and old compiled operating system applications (e.g., Software Heritage and EaaSI).

Gaps in policy and culture

Participants pointed to gaps in policy and culture when it comes to preservation and reproducible research. These gaps need to be addressed if the community is interested in ensuring that responsibility for preservation is optimally distributed.

• Do we need preservation policies that are specific to reproducibility, or do general policies (i.e. on other research objects) suffice?

In terms of policies governing preservation and reproducible research, there’s a need to involve experts from various disciplines from the beginning (researchers, technology developers, publishers, etc.). There is currently a gap is communicating preservation requirements to tool developers and publishers and funders who write policies. There are also inconsistent practices on the part of researchers, and varying levels of awareness and expertise of the issues across disciplines. Computationally heavy disciplines naturally understand reproducibility and are able to communicate their needs to those developing systems, tools, services. A community-wide effort to clarify requirements and establish standards will support more comprehensive and responsive preservation efforts.

The following topics could be clarified:

The difference between preservation of data and preservation of materials for reproducibility. Standards would support scaling efforts which are necessary in order to move beyond bit-level preservation. Data-specific guidelines, e.g., legal or ethical implications to consider when preserving human-related data used in reproducible research, continue to apply and should be reinforced.  

Preservation of the linkage between the artifacts. What are the pros and cons of different approaches when it comes to preservation? e.g. renku tool, research compendia, Docker, ReproZip. Making all research outputs FAIR can help in this regard.

Preservation time horizon, or the duration that the research is reproducible. Some funders have requirements around the timeframes for the preservation of research data. If these were to be extended to “reproducible research outputs,” it raises a question around what is feasible in practice given current technological means and the constant evolution of computing. Most of the required timeframes for data preservation seem relatively short – could they become “indefinite”? Should they? Educating stakeholders about the limits of digital preservation is important.

• What should be the role of the ACM in advancing preservation and reproducible research?

ACM general guidelines on workflows that facilitate preservation of reproducible research would lead to consistency across authors and conferences. Already practices in Computer Science, such as working reproducibly and collaboratively and incorporating code review, are common. Each SIG can then adopt workflows that suit its needs while adhering to the general standards. Speaking in a unified voice, the organization is positioned to be a leader in this area.

A note: Thank you to everyone who has participated in this virtual conversation series! Please take a moment to fill out the feedback survey.

By Limor Peer and Vicky Rampin

August 6, 2021

The post Reproducibility PRESERVATION: Taking the Pulse appeared first on EIG Repro.

We are hoping in particular to discuss the following topics:

• What are some of the minimum requirements for long-term reproducibility? Is there a shared understanding in the academy of the requirements for reproducibility? 
  • Including, what metadata do we want to preserve and make FAIR?
• How is preserving data, software, and workflows used dynamically in computation different from preserving static digital objects?
• What are the existing initiatives out there around curating materials that are meant to be rerun and reproduced indefinitely?
• What resources are needed to preserve a complete and reproducible scholarly record?
• What recent developments in the digital preservation community are particularly relevant to research reproducibility? (e.g., Emulation as a Service Infrastructure, NDSA Levels of Preservation, the Software Metadata Recommended Format Guide)
• How can researchers and digital preservation practitioners collaborate in service of long-term reproducibility?
• How are publishers supporting basic infrastructure or developing policies for the preservation of reproducible research?

We will convene on July 29, 2021 from 12 – 1 PM EDT. RSVP here for a personalized Zoom link.

What to expect: This is an interactive session. We are interested in “taking the pulse” and hearing your views about these issues. We won’t have presentations or panels. We’ll have prompts with questions and rely on participants to keep the conversation going. We’ll have a collaborative notes document for participants to drop in links to resources as well as take notes about our discussion. Afterwards, we’ll publish a blog post summarizing what we discussed on this blog. The conversation will be moderated by the P-WG leaders: Limor Peer and Vicky Rampin.

Important information:

• Conversation: July 29, 2021, 12 -1 PM EDT (16:00 UTC) [registration link]
• Collaborative notes: (google doc)

About this blog: P-WG is extending an invitation to the community to engage in conversation about topics related to the reproducibility of computational research via the blog, “Taking the Pulse.” In a series of open community meetings, the P-WG will explore issues related to reproducibility and document the views of different communities, culminating in recommendations informed by current practices. The P-WG held a forum for conversation about these topics over the course of several months:

The post Reproducibility PRESERVATION: Join the discussion! appeared first on EIG Repro.

These are the main themes and key questions that emerged from the conversation: goals and standards, reforms and initiatives, peer review, and support for conducting reproducible research in the first place.

Goals and standards of publishing reproducible research

Doing reproducible research can be difficult in and of itself and publishing reproducible research has its own set of challenges. To overcome some of the socio-technical challenges to publishing reproducible research, the community would do well to be very clear on its meaning and its value.

·        Why publish reproducible research?

·        Who is responsible for ensuring that published materials are reproducible?

There is broad recognition of the rationale for computational reproducibility as a mechanism for achieving the goals of transparency and verifiability. The 2019 report, Reproducibility and Replicability in Science, by the National Academies of Science, Engineering, and Medicine (NASEM) recommends that journals, “consider ways to ensure computational reproducibility for publications that make claims based on computations, to the extent ethically and legally possible,” recognizing that this poses technological and practical challenges for journals (as well as for researchers). Publishers and scholarly societies are beginning to write policies that reflect their commitment to reproducibility.

For example, the American Economic Association articulated its goals when it revised its data and code policy in 2019: “(1) to encourage and reward incorporating basic principles of replicability into researchers’ workflow; (2) to prioritize linking to existing data and code repositories, as the primary mechanism of providing source materials, with a journal-sanctioned repository as a fallback archive; (3) to require and facilitate proper documentation of restricted-access data; (4) to enforce a limited measure of verification; and (5) to balance the previous goals with the need to reduce the burden on authors, not increase it.”

·        What is it that we want to reproduce? What does it mean to publish reproducible research?

Participants indicated that there is no consensus about what should be published in support of reproducibility. If there is agreement that the basic elements of computational research include code or software and data (and associated documentation and metadata), we recognize that each is subject to broad variation in usage and interpretation among disciplines and research traditions. This is closely tied to another fundamental question:

·        If computational reproducibility is not a one-size-fit-all, can we expect (or should we work toward) universal standards for publishing reproducible research? Or is the system we envision a federated one?   

In some scenarios, the goal of computational reproducibility is to test that software is executable. In others, the goal of computational reproducibility is to verify that a particular finding is correct. Accordingly, the source materials that need to be published in order to satisfy reproducibility in each scenario vary greatly. For example, what does it mean to publish executable code? In some disciplines the code requires resource-heavy builds, in others it may be a few lines of script using open-source software. Similarly, requirements and conventions around publishing data may vary depending on the type and source of the data. In some disciplines, it might be completely unobjectionable to publish a simulated dataset while in others access to original data is critical.

Given domain-specific checklists, tools, reproduction infrastructure, and guidelines intended to help researchers work reproducibly, is it realistic to expect we can achieve a universal standard for metadata and semantics for publishing this work?

·        How should journals approach threats to computational reproducibility over time?

Another question is whether publication is “one and done” or needs to be ongoing. Is it sufficient (i.e., is it in the interest of scientific progress) that journals publish reproducible research on the day of publication? (Note: we’ll explore questions of preservation next month.)

Several participants expressed the opinion that, while active maintenance is an ideal state, transparency is perhaps the most likely or even reasonable outcome from reproducibility efforts. Keeping track of the data, software (exact versions), compute logs, etc., enable users to go back later and determine how results were produced and, more importantly, why results might have changed.   

Scholarly communication reforms in support of reproducibility

Several initiatives in scholarly communication in support of publishing reproducible research were mentioned in the meeting. Many are experimental, having been tested and implemented in a particular domain or journal. These include,

New roles on journal editorial boards and publishers’ staff specializing in reproducible and open science. For example, an Innovation Officer at eLife who is on the Executive staff, overseeing the eLife Innovation Initiative. Another example is a Data Editor at the American Economic Association).

New formats for published research articles (i.e., beyond PDF), such as executable research articles (see for example, Stencila).

New policies. Journals are increasingly requiring not only that data and code are made available (there is great variation in what that actually means) but mandatory full reproductions.

Reproducibility checking. Some journals are investing in efforts to verify that materials they publish are indeed reproducible. This process may be manual: Some journals are contracting with third parties to perform human-based reproducibility checking, such as CASCAD and the Odum Institute. In other cases, it might be automated. The software community uses reproducible builds for building software and addresses a similar problem, and can be a promising source of information, toolchains, and functionality. Examples include a “software assemblage” for each paper that is continually re-executed using a continuous integration system, such as Travis (more on this from James Howison), or software development practices that create an independently-verifiable path from source to binary code, such as reproducible builds.

·        How do journal policies on reproducibility manifest in our communities (e.g., TOP Guidelines)? How well are they followed and enforced?

Participants expressed the sense that this is an evolving space, with a fair degree of experimentation. More is needed, especially in the realm of automation. Automation can be useful as a means for reporting back to authors any problems with reproducibility at the time of submission can encourage authors to deliver higher quality source materials. Additionally, automatic (and standard-based) packaging and publication of a certifiably-reproducible research would cut down on publisher cost.

The question of peer review

There is a fair amount of labor involved in verifying reproducibility during the journal publication process and this topic garnered quite a bit of interest from this group. In particular, labor provided by peers (as opposed to specialized third parties) to verify reproducibility was a topic of conversation. The group identified several problematic aspects of peer review in this context.

·        Who reviews reproducibility materials before publication? How is that process integrated with the general publication process? How can or should peer review be updated as a process to align with the goals of promoting reproducible research?

Cultural issues relating to a legacy of free labor by peer (manuscript) reviewers are problematic. Computational reproducibility is more complex, requiring more time investment and access to resources. Moreover, reproducibility review by humans is time consuming and researchers are currently not adequately incentivized to perform it as part of standard peer review. Participants expressed that this work should be compensated. One suggestion to help encourage a cultural shift is for funders to allow this as a budget item. As reproduction infrastructure matures, human labor may decrease.

(Note: a recent article by Willis and Stodden explores how “expanding the peer review process can be used to improve or even ensure the reproducibility of computational research at the time of publication” and is relevant to this conversation.)

Finally, this topic relates to the broader issue of the labor of reproducible research (this topic was addressed also at our previous conversation about reproducibility principles). While the actors may be different depending on the stage in the research lifecycle – graduate students during active research vs. peers during journal review, for example – unresolved problems including proper credit, training, capacity, and resources are evident.  

To publish reproducible research, we need to practice reproducible research

Publishing reproducible research is much more difficult when the research was not conducted in a reproducible way from its inception. To stay on top of fast-evolving policies, technologies, and norms in this space, researchers need support. Importantly, their work needs to be recognized and rewarded.

·        How can we align our assessment and metrics to foster an environment that promotes reproducible research?

In many cases, more training is the answer. Some researchers want to do open and reproducible research, but don’t know how to do it and more training and education is required (see our previous blog post on reproducibility training and education). In other cases, especially in certain fields, to work reproducibly researchers must rely on professionals who support their data and computation needs. Funding and developing such a workforce is a growing concern (an Interest Group at the Research Data Alliance is working on this topic).   

Finally, two things. We note an issue we raised in our provocation post that we did not get to at the meeting: open scholarship, FAIR, and CARE and how they relate to publishing reproducible research. Please let us know if this is of interest for further discussion. We also note that several of the themes in this meeting echo previous conversations held in this forum.

We’ll be holding one more topical meeting, on reproducibility and preservation, on July 29, 2021 (join us! RSVP here). Please stay tuned for future blog posts to continue the conversation!

By Limor Peer and Vicky Rampin

The post Reproducibility PUBLISHING: Taking the Pulse appeared first on EIG Repro.

We are hoping in particular to discuss the following topics:

• How can and should we update scholarly publishing to reflect the richness of methods and underlying materials to support claims?
• Is there a consensus about what should be published in support of reproducibility?
• How do Open Scholarship principles interact with reproducibility publishing? 
  • What about FAIR and CARE Principles? 
• Who is responsible for ensuring that published materials are reproducible?
• How do journal policies on reproducibility manifest in our communities (e.g.TOP Guidelines)? How well are they followed and enforced?
• Who reviews reproducibility materials before publication? How is that process integrated with the general publication process? How can or should peer review be updated as a process to align with the goals of promoting reproducible research?
• How can we align our assessment and metrics to foster an environment that promotes reproducible research?

We will convene on June 24, 2021 from 12 – 1 PM EDT. RSVP here for a personalized Zoom link.

What to expect: This is an interactive session. We are interested in “taking the pulse” and hearing your views about these issues. We won’t have presentations or panels. We’ll have prompts with questions and rely on participants to keep the conversation going. We’ll have collaborative notes document for participants to drop in links to resources as well as take notes about our discussion. Afterwards, we’ll publish a blog post summarizing what we discussed on this blog. The conversation will be moderated by the P-WG leaders: Limor Peer and Vicky Rampin.

Important information:

• Conversation: June 24, 2021, 12 -1 PM EDT (16:00 UTC) [registration link]
• Collaborative notes: (google doc)

About this blog: P-WG is extending an invitation to the community to engage in conversation about topics related to the reproducibility of computational research via the blog, “Taking the Pulse.” In a series of open community meetings, the P-WG will explore issues related to reproducibility and document the views of different communities, culminating in recommendations informed by current practices. The P-WG will hold a forum for conversation about these topics over the course of several months:

The post Reproducibility PUBLISHING: Join the discussion appeared first on EIG Repro.

In this post, we will summarize the main themes that emerged from the conversation and highlight key remaining questions. 

The goal of reproducibility training and education

An important distinction needs to be recognized between the goals of instilling basic values and ensuring good (or best) practices on an ongoing basis. Both goals are essential and each has its own set of considerations. To instill basic values of responsible conduct, scientific rigor, and research integrity, the academy requires that researchers receive training (e.g., Responsible Conduct of Research at the NIH). In that context, reproducibility is often taught as a central concept and a guiding principle.

When it comes to fixing good habits and ensuring that reproducible practices are widely implemented (and improved), training is often taught haphazardly and “on the job.” There are weak incentives for researchers to refresh their practical reproducibility skills and knowledge and generally weak enforcement mechanisms.

·        What are the fundamentals of reproducibility training?

·        What is the right sequence for teaching reproducibility?

Research integrity and transparency are key concepts, as are data management and organization.

In the United States the NIH curriculum on Responsible Conduct of Research (RCR) covers topics related to scientific rigor and integrity with pre-clinical research. These trainings are critical but tend to be taught once, as a requirement for new faculty or post-doc, for example, but are not broadly embedded or enforced in the practice of research. The signal new researchers may be receiving is that fundamentals such as integrity, transparency, and ethics as they relate to reproducibility are somehow “extra” on top of what they actually need to do to get a paper published in a high-impact journal.

The observation that the daily grind of research practice, in some corners of the academy, sometimes works against some ethical norms (e.g., who gets authorship) in the race to publish is not new. In the community meeting, there was strong consensus that for ethical concepts to take hold they ought to be applied to the daily practice of research on an ongoing basis. This includes training in data management and basic organization skills, including directory structure. For example, educating researchers to use a common directory structure for research projects to follow FAIR principles or the TIER Protocol. Training (and tool development) in automated and standardized data collection is a worthy investment and will facilitate reproducibility. The RCR (and similar) training is often a successful outreach point for librarians teaching data management. Among the participants in the community meeting who are teaching reproducibility, the sentiment was that transparency is an underlying focus in all their training, but that “90% of reproducibility is research data management.”

This may be thought of as an issue of progression, sequence, timing, and frequency. Currently there is not enough thought put into how all the pieces fit together.

Approaches to teaching reproducibility

·        What is the most effective format for teaching reproducibility?

The short answer is that it’s difficult to say. There are various formats in use, ranging from a single “one and done” lesson, to more in-depth workshops, to embedding reproducibility in formal academic training, to online self-taught modules and MOOCS, lectures, and videos. The NSF Advisory Committee for Cyberinfrastructure (ACCI) is looking into the creation of educational modules about reproducibility that can be added into academic curricula. In addition, efforts to cultivate “communities of practice” such as ReproducibiliTea, which is run by graduate students, can be an effective mechanism to introduce or supplement reproducibility training on campus. The longer answer is that all formats are useful and have their place. 

There is not much by way of assessing reproducibility training. Some efforts were reported by librarians working on reproducibility in the conference, Librarians Building Momentum for Reproducibility, including an ethnographic study of a hands-on workshop on reproducibility. It was reported in the meeting that UC Berkeley does informal assessment and case studies at the time of RCR training looking at re-invitations and tracking consultations requests. This may point to the difficulty with assessing the success of practical training. For some high level perspective, see this issue of the Harvard Data Science Review.

·        Who are the main recipients of this training?

The observation was made that reproducibility training can benefit a large and broad constituency. From PIs who are increasingly asked to comply with funder, journal, and institutional requirements, to junior faculty and graduate students, who can benefit from fixing ideas and habits at the start of their academic career, to research administrators and staff who have various touch points with the research process and therefore opportunities to support (and enforce) reproducibility.

The group seemed to agree that a lot of progress can be made, in particular, by creating pathways for graduate students, both because they are often intrinsically motivated and because there are opportunities to educate them about reproducibility in the curriculum.

·        What incentives are in place for reproducibility training?

Required reproducibility training in universities typically focuses on RCR, as mentioned above. There are fewer formalized incentives for researchers to seek additional, more practical training. Viewing replications (successful and unsuccessful) as part of tenure and promotion may be a strong incentive for researchers to seek more training. For example, one university changed its promotion structures to emphasize and encourage collegiality. “Top down” approaches that emphasize new norms for DEI (diversity, equity and inclusion) and recruitment can raise the profile of reproducibility and transparency practices and encourage people to seek more training. Additionally, universities and funders can do more to promote work that supports reproducibility infrastructure, for example, taking code from a prominent publication and turning it into reproducible software.

The desire to publish in top journals may incentivize researchers to seek training if their target journal requires reproducibility. The American Economic Association, for example, requires reproducibility verification in its journals. ReScience is a journal dedicated to publishing replications. Obviously, some researchers are self-motivated and will seek out training opportunities (and badges may provide a nudge for some).

Who trains?

In the context of individual universities, RCR training is often conducted by the office of research. When offered, data management is generally taught by librarians, and relevant tools and solutions offered on campus are typically taught by librarians and research support professionals. Discipline-specific reproducible research techniques and tools may be taught in methods classes and may be modeled at labs. A promising development at some universities is something akin to a reproducibility resource renter (e.g., University of Michigan Reproducibility Hub; see also TU Delft’ Research Data Management group and Digital Competence Center; TU Graz program on reproducibility plans).

Use of university-provided resources varies. One participant reported that at their university there is a designated Academic Lead for Research Improvement and Research Integrity who has reproducibility in their remit, and that actual support varies from department to department. Some universities mount an effort to train researchers to use reproducibility-aligned tools, for example, HPC and Carpentries in bioinformatics but “getting researchers trained in programming feels Sisyphean.” The general point is that reproducibility training at universities should be located close to research labs, rather than campus-wide. Ideally, one person per lab would be named the “reproducibility expert.”

We note that several organizations and groups offer reproducibility training (see list compiled by attendees in the collaborative notes).

Perceived gaps in reproducibility training

Not enough focus on practical skills: Creating sustainable habits is a challenge. Reproducibility is often taught in the context of training about responsible conduct of research, as a concept related to transparency and integrity, but institutions often do not require that it be taught as part of practical training (or enforce that it is practiced).

Too much focus on tools: Much training is focused on cool tools at the expense of the fundamentals such as data management. Demand for training on popular tools may affect the allocation of teaching resources away from more fundamental (but less trendy) topics. 

Teaching documentation needs to be a priority: Teaching best practices in documentation is a priority given its high value for reproducibility as compared with the dismal implementation. More work can be done to highlight the benefits of applying good data management practices from the beginning (for example, by pointing out to researchers and universities the sunk cost of irreproducible or unusable research).

Unsupported mandate: Journals that systematically check reproducibility do not train their reviewers. There is opportunity here for the ACM to play a role and recommend training for authors and reviewers.

Lack of enforcing norms: Reproducibility training will not take hold until responsible and ethical conduct of research is a social norm enforced by institutions, funders, and journals.

Lack of incentives: Incentives to produce and publish a lot are often perceived as at odds with the working reproducibly.

By Limor Peer and Vicky Rampin

June 2, 2021

The post Reproducibility TRAINING & EDUCATION: Taking the Pulse appeared first on EIG Repro.

We are hoping in particular to discuss these questions about training and education:

• How is reproducibility currently being taught, if it is?
• Where are there gaps in the current curriculum? What types of training are needed (e.g., metadata for common software; key skills)? Where is it done well?
• Who should be trained and are the training needs different? (e.g., faculty; grad students; archivists/curators; research scientists)
• When should people receive reproducibility training?
• Who should do the training? Should it be coordinated?
• How are training efforts assessed? How do we know they’re working?

We will convene on May 20, 2021 from 12 – 1 PM EDT. RSVP here for a personalized Zoom link.

What to expect: This is an interactive session. We are interested in “taking the pulse” and hearing your views about these issues. We won’t have presentations or panels. We’ll have prompts with questions and rely on participants to keep the conversation going. We’ll have collaborative notes document for participants to drop in links to resources as well as take notes about our discussion. Afterwards, we’ll publish a blog post summarizing what we discussed on this blog. The conversation will be moderated by the P-WG leaders: Limor Peer and Vicky Rampin.

Last month, we discussed some hands-on tools and solutions to support reproducible research, and in March we talked about reproducibility principles.

Important information:

• Conversation: May 20, 2021, 12 -1 PM EDT (16:00 UTC) [registration link]
• Collaborative notes: (google doc)

About this blog: P-WG is extending an invitation to the community to engage in conversation about topics related to the reproducibility of computational research via the blog, “Taking the Pulse.” In a series of open community meetings, the P-WG will explore issues related to reproducibility and document the views of different communities, culminating in recommendations informed by current practices. The P-WG will hold a forum for conversation about these topics over the course of several months:

The post Reproducibility TRAINING & EDUCATION: Join the discussion! appeared first on EIG Repro.

After brief introductions and background on the goals of the Emerging Interest Group (EIG) on Reproducibility and Replicability and the P-WG, the group expressed its priorities for the conversation. We bring you a summary of the conversation here. For each of the topics we discussed — finding the right tools, optimizing their use, and supporting their sustainability — we highlight the key questions.

Finding the right tools

Is there an inventory of reproducibility enabling or enhancing tools? What criteria do / can we use to assess tools and solutions? (e.g., wide-spread use, open source, interoperability, FAIR)

Resources for reproducibility tools were mentioned, including ReproMatch and Reproducibility Rubric.

It was noted that the P-RECS Workshop, which focuses heavily on practical, actionable aspects of reproducibility, is inviting submissions for P-RECS’21 and suggesting the following tools to automate experiments (not an exhaustive list): CK, CWL, Popper, ReproZip, Sciunit, Sumatra.

• What standards should reproducibility tools meet?

An important consideration is the standards the community would want tools to meet. In addition to the standards mentioned in our pre-meeting blog post (wide-spread use, open source, interoperability, FAIR), participants suggested two other standards: simplicity and veracity.

Simplicity captures the idea that use of the tool should be easy and easily integrated with current research practices so as to increase the likelihood that researchers will use it.

Veracity refers to the tool’s ability to execute computation as part of the full research cycle, including veracity of results. That is, not only the tool’s ability to generate a result, but to generate the same result as originally claimed. This ties in nicely with current conversations around reproducibility and quality assessment (QA) and quality control (QC), for example, ACM Policy on Artifact Review and Badging in computer science, Willis & Stodden, 2020 on journals addressing concerns about the quality and rigor of computational research.

This also raises a practical question of how to verify results at scale. A comment was made that the ability to determine comparability of results based on an automated comparison should be added to a “reproducibility wish list.”

• How can standards be applied at scale?
• Who should be entrusted to do a public evaluation of reproducibility tools?

Optimizing the use of reproducibility tools

How can we collaborate towards an interoperable ecosystem of tools to satisfy different reproducibility needs? Do we want to discourage one-off tools? Do we want to aim toward a unified system that can accommodate specialized tools?

With a proliferation of tools, it may be challenging for researchers to know what to use. Common tools and home grown solutions may meet immediate needs but fall short as far as other aspects of reproducibility (e.g., openness).

Participants expressed the opinion that no single tool provides a comprehensive solution for reproducibility due to the variety and complexity of issues and contexts, and that there is a need to create an ecosystem of reproducibility tools.

• How do we map the boundaries of an ecosystem of reproducibility tools?
• How do we build an ecosystem of reproducibility tools?

Whether a formal ecosystem of reproducibility tools emerges or a list of available tools is both a conceptual and practical question. It was suggested that a useful first step toward conceptualizing an ecosystem might be to categorize reproducibility tools, for example:

By the phase in the research cycle: “reproducibility in hindsight” vs “planning for reproducibility.”

By the main function: Web-based integrated development environment (IDE) (e.g. WholeTale), Web–based replay systems (e.g. Binder), packaging tools (e.g. ReproZip), containers (e.g. Docker, Singularity).

By level of specificity: tools that capture the computation steps vs the computation environment.

It was also suggested at the meeting that workflows need to be considered. Progress mentioned include work to identify the crucial research challenges related to workflows, (e.g., the Workflows Community Summit, Ferreira da Silva et al, 2021), work to define canonical components in the research lifecycle (Hardisty & Wittenburg, 2020), and efforts to capture information such as provenance and performance metrics about them (e.g., Pouchard et al, 2019). It is common in some disciplines to use workflow tools (e.g., biotech, machine learning, computing systems) and it was suggested that these communities can be brought together to collaborate around issues of reproducibility.

Supporting the sustainability of reproducibility tools

What are the lessons of recent efforts around research software sustainability? 

Efforts around research software sustainability are highly relevant to this community. The time horizon for functional software is typically shorter than it is for data, with major implications for computational reproducibility. A working group at the Research Data Alliance is a good source of information about these issues. 

• What assurance does the community want to have about the long-term usability of reproducibility tools? 
• Who holds the responsibility for sustaining research software that supports reproducibility?

In order to bring the use of reproducibility tools to the fore, and therefore help make them visible to funders and potentially increase the likelihood that they will be sustained, the use of a “reproducibility plan” was suggested. This idea, and the educational and training effort required to support it, will be discussed at a future “Taking the Pulse” conversation. 

This is the second in a series of open community meetings in which the P-WG explores issues related to reproducibility and document the views of different communities, culminating in recommendations informed by current practices. See our previous post on reproducibility principles. Each month we focus on a particular topic related to reproducibility: principles, solutions, training, publishing, and preservation.

By Limor Peer and Vicky Rampin

April 30, 2021

The post Reproducibility SOLUTIONS: Taking the Pulse appeared first on EIG Repro.

We are hoping in particular to discuss these facets of the current tools and approaches:

• Is there an inventory of reproducibility enabling or enhancing tools?
• What criteria do / can we use to assess tools and solutions? (e.g., wide-spread use, open source, interoperability, FAIR)
• Do we want to discourage one-off tools? Do we want to aim toward a unified system that can accommodate specialized tools?
• How can we collaborate towards an interoperable ecosystem of tools to satisfy different reproducibility needs?
• What are the lessons of recent efforts around research software sustainability? 

We will convene on April 22, 2021 from 12 – 1 PM EDT. RSVP here for a personalized Zoom link.

What to expect: This is an interactive session. We are interested in “taking the pulse” and hearing your views about these issues. We won’t have presentations or panels. We’ll have prompts with questions and rely on participants to keep the conversation going. We’ll have collaborative notes document for participants to drop in links to resources as well as take notes about our discussion. Afterwards, we’ll publish a blog post summarizing what we discussed on this blog. The conversation will be moderated by the P-WG leaders: Limor Peer, Daniel Oberski, and Vicky Rampin.

Important information:

• Conversation: April 22, 2021, 12 -1 PM EDT (16:00 UTC) [registration link]
• Collaborative notes: (google doc)

About this blog: P-WG is extending an invitation to the community to engage in conversation about topics related to the reproducibility of computational research via the blog, “Taking the Pulse.” In a series of open community meetings, the P-WG will explore issues related to reproducibility and document the views of different communities, culminating in recommendations informed by current practices. The P-WG will hold a forum for conversation about these topics over the course of several months:

The post Reproducibility SOLUTIONS: Join the discussion appeared first on EIG Repro.

The working definition of reproducibility for this group is the ability of a different team to arrive at the same scientific results using the same experimental setup, as described in the recent report on Reproducibility and Replicability in Science by the National Academies of Science, Engineering, and Medicine (NASEM). “For computational experiments,” the ACM adds, “this means that an independent group can obtain the same result using the author’s own artifacts.”

After brief introductions and background on the goals of the Emerging Interest Group (EIG) on Reproducibility and Replicability and the P-WG, the group expressed its priorities for the conversation: The labor involved in reproducible research garnered the greatest interest, followed by issues relating to open source, and to ethics and research integrity.

For each topic, we present the questions posed to the group, and a summary of the main themes and questions.

Labor

Who is responsible for the work of reproducibility? How should that work be rewarded?

For reproducibility to be ubiquitous, it must be understood as a broad expectation and as inherent to the scientific enterprise.

• How to effectively instill the notion that research should be reproducible?
• How to think about incentives that are normative, not only transactional?
• Do we have clear expectations and guidelines for reproducibility? What are the minimal requirements for reproducibility? What are acceptable exceptions for reproducibility and how should that be treated?

Reproducibility involves labor. It is an investment of time and expertise. Ideally, reproducibility labor is baked into the research process so that the effort it takes is indistinguishable from the research effort itself.

• How do we measure the extent and the value of reproducibility labor?

Reproducible research is ultimately conducted by researchers. In reality, however, labor related to reproducibility often involves specialized actions that are treated as separate from the core research process. This labor is often considered “extra” and the responsibility for undertaking it gets shifted on to others in the research team, often graduate students who may not receive appropriate training – or recognition – for the work.

In addition to reproducible practices incorporated in the research process itself, reproducibility also involves labor on the part of users of the research. That labor is even less rewarded, incentivized, or recognized. The archiving and preservation community is an important ally.

• How do we make reproducibility labor visible? “It takes a village.”

It is not clear if researchers are held accountable for the reproducibility of their research, or whether they should be. Currently, reproducibility is a choice for most researchers and the incentives for doing reproducible research are only somewhat effective. Efforts around citations can increase recognition, an important reward in academia. But most incentives have proved insufficiently effective. Publishers can help enforce reproducibility standards that emerge from the community; both funders and publishers have an important role to play but cannot in themselves drive change.

• There should be accountability for reproducible research.
• This community can signal to ACM that it wants reproducibility to be consequential.

Open source

What are the SIGs’ position on open-source solutions vs. proprietary solutions that support reproducibility? Are there promising approaches currently being tested or implemented, for example, around licensing?

At our meeting, open source software is preferred, but it was recognized as not always available or possible. The base position should be, “open when possible.” Generally, the consensus is that tools or platforms can be a step towards improving reproducibility and the community should embrace experimenting with different tools. However, there is no approved list of tools, or a checklist of tool attributes for reproducibility. For example, vendor lock-in is problematic when it applies to a component deemed an essential part of the reproducibility tool chain.   

• What is the ACM’s position on including proprietary software in the reproducibility tool chain, and on how to manage interaction between open and proprietary tools?
• Is there a set of recommended tools that meet minimum requirements for reproducibility? And should the ACM maintain such a list? (for example, what would be required to make something like Matlab work within a reproducibility framework?)
• What is the ACM’s position on workflow steps that are not automated or machine readable?

Closed or proprietary software is an impediment to reproducibility. At its core, this is an issue of access. The community acknowledges restricted access to data – for legal, ethical, and privacy considerations – and has devised ways to accommodate it. What about software?

• What is the ACM’s position with respect to restricted access to software (e.g., because of cost, licensing)? Should commercial interests of private companies be accommodated in the same way as restricted access to data?
• What is the ACM’s position on the 2020 Principles of Open Scholarly Infrastructure which state that, “Open source – All software required to run the infrastructure should be available under an open source license”?
• What is the ACM’s position on sustainability of free and open software, and about organizations such as CZI and NumFocus as a way to support essential parts of the tool chain?

Ethics

What are the stated ethical guidelines that matter for reproducibility? e.g., algorithmic bias, unfair data practices. How does and should reproducibility touch on issues of research integrity? e.g., fraud, abuse, questionable research practices, etc.

The extent to which questions of ethics overlap with reproducibility needs to be clarified. It could be argued that ethical questions relating to research conduct and integrity are directly relevant to the discussion of reproducibility. Behaviors involving data fraud, online harassment, and bad faith reproducibility attempts fall under this category. The ACM Code of Ethics lists general principles that should apply to all parties involved in reproducibility.

• How effective is the ACM Code of Ethics?
• Are ethical violations relating to reproducibility a special case?

Also relevant are other timely topics that fall under ethics, such as algorithmic bias and unethical data scraping and analysis, which are at the intersection of ethics and computation.

• What lines can we draw between reproducibility and concerns about unfair research practices?

By Limor Peer and Vicky Rampin

April 8, 2021

The post Reproducibility PRINCIPLES: Taking the pulse appeared first on EIG Repro.

This leaves many questions to be sorted out. Over the next few months, we will explore related issues such as principles, solutions, training, publishing, and preservation in the context of reproducibility.

We invite the community to join us for these essential conversations.

We begin with fundamental principles related to reproducibility. What are the principles we can agree on? What still needs to be clarified? To start the conversation, we identify a few key issues:

• Open source: What are the SIGs’ position on open-source solutions vs. proprietary solutions that support reproducibility? Are there promising approaches currently being tested or implemented, for example, around licensing?
• Ethics: What are the stated ethical guidelines that matter for reproducibility? e.g., algorithmic bias, unfair data practices.
• Research integrity: How does/should reproducibility touch on these issues? e.g., fraud, abuse, questionable research practices, etc.
• Labor: Who is responsible for the work of reproducibility? How should that work be rewarded?

Join us March 25, 2021 at 12 PM (EDT)!

Conversation will be moderated by the P-WG leaders: Limor Peer , Daniel Oberski , Vicky Rampin

Important information:

• Conversation: March 25, 2021, 12 -1 PM EDT (5-6 PM CET) [registration link]
• Collaborative notes: (google doc)
• Resources: The ACM has led significant efforts toward reproducibility (see summary)

What to expect: This is an interactive session. We are interested in “taking the pulse” and hearing your views about these issues. We’ll take a similar approach for each of the issues: An initial blog post describing the issue, community feedback via open meeting and online conversation, and proposed recommendations to advance the issue.

—

About this blog: P-WG is extending an invitation to the community to engage in conversation about topics related to the reproducibility of computational research via the blog, “Taking the Pulse.” In a series of open community meetings, the P-WG will explore issues related to reproducibility and document the views of different communities, culminating in recommendations informed by current practices. The P-WG will hold a forum for conversation about these topics over the course of several months starting in March 2021. Each month we will focus on a particular topic related to reproducibility: principles, solutions, training, publishing, and preservation.

The post Reproducibility PRINCIPLES: Join the discussion appeared first on EIG Repro.