Blog

This month at Edinburgh ReproducibiliTea we hosted Sean Smith, a Data Analyst in the CAMARADES group at the University of Edinburgh’s Centre for Clinical Brain Sciences. Sean discussed his work with iRISE, an EU project for improving Reproducibility In SciencE. The project seeks to build evidence-based knowledge around what interventions are (and aren’t) effective for improving reproducibility. Sean’s work within this project, iRISE-SOLES (Systematic Online Living Evidence Summaries), aims to produce an overview of the existing interventions to improve reproducibility. You can view the output of Sean’s work on the iRISE-SOLES webpage. During this session, Sean walked us through his workflow for data curation in three steps: Identification, screening, and annotation.

1. Identification

iRISE-SOLES identified over 123,000 relevant papers using a string search over multiple databases (including Web of Science, Medline, PsychInfo, and others). About 55,000 duplicates were removed, leaving 68,995 articles to work with.

2. Screening

From the ~69,000 articles identified, 5,000 of these were randomly selected to be manually screened for relevance by two people. Screened papers were then used to train and test a machine learning algorithm. After algorithm refinement, the remaining articles were processes. This resulted in 16,832 relevant articles which evaluated interventions to improve reproducibility.

3. Annotation

The iRISE-SOLES project used automated methods to annotate articles for:

1. Intervention evaluated
2. Intervention provider
3. Target population
4. Discipline
5. Research stage affected
6. Location
7. Outcomes, and
8. Evidence type (controlled or uncontrolled studies).

Normally in the annotation process, regular expressions (RegEx) would be used to tag mentions of particular words and phrases. RegEx wasn’t well suited for this work, though, so the iRISE team decided to utilise large language models (LLMs) to annotate instead. LLMs are artificial intelligence systems which are trained to generate human-like responses. They can be used to generate text and code, among other tasks. The iRISE-SOLES project assessed the suitability of OpenAI (ChatGPT-4o), Meta (Llama 3), and Mistral AI platforms, eventually choosing to move forward with ChatGPT-4o. They also determined the ideal amount of data to include (paper title and abstract versus title, abstract, and methods) and settled on using paper title, abstract, and methods. Evaluating the full texts was too computationally intensive to be practical. They also considered the method of LLM use (querying, fine-tuning, zero-shot learning, few-shot learning, embeddings, etc.). They refined their annotation prompts and evaluated the performance of the LLM.

Initial Results

ChatGPT-4o generated better annotations in some categories than in others. It performed particularly well in identifying the target population and discipline in each paper, but struggled with the intervention provider. Looking more deeply into this, iRISE-SOLES found that the model performed better when annotating controlled versus uncontrolled studies. They decided to split the dataset into controlled and uncontrolled studies, finding overall improved annotations when using only the 2,441 controlled studies. ChatGPT-4o also performed best with few-shot learning, where a few examples were included within prompts.

The methodology for this type of study is difficult to reproduce, but the iRISE-SOLES team took several steps to combat this. They:

1. Specified the following arguments within the LLM:
   • Set seed (controlling randomness of model outputs)
   • Temperature (lower values create a more deterministic, less creative response)
   • Presence penalty (controls chances of the model creating new tokens), and
   • Version (used the exact sample model version each time).
2. Restricted category choice by asking ChatGPT-4o to choose from a list of categories instead of generating its own.
3. Evaluated performance by:
   • Being transparent at each stage,
   • Providing a range of expected results if the study were to be reproduced, and
   • Identifying trends in the data, not generating perfect annotations.

Next steps

Sean concluded by suggesting the next steps for the iRISE-SOLES project. They plan on making additions to the app available online. They will also make the database ‘living’ with regular automated updates. Additionally, they hope to improve and validate the LLM performance with updated models. Finally, they hope that SOLES will reach different users who might make recommendations to broaden its scope, and to extend the project beyond a systematic review. The iRISE-SOLES project can be viewed online here, which includes a dashboard of their data and more detailed information.

This blog post was written by Alex Colety

SOCIALS:

Edinburgh RT Twitter  
Edinburgh RT YouTube Channel 
Edinburgh RT OSF page  
Edinburgh RT mailing list  

For any questions/suggestions, please send us an email at [email protected]   

Welcome back to ReproducibiliTea for 2024 – 2025! At this first session, Emma Wilson gave a brief (re)introduction to open research and reminded us of the resources available around the University of Edinburgh.

Open Research

UNESCO defines open science, which we like to consider more broadly as open research, as “a set of principles and practices that aim to make scientific research from all fields accessible to everyone for the benefit of scientists and society as a whole. Open science is about making sure not only that scientific knowledge is accessible but also that the production of that knowledge itself is inclusive, equitable and sustainable.” They go on to identify four central values to open science:

• Quality/integrity
• Collective benefit
• Equity/fairness, and
• Diversity/inclusiveness

These values and guiding principles are discussed more thoroughly in UNESCO’s introduction to open science.

The Turing Way is another resource for understanding and generating reproducible research. Their book is available for free here. The Turing Way further specifies that differences between reproducible, replicable, robust, and generalizable data, though we broadly consider them all to fall under the umbrella of reproducibility.

Edinburgh ReproducibiliTea

Edinburgh ReproducibiliTea is an early career researcher (ECR) led journal club / seminar series focused on open research. It’s part of a global network of ReproducibiliTea clubs, and in Edinburgh is part of the larger Edinburgh Open Research Initiative (EORI).

Our casual ReproducibiliTea sessions are held on Microsoft Teams, usually on the second Friday of the month from 10 to 11am (UK time). They’re open to everyone including those from outside the University of Edinburgh. We share information through our mailing list, recordings of the main presentations on our YouTube channel, and written summaries on our blog.

EORI is a grassroots initiative at the University of Edinburgh. You can get involved by joining our Microsoft Teams channel. We are volunteer-run and always looking for more people to join the committee–please get in touch if you’re interested! We can be contacted via email at [email protected].

Centralized research support is also available to those at the University of Edinburgh through the library services. Learn more about these resources on their website here.

Next Session

The next ReproducibiliTea meeting will be on Friday the 11th of October from 10 to 11am (UK time). The topic will be the iRISE project (improving Reproducibility In SciencE), presented by Sean Smith. The presentation will be recorded and made available, but discussion sessions are not recorded so please come along to participate!

Links from this session

UNESCO Open Science Recommendations: https://unesdoc.unesco.org/ark:/48223/pf0000379949  
The Turing Way: https://book.the-turing-way.org/  

Edinburgh Open Research Support

Website: https://library.ed.ac.uk/research-support/open-research  
Newsletter: http://eepurl.com/hHLoIP 

Edinburgh ReproducibiliTea / Edinburgh Open Research Initiative

Mailing List: https://forms.gle/aDQyZRV6kJtor2TR9   
YouTube: https://www.youtube.com/channel/UC9y6VX6Dvs4-vC8eDuOKpNQ 
Website: https://edopenresearch.wordpress.com/   
Microsoft Teams Channel: https://teams.microsoft.com/l/team/19%3aaa9d8bdc2aaa4cf397d1f68069f26b83%40thread.tacv2/conversations?groupId=2a13e96d-71bc-4d55-b895-659694a7da70&tenantId=2e9f06b0-1669-4589-8789-10a06934dc61   
Email: [email protected]   

This blog post was written by Alex Colety

SOCIALS: 

Edinburgh RT Twitter 
Edinburgh RT OSF page 
Edinburgh RT mailing list 

For any questions/suggestions, please send us an email at [email protected]  

For the April 2024 session of ReproducibiliTea, we discussed big team science (BTS) with Dr Drew Altschul. Drew is currently a fellow in the department of Psychology at the University of Edinburgh. He is also a member of the BTS group ManyPrimates.

To start, Drew gave us an overview of big team science. BTS involve many collaborators who may be spread across different labs, disciplines, institutions, and cultures. Collaborating along this scale allows researchers to address questions that may not be answerable on smaller scales. BTS is suited for generating large sample sizes and generalizable results, which can be used to expand the scope of existing findings. BTS has notably increased since 2010 and there are now several conferences dedicated to BTS. Current BTS groups include the Psychological Science Accelerator (PSA), ManyBabies, and the Collaborative Replication and Education Project (CREP). And, of course, ManyPrimates.

Drew followed by discussing the ManyPrimates projects and giving insight into their processes. ManyPrimates is comprised of about 200 researchers from 50+ institutions based in approximately 30 countries. There are 60+ species represented by hundreds of animals. The researchers democratically decide on projects to pursue, collaborating through email lists, GitHub, OSF, Slack, and Google Drive. Generally, the study process is as follows:

1. Members democratically choose a research topic,
2. Members build voluntary taskforce to explore the literature, build a paradigm, and preregister the study
3. The taskforce puts out a call for sites to collect data
4. The taskforce liaises with the sites to make sure setup and data production are appropriate
5. After a specified window of time, data collection ends
6. Data are analyzed and write-up is done collaboratively

Drew noted that this process can be slow moving, as it involves many collaborators working voluntarily. As such, maintaining interest and keeping projects sustainable can be a challenge. Other challenges are leadership accountability and growing diversity. For primates in particular, many animals are located in the global south and so require cross-culture communication and collaboration. Funding is another primary challenge—journals, universities, and funders have not necessarily caught up to the idea that BTS studies produce meaningful research.

The presentation was followed by a discussion session (not included in the session recording). We talked about conservatism within large groups, processes for data management on this scale, and the barriers to BTS becoming more mainstream.

You can learn more about ManyPrimates at their website: https://manyprimates.github.io/

The session recording is available on our YouTube channel.

This month’s blog was written by Alex Colety.

SOCIALS:

Edinburgh RT Twitter
Edinburgh RT OSF page
Edinburgh RT mailing list

For any questions/suggestions, please send us an email at [email protected]

In the November 2023 session of Edinburgh ReproducibiliTea, we heard from three speakers on the advice and experiences with data management.

Simon Smith, who works in research data support for the University of Edinburgh, offered tips and anecdotes from his experiences. He made the following recommendations for researchers:

1. Know why you do what you do: Be aware of funder and institutional policies for data management and sharing early on, so that you can plan for this and integrate it into your project. (See the University of Edinburgh data protection policy and research data management policy).
2. Know what you’re doing it to: Data is a vague term, but generally can be defined as any materials required to support and understand your findings—it’s more than just spreadsheets! Beyond numerical data, ‘data’ may encompass your code, detailed methodology, and project context.
3. Documentation: Make sure to include enough information (including naming conventions) for another person to be able to access, interpret, and analyse your data. Do this throughout the project instead of having to recall everything later.
4. Don’t forget to deposit your data: Share your data in a repository (ideally with a cc-by license). Make it as open as possible and promote it in publications, on social media, with DOIs and ORCID numbers.
5. Keep data safe: Utilise safe storage, such as network storage provided by the University. Be mindful of the communication platforms you’re using for data collection—Teams is secure for participant interviews, but Zoom and WeChat are not. Be aware of your institution’s data protection policy, GDPR requirements, and ethical responsibilities.
6. Plan data management from the beginning: Create a data lifecycle from data creation to preservation. This will support a straightforward and efficient project.

Simon concluded by emphasising the importance of dedicating time to data management as an essential part of any project. For information on the Research Data Support team, visit their webpage here.

Following Simon’s talk, Dr. Kaitlyn Hair shared her experiences with data management for developing systematic reviews. Kaitlyn is currently a postdoctoral researcher in the CAMARADES (Collaborative Approach to Meta Analysis and Review of Animal Data from Experimental Studies) group at the University. She discussed the pros and cons of various platforms for systematic reviews, including citation managers (such as EndNote and Zotero), R, SQL databases, and collaborative platforms (such as SyRF). Platforms like R offer high user customisation but can be challenging to navigate if you haven’t used them before. Comparatively, SyRF is much more collaborative and user-friendly, but users should be conscious about how the platform stores metadata and information uploaded from other formats. Generally, her advice is to seek advice early on from others with more experience, to determine a detailed protocol, and to consider using a structured database for larger projects.

We concluded the session with Clara Sánchez-Izquierdo Lozano, who summarised her masters research on the impact of romantic relationships on University students’ mental health. She used an initial questionnaire followed by seven daily ecological momentary assessment (EMA) surveys which measured participant stress, depression, anxiety, and emotion regulation strategies. Clara’s study found different relationships between mental health metrics and student relationship status when analysing data from the initial questionnaire versus the daily EMA. Regarding data management, the study was successful in receiving a high sample size for the initial questionnaire. However, they ran into some issues with a low EMA sample size (though still relatively high compared to other EMA studies), questionnaire malfunctions, and technical challenges with the SEMA3 app that participants used.

Common themes that came up through the session were the importance of planning a data management strategy early on, utilising others for assistance, and that challenges will likely arise regardless.

Visit out YouTube channel to view the full session recording.

This blog is written by Alex Colety.

SOCIALS:

Edinburgh RT Twitter
Edinburgh RT OSF page
Edinburgh RT mailing list

For any questions/suggestions, please send us an email at [email protected]

Our final Edinburgh ReproducibiliTea session of the 2022-2023 term was all about Octopus. No – not the animal! Octopus.ac is a new platform for registering and publishing research in a way that’s free, fast and fair for everyone.

We were joined by Dr Alexandra Freeman from Cambridge University and Tim Fellows from Jisc who both work on creating and developing the Octopus platform.

Alexandra began the talk with an introduction of how Octopus came about and the problems it aims to fix. She drew on her experience working with the media to discuss how journals’ goal of publishing interesting and “impactful” stories conflicts with their purpose as a primary research record.

Octopus aims to overcome this problem by becoming a new platform for publishing primary research records. Alexandra notes that many parts of the research process involve different skillsets, resource requirements and expertise. In contract to journal articles, Octopus breaks research down into eight different types of publication (like the eight limbs of an octopus):

1. Problem
2. Hypothesis
3. Methods / protocol
4. Results / data
5. Analysis
6. Interpretation
7. Real world application
8. Peer review

Each component can have different authors and receives its own DOI. Different components can also branch off from each other, for instance two methods for testing the same hypothesis or two analyses of the same data.

In addition to registering and publishing new work, researchers can also add their open access publications to Octopus.

Tim finished the talk with a live demo of Octopus, and the session ended with a Q&A. Please note the Q&A was not recorded.

For more information on Octopus and to publish or register your own work, visit their website and follow them on Twitter @octopus_ac.

You can also watch the recording of the session on YouTube.

This blog is written by Emma Wilson

SOCIALS:

• Edinburgh RT Twitter
• Edinburgh RT OSF page
• Edinburgh RT mailing list

For any questions/suggestions, please send us an email at [email protected]

In our April 2023 session of Edinburgh ReroducibiliTea, Michelle Dodd spoke about replication studies within autism research, and shared her personal experiences on why robust and reproducible autism research is important.

Michelle began with a short introduction explaining what common terms used by the autism community mean. From her slides:

• Autism: a type of neurology that is characterised by social-communication differences and restricted, repetitive behaviours and interests.
• Neurodiversity: naturally occurring different neurology which can strengthen society with their diversity. No way of being is better than another and minority neurotypes, such as autistic people, are subject to similar challenges and stigma as other minority groups.
• Neurodivergent: a single person can’t be neurodiverse so they are neurodivergent (or neurotypical).

Previously, researchers thought that autistic people had communication deficits as autistic people often struggle to communicate with non-autistic people. However, we know that that’s not true because autistic people can communicate with each other. The communication differences between autistic and non-autistic people is known as the Double Empathy Problem.

The Double Empathy Problem was investigated in 2018 by Catherine Crompton, Sue Fletcher-Watson and others at the University of Edinburgh, which confirmed that autistic people have a different social interaction style, rather than a deficit, compared to non-autistic people (1).

Michelle is one of the researchers conducting a replication of this study. This time the team are taking an open research approach by publishing protocols on the Open Science Framework, writing a Registered Report, and increasing the sample size of participants (2).

Watch the recording of Michelle’s presentation on YouTube.

References

1. Autistic peer-to-peer information transfer is highly effective
2. Open science in experimental autism research: a replication study of information transfer within and between autistic and non-autistic people

This blog is written by Emma Wilson

SOCIALS:

• Edinburgh RT Twitter
• Edinburgh RT OSF page
• Edinburgh RT mailing list

For any questions/suggestions, please send us an email at [email protected]

In this month’s Edinburgh ReproducibiliTea session, we were joined by Neil Coleman (they/them) who has recently joined the Edinburgh of Edinburgh as the Library Citizen Science Engagement Officer.

Neil introduced us to the concepts of citizen science and participatory research and explained the different levels of citizen science – from using “citizens as sensors” in a crowdsource project to collaborating with citizens to define problems and collect and analyse data in “extreme” citizen science.

Neil also provided some examples of participatory research projects. Check them out below!

• Mapping for Change (https://mappingforchange.org.uk/)
• Living with Machines (https://www.zooniverse.org/projects/bldigital/living-with-machines)
• Genetics of Taste Lab (https://www.dmns.org/science/health-sciences/projects/genetics-of-taste/)
• ROMIO (Randomised Oesophagectomy: Minimally Invasive or Open, a feasibility study) (https://fundingawards.nihr.ac.uk/award/10/50/65)

Finally, Neil highlighted some of the ways the Library can help University of Edinburgh staff and students get involved with participatory research, including using pre-existing resources (Collections, Digital Research Services, Scholarly Communications, and Physical Spaces and Event Management) and developing new services, both internally and externally, such as networking across the university and connecting (ethically) with participants and communities.

Following Neil’s presentation, we had a discussion about our own experiences with participatory research, and potential challenges or barriers to conducting participatory research. This part of the session is not included in the recording.

The session recording is available on our YouTube channel.

This blog is written by Emma Wilson

SOCIALS:

• Edinburgh RT Twitter
• Edinburgh RT OSF page
• Edinburgh RT mailing list

For any questions/suggestions, please send us an email at [email protected]