1. Identification

iRISE-SOLES identified over 123,000 relevant papers using a string search over multiple databases (including Web of Science, Medline, PsychInfo, and others). About 55,000 duplicates were removed, leaving 68,995 articles to work with.

2. Screening

From the ~69,000 articles identified, 5,000 of these were randomly selected to be manually screened for relevance by two people. Screened papers were then used to train and test a machine learning algorithm. After algorithm refinement, the remaining articles were processes. This resulted in 16,832 relevant articles which evaluated interventions to improve reproducibility.

3. Annotation

The iRISE-SOLES project used automated methods to annotate articles for:

1. Intervention evaluated
2. Intervention provider
3. Target population
4. Discipline
5. Research stage affected
6. Location
7. Outcomes, and
8. Evidence type (controlled or uncontrolled studies).

Normally in the annotation process, regular expressions (RegEx) would be used to tag mentions of particular words and phrases. RegEx wasn’t well suited for this work, though, so the iRISE team decided to utilise large language models (LLMs) to annotate instead. LLMs are artificial intelligence systems which are trained to generate human-like responses. They can be used to generate text and code, among other tasks. The iRISE-SOLES project assessed the suitability of OpenAI (ChatGPT-4o), Meta (Llama 3), and Mistral AI platforms, eventually choosing to move forward with ChatGPT-4o. They also determined the ideal amount of data to include (paper title and abstract versus title, abstract, and methods) and settled on using paper title, abstract, and methods. Evaluating the full texts was too computationally intensive to be practical. They also considered the method of LLM use (querying, fine-tuning, zero-shot learning, few-shot learning, embeddings, etc.). They refined their annotation prompts and evaluated the performance of the LLM.

Initial Results

ChatGPT-4o generated better annotations in some categories than in others. It performed particularly well in identifying the target population and discipline in each paper, but struggled with the intervention provider. Looking more deeply into this, iRISE-SOLES found that the model performed better when annotating controlled versus uncontrolled studies. They decided to split the dataset into controlled and uncontrolled studies, finding overall improved annotations when using only the 2,441 controlled studies. ChatGPT-4o also performed best with few-shot learning, where a few examples were included within prompts.

The methodology for this type of study is difficult to reproduce, but the iRISE-SOLES team took several steps to combat this. They:

1. Specified the following arguments within the LLM:
   • Set seed (controlling randomness of model outputs)
   • Temperature (lower values create a more deterministic, less creative response)
   • Presence penalty (controls chances of the model creating new tokens), and
   • Version (used the exact sample model version each time).
2. Restricted category choice by asking ChatGPT-4o to choose from a list of categories instead of generating its own.
3. Evaluated performance by:
   • Being transparent at each stage,
   • Providing a range of expected results if the study were to be reproduced, and
   • Identifying trends in the data, not generating perfect annotations.

Next steps

Sean concluded by suggesting the next steps for the iRISE-SOLES project. They plan on making additions to the app available online. They will also make the database ‘living’ with regular automated updates. Additionally, they hope to improve and validate the LLM performance with updated models. Finally, they hope that SOLES will reach different users who might make recommendations to broaden its scope, and to extend the project beyond a systematic review. The iRISE-SOLES project can be viewed online here, which includes a dashboard of their data and more detailed information.

This blog post was written by Alex Colety

SOCIALS:

Edinburgh RT Twitter  
Edinburgh RT YouTube Channel 
Edinburgh RT OSF page  
Edinburgh RT mailing list  

For any questions/suggestions, please send us an email at [email protected]   

Open Research

UNESCO defines open science, which we like to consider more broadly as open research, as “a set of principles and practices that aim to make scientific research from all fields accessible to everyone for the benefit of scientists and society as a whole. Open science is about making sure not only that scientific knowledge is accessible but also that the production of that knowledge itself is inclusive, equitable and sustainable.” They go on to identify four central values to open science:

• Quality/integrity
• Collective benefit
• Equity/fairness, and
• Diversity/inclusiveness

These values and guiding principles are discussed more thoroughly in UNESCO’s introduction to open science.

The Turing Way is another resource for understanding and generating reproducible research. Their book is available for free here. The Turing Way further specifies that differences between reproducible, replicable, robust, and generalizable data, though we broadly consider them all to fall under the umbrella of reproducibility.

Edinburgh ReproducibiliTea

Edinburgh ReproducibiliTea is an early career researcher (ECR) led journal club / seminar series focused on open research. It’s part of a global network of ReproducibiliTea clubs, and in Edinburgh is part of the larger Edinburgh Open Research Initiative (EORI).

Our casual ReproducibiliTea sessions are held on Microsoft Teams, usually on the second Friday of the month from 10 to 11am (UK time). They’re open to everyone including those from outside the University of Edinburgh. We share information through our mailing list, recordings of the main presentations on our YouTube channel, and written summaries on our blog.

EORI is a grassroots initiative at the University of Edinburgh. You can get involved by joining our Microsoft Teams channel. We are volunteer-run and always looking for more people to join the committee–please get in touch if you’re interested! We can be contacted via email at [email protected].

Centralized research support is also available to those at the University of Edinburgh through the library services. Learn more about these resources on their website here.

Next Session

The next ReproducibiliTea meeting will be on Friday the 11th of October from 10 to 11am (UK time). The topic will be the iRISE project (improving Reproducibility In SciencE), presented by Sean Smith. The presentation will be recorded and made available, but discussion sessions are not recorded so please come along to participate!

Links from this session

UNESCO Open Science Recommendations: https://unesdoc.unesco.org/ark:/48223/pf0000379949  
The Turing Way: https://book.the-turing-way.org/  

Edinburgh Open Research Support

Website: https://library.ed.ac.uk/research-support/open-research  
Newsletter: http://eepurl.com/hHLoIP 

Edinburgh ReproducibiliTea / Edinburgh Open Research Initiative

Mailing List: https://forms.gle/aDQyZRV6kJtor2TR9   
YouTube: https://www.youtube.com/channel/UC9y6VX6Dvs4-vC8eDuOKpNQ 
Website: https://edopenresearch.wordpress.com/   
Microsoft Teams Channel: https://teams.microsoft.com/l/team/19%3aaa9d8bdc2aaa4cf397d1f68069f26b83%40thread.tacv2/conversations?groupId=2a13e96d-71bc-4d55-b895-659694a7da70&tenantId=2e9f06b0-1669-4589-8789-10a06934dc61   
Email: [email protected]   

This blog post was written by Alex Colety

SOCIALS: 

Edinburgh RT Twitter 
Edinburgh RT OSF page 
Edinburgh RT mailing list 

For any questions/suggestions, please send us an email at [email protected]  

To start, Drew gave us an overview of big team science. BTS involve many collaborators who may be spread across different labs, disciplines, institutions, and cultures. Collaborating along this scale allows researchers to address questions that may not be answerable on smaller scales. BTS is suited for generating large sample sizes and generalizable results, which can be used to expand the scope of existing findings. BTS has notably increased since 2010 and there are now several conferences dedicated to BTS. Current BTS groups include the Psychological Science Accelerator (PSA), ManyBabies, and the Collaborative Replication and Education Project (CREP). And, of course, ManyPrimates.

Drew followed by discussing the ManyPrimates projects and giving insight into their processes. ManyPrimates is comprised of about 200 researchers from 50+ institutions based in approximately 30 countries. There are 60+ species represented by hundreds of animals. The researchers democratically decide on projects to pursue, collaborating through email lists, GitHub, OSF, Slack, and Google Drive. Generally, the study process is as follows:

1. Members democratically choose a research topic,
2. Members build voluntary taskforce to explore the literature, build a paradigm, and preregister the study
3. The taskforce puts out a call for sites to collect data
4. The taskforce liaises with the sites to make sure setup and data production are appropriate
5. After a specified window of time, data collection ends
6. Data are analyzed and write-up is done collaboratively

Drew noted that this process can be slow moving, as it involves many collaborators working voluntarily. As such, maintaining interest and keeping projects sustainable can be a challenge. Other challenges are leadership accountability and growing diversity. For primates in particular, many animals are located in the global south and so require cross-culture communication and collaboration. Funding is another primary challenge—journals, universities, and funders have not necessarily caught up to the idea that BTS studies produce meaningful research.

The presentation was followed by a discussion session (not included in the session recording). We talked about conservatism within large groups, processes for data management on this scale, and the barriers to BTS becoming more mainstream.

You can learn more about ManyPrimates at their website: https://manyprimates.github.io/

The session recording is available on our YouTube channel.

This month’s blog was written by Alex Colety.

SOCIALS:

Edinburgh RT Twitter
Edinburgh RT OSF page
Edinburgh RT mailing list

For any questions/suggestions, please send us an email at [email protected]

This February 2024 session, researcher Dr Crispin Jordan from the School of Biomedical Sciences presented a paper by Nguyen et al. (2023) entitled “Biomedical doctoral students’ research practices when facing dilemmas: two vignette‑based randomized control trials” published in Scientific Reports.

This study examined the attitudes of doctoral researchers in biomedical sciences faced with dilemmas related to research integrity, namely the likelihood of them following “detrimental research practices” (DRPs) when under pressure. Interestingly, this was an interventional study – the experiment specifically studied the impact of an unfavourable research environment, with pressure from higher-ups: exposure to a post-doctoral researcher partaking in DRP(s) in similar situation, or a supervisor not opposing it. To achieve this, the authors designed 2 two-arm randomised controlled trials, and created 10 “vignettes”, i.e., a brief story or scenario, with two alternative courses of action, one being a DRP. 630 PhD students were randomised to these scenarios and to whether or not they would have an additional description of a higher-up’s opinion on the matter, to see if this would influence their decision.

Dr Jordan started by detailing ten different detrimental research practices that were selected to inform the vignettes. Most of these were on the theme of manipulating aspects of your analysis or writing of a manuscript to frame results in a more favourable light: changing the focus of your study post hoc, exaggerating results, HARKing (“Hypothesising After Results are Known”), “fishing” results (i.e., running multiple analyses and only reporting favourable ones), or excluding data from analysis. Others related more to publishing pressures: adding a big name to your manuscript even if they didn’t meet the threshold for authorship, giving in to peer review pressure and modifying study outcomes or conclusions to increase your chances of acceptance, or publishing results in two or more different places without proper disclosure.

Worth mentioning are a couple of points found in the “participant characteristics” section, highlighted by Dr Jordan. Nearly half of all PhD students indicated experiencing a great deal of pressure to publish their work, and nearly 2/3rds had received research integrity training. However, this training appeared to consist of a single session during their PhD for 65% of students.

And finally, the results of this study? They differed depending on which DRP was the object of the vignette, but there seemed to be a high “background probability” of students to partake in DRPs, i.e., quite a few students favoured the DRP before we even consider the intervention (negative influence from a superior), which could be concerning. However, in better news, exposure to a postdoc or supervisor willing to let these detrimental practices occur didn’t seem to negatively influence students at all. Interestingly, the effect of research integrity training on likelihood to partake in DRPs was not statistically significant.

We followed this presentation with a brief discussion (off-camera!).

The recording of this session can be viewed on YouTube.

This month’s blog was written by Fiona Ramage.

SOCIALS:

Edinburgh RT Twitter
Edinburgh RT OSF page
Edinburgh RT mailing list

For any questions/suggestions, please send us an email at [email protected]

In our last ReproducibiliTea session before the winter break, we welcomed Kaveh Bazargan to give a fascinating talk about his work on detecting problematic images and figures in scholarly publications. Kaveh developed an early interest in graphics doing research on display holography and has spent decades of his career working in scholarly publishing. This then translated to an interest in detecting problematic images and figures in academic publishing.

The detection of manipulated or otherwise problematic imaging has gained traction in the scientific community, with many key actors (see: Elisabeth Bik, Cheshire, Sholto David, and Smut Clyde) attracting social media followings and openly communicating their findings on platforms such as X (formerly Twitter). This work has been invaluable to the scientific community but has not been well received by all, with some whistle-blowers being subject to online threats. Reporting of any concerns over manipulated images currently tends to happen through the website PubPeer, which provides a public platform for commenting on and communicating any concerns with academic publications.

Kaveh proceeded to give an in-depth demonstration of different ways by which images and figures may be falsified or manipulated in scientific publications, illustrated with real examples – we highly recommend watching the recording of his talk (linked below) to view these! He started demonstrating the use of so-called “tortured phrases” to evade detection of plagiarised text by specialised software, resulting in often amusing phrasings that don’t quite manage to convey the original meaning. For image manipulation, Kaveh talked us through different manipulations, from fairly simple duplications of images or parts of an image or image rotations, to more elaborate methods such as using different image scaling, adding points or hiding figure labels, or using clone stamp-style tools in software such as Photoshop.

Kaveh also offered some insight into how we may be able to detect these types of manipulations. Other than simply having a keen eye, we may consider overlaying the copy with the original to spot similarities, or matching background noise of different images, proving their similarity. We may also be more suspicious of images with small filesizes. Finally, our speaker offered some recommendations: namely the use of vector files for images, to demand and publish high resolution images, and minimising image compression.

The recording of this session (highly recommended!) can be viewed here. Our speaker also kindly provided slides used during the presentation, viewable here.

This month’s blog was written by Fiona Ramage.

SOCIALS:

Edinburgh RT Twitter
Edinburgh RT OSF page
Edinburgh RT mailing list

For any questions/suggestions, please send us an email at [email protected]

Simon Smith, who works in research data support for the University of Edinburgh, offered tips and anecdotes from his experiences. He made the following recommendations for researchers:

1. Know why you do what you do: Be aware of funder and institutional policies for data management and sharing early on, so that you can plan for this and integrate it into your project. (See the University of Edinburgh data protection policy and research data management policy).
2. Know what you’re doing it to: Data is a vague term, but generally can be defined as any materials required to support and understand your findings—it’s more than just spreadsheets! Beyond numerical data, ‘data’ may encompass your code, detailed methodology, and project context.
3. Documentation: Make sure to include enough information (including naming conventions) for another person to be able to access, interpret, and analyse your data. Do this throughout the project instead of having to recall everything later.
4. Don’t forget to deposit your data: Share your data in a repository (ideally with a cc-by license). Make it as open as possible and promote it in publications, on social media, with DOIs and ORCID numbers.
5. Keep data safe: Utilise safe storage, such as network storage provided by the University. Be mindful of the communication platforms you’re using for data collection—Teams is secure for participant interviews, but Zoom and WeChat are not. Be aware of your institution’s data protection policy, GDPR requirements, and ethical responsibilities.
6. Plan data management from the beginning: Create a data lifecycle from data creation to preservation. This will support a straightforward and efficient project.

Simon concluded by emphasising the importance of dedicating time to data management as an essential part of any project. For information on the Research Data Support team, visit their webpage here.

Following Simon’s talk, Dr. Kaitlyn Hair shared her experiences with data management for developing systematic reviews. Kaitlyn is currently a postdoctoral researcher in the CAMARADES (Collaborative Approach to Meta Analysis and Review of Animal Data from Experimental Studies) group at the University. She discussed the pros and cons of various platforms for systematic reviews, including citation managers (such as EndNote and Zotero), R, SQL databases, and collaborative platforms (such as SyRF). Platforms like R offer high user customisation but can be challenging to navigate if you haven’t used them before. Comparatively, SyRF is much more collaborative and user-friendly, but users should be conscious about how the platform stores metadata and information uploaded from other formats. Generally, her advice is to seek advice early on from others with more experience, to determine a detailed protocol, and to consider using a structured database for larger projects.

We concluded the session with Clara Sánchez-Izquierdo Lozano, who summarised her masters research on the impact of romantic relationships on University students’ mental health. She used an initial questionnaire followed by seven daily ecological momentary assessment (EMA) surveys which measured participant stress, depression, anxiety, and emotion regulation strategies. Clara’s study found different relationships between mental health metrics and student relationship status when analysing data from the initial questionnaire versus the daily EMA. Regarding data management, the study was successful in receiving a high sample size for the initial questionnaire. However, they ran into some issues with a low EMA sample size (though still relatively high compared to other EMA studies), questionnaire malfunctions, and technical challenges with the SEMA3 app that participants used.

Common themes that came up through the session were the importance of planning a data management strategy early on, utilising others for assistance, and that challenges will likely arise regardless.

Visit out YouTube channel to view the full session recording.

This blog is written by Alex Colety.

SOCIALS:

Edinburgh RT Twitter
Edinburgh RT OSF page
Edinburgh RT mailing list

For any questions/suggestions, please send us an email at [email protected]

The three key rules of buffets (and Open Science!) are:

• You might not be able to try everything. You must pick food/ practices that suit your specific diet. You don’t have to go “all in” either.
  
• You don’t have to try everything at once. This rule advocates little steps forwards towards ideal research practices, and the ability to “pick and mix” practices rather than eating the whole buffet at once.
  
• Label everything. To avoid any issues at a buffet or in your research, good documentation is essential.
  

The end of the article as well as our meeting discussion afterwards weighed up whether a “buffet” approach is the right way to proceed. One meeting attendee felt that a buffet approach is insufficient and encourages the perseverance of poor practice, which prompted a discussion on whether privilege is involved in our willingness or our ability to adopt Open Science practices, particularly as Early Career Researchers (ECRs). There was a dialogue about the difference between bottom-up and top-down improvements, with the latter arguably being more important and easier to implement, or less likely to face resistance.

We further argued possible reasons for the persistence of non-reproducible research practices, highlighting that these are rarely malicious and very likely simply due to the pressures placed on academics. It was emphasised that Open Research approaches were very much non-controversial and that practices being effort-intensive was likely the main barrier to their adoption. Greater incentives may also increase their use.  Changing the values we hold in the way we carry out research would be beneficial here, and we discussed how we may be able to do so. University initiatives and policies like DORA and CoARA were mentioned, as were broader initiatives like OPUS and GraspOS. We also discussed the scenario where a student or ECR may face resistance from their supervisor to use Open Research methods, and came up with solutions like an expansion of the University’s virtual suggestions box., either within ReproducibiliTea or more broadly.

Our overall conclusion was that if all researchers are under pressure due to the way the system is set up and this is detrimental to their work, then the obvious solution is to change the system. One attendee summed it up nicely:

“[today’s discussion] shows how much we need cultural change to realise the benefits of open scholarship”.

ReproducibiliTea session recording: a “buffet approach to Open Science”

Blog post written by Fiona Ramage

We were joined by Dr Alexandra Freeman from Cambridge University and Tim Fellows from Jisc who both work on creating and developing the Octopus platform.

Alexandra began the talk with an introduction of how Octopus came about and the problems it aims to fix. She drew on her experience working with the media to discuss how journals’ goal of publishing interesting and “impactful” stories conflicts with their purpose as a primary research record.

Octopus aims to overcome this problem by becoming a new platform for publishing primary research records. Alexandra notes that many parts of the research process involve different skillsets, resource requirements and expertise. In contract to journal articles, Octopus breaks research down into eight different types of publication (like the eight limbs of an octopus):

1. Problem
2. Hypothesis
3. Methods / protocol
4. Results / data
5. Analysis
6. Interpretation
7. Real world application
8. Peer review

Each component can have different authors and receives its own DOI. Different components can also branch off from each other, for instance two methods for testing the same hypothesis or two analyses of the same data.

In addition to registering and publishing new work, researchers can also add their open access publications to Octopus.

Tim finished the talk with a live demo of Octopus, and the session ended with a Q&A. Please note the Q&A was not recorded.

For more information on Octopus and to publish or register your own work, visit their website and follow them on Twitter @octopus_ac.

You can also watch the recording of the session on YouTube.

This blog is written by Emma Wilson

SOCIALS:

• Edinburgh RT Twitter
• Edinburgh RT OSF page
• Edinburgh RT mailing list

For any questions/suggestions, please send us an email at [email protected]

Michelle began with a short introduction explaining what common terms used by the autism community mean. From her slides:

• Autism: a type of neurology that is characterised by social-communication differences and restricted, repetitive behaviours and interests.
• Neurodiversity: naturally occurring different neurology which can strengthen society with their diversity. No way of being is better than another and minority neurotypes, such as autistic people, are subject to similar challenges and stigma as other minority groups.
• Neurodivergent: a single person can’t be neurodiverse so they are neurodivergent (or neurotypical).

Previously, researchers thought that autistic people had communication deficits as autistic people often struggle to communicate with non-autistic people. However, we know that that’s not true because autistic people can communicate with each other. The communication differences between autistic and non-autistic people is known as the Double Empathy Problem.

The Double Empathy Problem was investigated in 2018 by Catherine Crompton, Sue Fletcher-Watson and others at the University of Edinburgh, which confirmed that autistic people have a different social interaction style, rather than a deficit, compared to non-autistic people (1).

Michelle is one of the researchers conducting a replication of this study. This time the team are taking an open research approach by publishing protocols on the Open Science Framework, writing a Registered Report, and increasing the sample size of participants (2).

Watch the recording of Michelle’s presentation on YouTube.

References

1. Autistic peer-to-peer information transfer is highly effective
2. Open science in experimental autism research: a replication study of information transfer within and between autistic and non-autistic people

This blog is written by Emma Wilson

SOCIALS:

• Edinburgh RT Twitter
• Edinburgh RT OSF page
• Edinburgh RT mailing list

For any questions/suggestions, please send us an email at [email protected]

Neil introduced us to the concepts of citizen science and participatory research and explained the different levels of citizen science – from using “citizens as sensors” in a crowdsource project to collaborating with citizens to define problems and collect and analyse data in “extreme” citizen science.

Neil also provided some examples of participatory research projects. Check them out below!

• Mapping for Change (https://mappingforchange.org.uk/)
• Living with Machines (https://www.zooniverse.org/projects/bldigital/living-with-machines)
• Genetics of Taste Lab (https://www.dmns.org/science/health-sciences/projects/genetics-of-taste/)
• ROMIO (Randomised Oesophagectomy: Minimally Invasive or Open, a feasibility study) (https://fundingawards.nihr.ac.uk/award/10/50/65)

Finally, Neil highlighted some of the ways the Library can help University of Edinburgh staff and students get involved with participatory research, including using pre-existing resources (Collections, Digital Research Services, Scholarly Communications, and Physical Spaces and Event Management) and developing new services, both internally and externally, such as networking across the university and connecting (ethically) with participants and communities.

Following Neil’s presentation, we had a discussion about our own experiences with participatory research, and potential challenges or barriers to conducting participatory research. This part of the session is not included in the recording.

The session recording is available on our YouTube channel.

This blog is written by Emma Wilson

SOCIALS:

• Edinburgh RT Twitter
• Edinburgh RT OSF page
• Edinburgh RT mailing list

For any questions/suggestions, please send us an email at [email protected]