🎯 Research Focus: Poisson Log-Normal Models

My research at INRAE will focus on comprehensive benchmarking of Poisson Log-Normal (PLN) models, a cutting-edge statistical framework for analyzing microbial data. This work represents a natural extension of my published research on cross-validation methods for network inference algorithms.

What are PLN Models?

Poisson Log-Normal models provide a sophisticated framework for analyzing count data with complex correlation structures - exactly the kind of data we encounter in microbiome research. These models excel at:

• Principal Component Analysis for high-dimensional biological data
• Abundance prediction in microbial communities
• Network inference for understanding species interactions

The Research Challenge

While PLN models show great theoretical promise, there’s been limited systematic comparison of their performance against traditional approaches. My internship will address this gap by:

1. Benchmarking PLN models across three key applications
2. Running large-scale cross-validation experiments to quantify performance gains
3. Demonstrating practical advantages over generalized linear models for microbial count prediction

💰 Prestigious International Funding

This opportunity has been made possible through an extraordinary combination of international research awards totaling over €25,000:

🇨🇦 PBEEE Research Mobility Scholarship ($12,000 CAD)

The Quebec government’s prestigious scholarship for short-term international research collaborations. This highly competitive award supports research exchanges between institutions in different countries.

🇪🇺 International Mobility Grant (€13,500)

Awarded by the DATAIA Institute, this grant is specifically designed to support research collaboration and student exchange between institutions in different countries.

🇫🇷 INRAE Junior Scientific Mobility Program (€2,300)

The INRAE Hadamard initiative designed to foster scientific exchange and collaboration among researchers.

🌍 Building International Research Networks

This internship represents more than just a research project - it’s about building lasting international collaborations that will shape the future of computational biology research.

Key Collaborators

I’ll be working with leading researchers in statistical modeling and computational biology:

• Dr. Julien Chiquet - Expert in statistical learning for biological networks
• Dr. Mahendra Mariadassou - Specialist in microbiome data analysis
• Dr. Stéphane Robin - Pioneer in statistical models for biological data

Their collective expertise in PLN models, combined with my background in cross-validation and high-performance computing, creates an ideal environment for groundbreaking research.

Bridging Continents

This collaboration creates a unique tri-continental research network:

• 🇺🇸 North America: High-performance computing expertise (NAU’s Monsoon cluster)
• 🇫🇷 Europe: Advanced statistical modeling frameworks (INRAE’s PLN expertise)
• 🇨🇦 Quebec Network: Methodological validation through government-supported collaboration

🔬 Expected Impact

The outcomes of this internship will have broad implications for:

Academic Research

• Methodological advances in statistical modeling for biological data
• Open-source software contributions to the R/Python ecosystem
• Joint publications bridging American and European research communities

Practical Applications

• Improved microbiome analysis for medical diagnosis
• Better understanding of microbial community dynamics
• Enhanced predictive models for ecological and health applications

📈 Looking Forward

The four months in Paris will be intense and productive, but I’m most excited about the long-term collaborations this will establish. International research partnerships are crucial for tackling the complex challenges in computational biology and statistical modeling.

What’s Next?

Over the coming months, I’ll be:

1. Finalizing research protocols and computational approaches
2. Preparing for the cultural transition - brushing up on my French! 🇫🇷
3. Setting up remote access to NAU’s computing resources for seamless collaboration
4. Planning knowledge transfer activities to share insights with both institutions

💭 Reflections on International Research

This opportunity reinforces my belief that the most impactful research happens at the intersection of different expertise, cultures, and approaches. The combination of American innovation in high-performance computing with European sophistication in statistical theory promises to yield insights neither approach could achieve alone.

As I prepare for this journey, I’m grateful for the support from multiple funding agencies that recognize the value of international collaboration in advancing scientific knowledge.

Stay tuned for updates from Paris! I’ll be sharing insights about the research process, cultural experiences, and hopefully some exciting preliminary results as they develop.

Have questions about international research opportunities, funding applications, or PLN models? Feel free to reach out - I’m always happy to discuss research and collaboration opportunities.

🔗 Related Resources

• My BMC Bioinformatics Publication
• INRAE Research Institute
• PBEEE Scholarship Program
• DATAIA Institute

Welcome to the ML with Monsoon project! This blog post will guide you through some of the essential steps in setting up and executing machine learning workflows using the Monsoon HPC cluster. Monsoon is a high-performance computing (HPC) cluster at Northern Arizona University. It is a shared resource that provides researchers with the computational power needed to run large-scale simulations and data analysis. Although Monsoon is a powerful tool, it can be challenging to use for beginners. This blog post aims to provide a step-by-step guide to help you get started with machine learning on Monsoon. The concepts covered in this project will be applicable to any HPC cluster, so even if you are not using Monsoon, you can still benefit from this guide. We will cover how to connect to Monsoon, perform interactive debugging, submit batch jobs, and visualize key results.

Connecting to Monsoon

To start using Monsoon, you can connect via the dashboard or through a secure shell (ssh).

• Access through the monsoon dashboard Navigate to the Monsoon Dashboard and log in with your credentials.

• Access through the secure shell (ssh). Open your terminal and run the following command, replacing with your actual NAU username. You will be prompted to enter your password.

  ssh -Y <username>@monsoon.hpc.nau.edu
  

Interactive/Debug Work

For interactive or debugging tasks, you can request a compute node. Here are two examples:

• Request a node with 4GB of RAM and 1 CPU for 24 hours:

  srun -t 24:00:00 --mem=4GB --cpus-per-task=1 --pty bash
  

• Request a node with 8GB of RAM and 1 CPU for 1 hour to run a Python script.

  srun -t 1:00:00 --mem=8GB --cpus-per-task=1 python analysis.py
  

Submitting Jobs

For longer-running tasks, you can write a job script and submit it to the queue. Here is an example of a job script (jobscript.sh):

#!/bin/bash
#SBATCH --job-name=test
#SBATCH --output=/scratch/da2343/output.txt 
#SBATCH --error=/scratch/da2343/error.txt 
#SBATCH --time=20:00
#SBATCH --mem=1GB
#SBATCH --cpus-per-task=1
python analysis.py

Submit the job script using the following command:

sbatch jobscript.sh

Visualizing Results

Time Graph

Understanding the time performance of your machine learning models is crucial. The following graph illustrates the time taken for different stages in the workflow.

Algorithm Selection

Choosing the right algorithm can significantly impact the performance and accuracy of your model. Below is a graph showing the accuracy of different parallel algorithms.

Hyper-Parameter Tuning

Hyper-parameter tuning is essential for optimizing the performance of your machine learning models. The following graph shows the loss during hyper-parameter tuning.

Conclusion

In this post, we’ve walked you through the essential steps for setting up and managing machine learning workflows on the Monsoon HPC cluster. From connecting to Monsoon and performing interactive debugging, to submitting jobs and visualizing results, these guidelines are designed to help you efficiently utilize Monsoon’s powerful resources.

Harnessing the capabilities of high-performance computing can significantly accelerate your machine learning projects, enabling you to handle larger datasets, run more complex models, and achieve faster results. By following the steps outlined here, you can optimize your workflow, reduce development time, and focus on what matters most—building and refining your machine learning models.

Rperform has great flexibility and usability. It internally supports automating the tests of your code on Github and creating comments to display the results. To set-up Rperform for your GitHub CI worflow, initialize Rperform with the command :

Rperform:init_rperform()

• The above command creates the Github workflow directory, which is the .github/workflow/ directory and populates it with the needed workflow files (rperform-receive.yaml and rperform-comment.yaml).

• It also creates the rperform directory and populates it with files which make test configuration and customisation easier.

Running the script

One of the most important files in the rperform directory is the script.R file, which is where the Rperform test functions should be written. The functions written in the file will be ran when the workflow is triggered. You can write multiple Rperform functions in the script.R file.

With only these two functions, you can run four different types of tests:

1. Rperform::plot_metrics()
2. Rperform::plot_branchmetrics()

NOTE: The save_plot and save_image parameters must always be set to TRUE so that the outputs will be saved and displayed without any issues.

After this set-up, anytime there is a Pull Request in the repository, the workflow will be triggered and the tests will be run. The results will be commented out by Github Bot in the PR.

Customising the PR comment

There are the header.R and footer.R files inside the rperform directory which you can edit to customize the PR comment. The glue package is used to format and interpolate the strings so as to join the different sections of the comment. It must be noted that since any comment text is rendered in Markdown format, you must use the markdown syntax to edit the comment.

Results

Below is an example of the auto comment that is generated when the workflow is triggered by a PR.

Testing Rperform on Popular R packages

Rperform Test 1

• Test on the styler package, main repository here.

• Inside the script.R file, the following changes were made. Basically, the tests paths were updated to run the time_metrics functions. Below is the image of the updated script.R file.

## TEST 1
Rperform::plot_metrics(
  test_path = "tests/testthat/test-rmd.R",
  metric = "time", num_commits = 140,
  save_data = TRUE,
  save_plots = TRUE,
  total_width_in = 20
)

## TEST 2
Rperform::plot_metrics(
  test_path = "tests/testthat/test-escaping.R",
  metric = "time", num_commits = 140,
  save_data = TRUE,
  save_plots = TRUE,
  total_width_in = 20
)

• Sample comment created by running the tests.

Rperform Test 2

• Test on the devtools package, original repository here.

• The script.R file is shown below :

## TEST 1
Rperform::plot_metrics(
    test_path = "tests/testthat/test-utils.R",
    metric = "time", num_commits = 100,
    save_data = TRUE,
    save_plots = TRUE,
    total_width_in = 30
)

## TEST 2
Rperform::plot_metrics(
    test_path = "tests/testthat/test-check.R",
    metric = "memory", num_commits = 100,
    save_data = TRUE,
    save_plots = TRUE,
    total_width_in = 30
)

• Sample comment created by running the tests.

In my previous article, I established the reason why Github Actions was the preferred choice over other tools like Travis CI and Appveyor. The main goal of this project as described here is to get Rperform working and easy to use with Github Actions. GitHub Actions enable DevOps to be done on Github and allows workflows to be run when other events are triggered in the repository. GitHub provides macOS, Linux and Windows virtual machines to run workflows. A workflow is a configurable process defined by YAML file which contains one or more jobs that may be run sequencially or parallel. Each job runs inside it’s own container or runner. Under each job, there are steps that runs a script or an action.

DEVELOPMENT CONSIDERATIONS

The repository

To develop a custom Github Actions, I had to consider whether to keep it in the main repository or create a new repository just for the Github Actions. I decided to keep it in the main repository because it is easier to maintain along with the main package as well. The main reason why I chose to do that was because this particular Github Action is specific to only this package and it is not generalized enough to be used with other packages.

The only issue that I thought of was version control of the action. However, it can be fixed by specifying the version number of the action which must be the same as the version number of the Rperform package. e.g. Rperform v0.0.1:

- uses EngineerDanny/Rperform/actions/receive@v0.0.1

The Type of Action

According to the documentation, there are only three types of actions namely :

• Docker Container which supports only Linux OS
• Javascript which supports Linux, MacOS and Windows
• Composite Actions which also support Linux, MacOS and Windows.

Even though all the other types of actions are very important, I chose to use composite actions. It allows the combination of multiple workflow steps within one action. This feature makes it possible to combine multiple run commands into an action. If a workflow is set-up, it can execute the commands as a single step with that action. This flexibility is the main reason why I adopted this approach.

IMPLEMENTING CUSTOM GITHUB ACTIONS PROTOTYPE

I submitted a PR which intends to create Github Actions to make it easier for package developers to use Rperform to test their code. The Actions are tested on a separate repository stringr.

To make set-up easier, the Github workflow directory, which is the .github/workflow/ directory, is created automatically when you ran Rperform::init_rperform() function. It is also populated with the needed workflow files to be able to run Rperform with ease. It adds the rperform directory and populates it with files which make test configuration and customisation easier. One of the most important files in that directory is the script.R file which is where the Rperform test functions should be written.

When writing the Github Action for Rperform, I divided the action into two parts. These are the receive and comment parts. This decision was taken due to security reasons.

The Receive Action

• This action is represented by EngineerDanny/Rperform/actions/receive
  • Triggered via PR or push to main branch.
  • Reads config.json to prepare and run the benchmark job.
  • Does not have read or write access due to security reasons.
• Below is the script inside receive.yaml file

name: "receive"
description: "Action to run Rperform benchmarks and upload the results."
inputs:
  cache-version:
    description: "Integer to use as cache version. Increment to use new cache."
    required: true
    default: 1
  rperform_ref:
    description: "Which branch or tag of Rperform should be used. Mainly for debugging."
    required: true
    default: "@github-actions"

runs:
  using: "composite"
  steps:
    - name: Checkout repo
      uses: actions/checkout@v3
      with:
        fetch-depth: 0
    - name: Set up git user
      run: |
        git config --local user.name "GitHub Actions"
        git config --local user.email "actions@github.com"
      shell: bash
    - name: Setup R
      uses: r-lib/actions/setup-r@v2
    - name: Install dependencies
      uses: r-lib/actions/setup-r-dependencies@v2
      with:
        cache-version: $
        extra-packages: |
          any::ggplot2
          any::dplyr
          any::gert
          any::glue
          github::EngineerDanny/Rperform$
    - name: Remove global installation
      run: |
        pkg <- unlist(read.dcf('DESCRIPTION')[, 'Package'])
        if (pkg %in% rownames(installed.packages())) {
          remove.packages(pkg)
          cat('removed package ', pkg, '.', sep = "")
        }
      shell: Rscript {0}
    - name: Run benchmarks
      run: Rperform::run_script("rperform/script.R")
      shell: Rscript {0}
    - name: Uploading Results
      run: |
        echo "Uploading results..."
      shell: bash

    - name: Saves the PR number in an artifact
      shell: bash
      env:
        PULL_REQUEST_NUMBER: $
      run: echo $PULL_REQUEST_NUMBER > ./rperform/pr-comment/PR_NO

    - uses: actions/upload-artifact@v2
      with:
        name: pr
        path: rperform/pr-comment/

The Comment Action

• This action is represented by EngineerDanny/Rperform/actions/comment
  • Comments the results on the PR that originated the workflow run.
  • Starts right after the receive job finishes.
  • Will create an additional commit status to the PR check suite.
  • Has read & write access.
• Below is the script inside comment.yaml file

name: "comment"
description: "Action to comment Rperform results on the appropriate PR. Needs read/write access."
inputs:
  GITHUB_TOKEN:
    description: "The GITHUB_TOKEN secret."
    required: true

runs:
  using: "composite"
  steps:
    - name: "Download artifact"
      id: "download"
      uses: actions/github-script@v3.1.0
      with:
        script: |
          var artifacts = await github.actions.listWorkflowRunArtifacts({
             owner: context.repo.owner,
             repo: context.repo.repo,
             run_id: $,
          });
          var matchArtifact = artifacts.data.artifacts.filter((artifact) => {
            return artifact.name == "pr"
          })[0];
          var download = await github.actions.downloadArtifact({
             owner: context.repo.owner,
             repo: context.repo.repo,
             artifact_id: matchArtifact.id,
             archive_format: 'zip',
          });
          var fs = require('fs');
          fs.writeFileSync('$/pr.zip', Buffer.from(download.data));
    - run: unzip pr.zip
      shell: bash
    - name: "Comment on PR"
      id: "comment"
      uses: actions/github-script@v6
      with:
        script: |
          var fs = require('fs');
          var pr_number = Number(fs.readFileSync('./PR_NO'));
          var body = fs.readFileSync('./results.txt').toString();
          await github.rest.issues.createComment({
            owner: context.repo.owner,
            repo: context.repo.repo,
            issue_number: pr_number,
            body: body
          });
          
    - uses: actions/github-script@v5
      if: always()
      with:
        script: |
          let url = '$'
          let any_failed = $
          let state = 'success'
          let description = 'Commenting succeeded!'

          if($ || any_failed) {
            state = 'failure'
            description = 'Commenting failed!'
            if(any_failed) {
              url = "https://github.com/$/actions/runs/" + 
                    "$"
            }
          }
          github.rest.repos.createCommitStatus({
            owner: context.repo.owner,
            repo: context.repo.repo,
            sha: '$',
            state: state,
            target_url: url,
            description: description,
            context: 'rperform comment'
          })

CONCLUSION

This is the prototype of the Github Actions. Initially due to separation of actions, I had an issue with the Github create comment script which was similar to this issue. After several debugging and testing, I was able to successfully automate the creation of comments. (Auto Comment On PR). The actions are implemented in such a way that testing results are firstly uploaded as an artifact on Github. Then afterwards, a comment is created by Github actions bot on the PR. At this point, I only tested it with the plot_metrics function. More functionalities will be added to the actions in time. I am thinking about limiting the number of functions that can be run in the script.R file.

When it comes to publishing an R package, it is very prerequisite and crucial for the package to pass R CMD Check. R CMD Check is basically a test performed on an R package to check code for common problems whilst suggesting ways of fixing them. This means that it is also great tool to even utilize during the whole development process of a package. R CMD Checks comprises of about 50 different tests which are explained here

RUNNING R CMD CHECK LOCALLY

Running R CMD Check on Rperform is a simple task. During development, it can be run locally by using the famous devtools package.

devtools::check()

Also, the R Studio IDE has in-built features which makes it easy to run R CMD Checks right inside the IDE. Under Build section, there is a check button which does that. Using various methods above is synonymous to running the following command on the R console:

R CMD check Rperform

However, when developers need to share or collaborate on a project, the check comes in handy if it can be run online in a container for everyone to see the results. It is more challenging to set-up the checks to run online. Mostly, this approach is used when the need arises to run R CMD check every time changes are made in code. Since we use GitHub, it is recommended that we do that with Github Actions. There are others such as Travis CI and Appveyor but we will stick with Github Actions due to its simplicity and power. In addition to that, it implements modern technologies with vast functionalities and growing community.

It must be taken note that running R CMD Check on any package will return only three types of notices:

ERRORS : These are issues which are very fatal and likely to cause the package or its dependencies to fail to install.

WARNINGS : Warnings are statements that are made to caution the developer to take note of certain vital issues. These issues are unlikely to be detrimental to the package but it is very crucial to not ignore them.

NOTES : Notes are not crucial issues. They are just mild problems which are meant for developers to take note of.

R CMD Check passes successfully when none of the above notices are returned

R CMD CHECK USING GITHUB ACTIONS

GitHub Actions is an open-source technology that makes it easy for automation of software workflows. It has exceptional Continuous Integration/ Continuous Deployment (CI/CD) tools which enables developers to build, test and deploy code right from their GitHub repositories.

It is very exciting that the R community has a number of repositories of Github Actions which can be used to perform any variety of CI tasks for R projects with example workflows. In Rperform, we use the following actions sequencially :

1. actions/checkout - Checks out your repository to the $GITHUB_WORKSPACE so that repository’s workflow can utilize it.
2. r-lib/actions/setup-r - Sets up an R environment for use in GitHub Actions by basically downloading and caching a version of R.
3. r-lib/actions/setup-r-dependencies - Downloads and installs all the packages declared in the DESCRIPTION file.
4. r-lib/actions/check-r-package - Runs R CMD check on an R package.

If every check passes without any discrepancies, it implies that the package is stable. However, if any of the check fails, detailed logs will be shown at the end of every action for you to rectify this issue. It can be very confusing at first when some of the checks fail but with a little bit of debugging and testing, you can get the package to pass. When you check my workflow history on my Rperform repository here, you would notice that I have about 50% failed workflow runs. I wrote an article describing a total review of how I went about fixing the issues with Rperform’s initial dependencies. At that time, I had successive workflow failures (From R-CMD-check #17 to R-CMD-check #21 ). After fixing the various issues that popped up, I submitted a Pull Request explaining the series of failures and the steps I took to fix them.

I will be diving deep into the PR to explain some of the issues encountered.

From the Check-RPerform #3, Github showed a visual evidence that points to r-lib/actions/check-r-package as the specific action that failed.

Initially, the issue was invalid link in the test-repo-metrics.R file. I found out that the hadley/stringr link does not exist. Even though, when you open the link you will be redirected to the actual repository which is the tidyverse/stringr.

My test results were inconsistent (Failing more times and occassionally passing) due to this issue. Therefore, I changed the github link to tidyverse/stringr After running Rperform check locally, even though the test didn’t explicitly give me an error, it failed. The only warning was :

checking Rd cross-references … WARNING Missing link or links in documentation object ‘Rperform-package.Rd’: ‘[git2r:git2r-package]{git2r}’

See section ‘Cross-references’ in the ‘Writing R Extensions’ manual.

I fixed the warning by inserting the git2r official github link.

Although the issue with the link was fixed and R CMD Check on my local machine passed, the GitHub CI action still failed. So I started to dig into the tidyverse/stringr package tests.

Inside the https://github.com/tidyverse/stringr/blob/main/tests/testthat/test-dup.r file, there are three tests which are ran internally.

As shown above, all the other tests passed except the uses tidyverse recycling rules test. This led to my conclusion that, this may be an internal issue with the tidyverse/stringr package.

The first two tests however, could be used in checking Rperform.

FINAL FIX

After several debugging and testing, I finally found the final error that was causing the Rperform R CMD Check test to fail on GitHub CI. It turns out that during my adventures and explorations, I mistakenly created a file( [tests/testthat/test-plot-metrics.R]) on the master branch.

As recorded from the logs above, in the file, there is an issue with the str_c function because it does not exist. Therefore, I removed this file since it’s not needed. It was created for experimental purposes.

After fixing all the issues, all the checks passed ✅ withCheck-RPerform #11 workflow run.

CONCLUSION

During my tests, I realized the cause of the inconsistencies in my workflow results. When running R CMD Check locally, I noticed that if your internet network is slow, because the dependencies and other stuffs can not be downloaded, the test will pass with no warnings or errors. This may be a flaw in the R CMD Check tool. It can lead to false positives. The main reason why I didn’t notice the mistakenly created file locally was because I was on another branch (test-pkg) instead of the main branch (master). The file was rather created on the master branch.

R CMD Checks are very crucial to the success of a package especially when the package is to be published on CRAN. When a package passes the checks with no errors, warnings and notes, there would be no need for human intervention before the package is published on CRAN.

In R, it is the main purpose of the DESCRIPTION file to convey the metadata about a package. It is also needed to list all the dependencies that the package uses. Inside the DESCRIPTION file, there are Imports and Suggests section under which, you can list your dependencies. There is a minor difference between the two. The packages under Imports are needed by your users at runtime. Packages under Suggests however, may be optional because it is either used for development or optional functionalities. There was a major issue with the major dependencies used by Rperform. The issue was that, the versions of the dependencies used were too old and one of the them was discontinued.

The git2r package

The git2r package is a popular R package which is mainly known for using git with R. It gives users programmatic experience with git using R. It is very fast because it implements git core methods developed with C. As the developers of the package improved and updated the package, they made a breaking transition. They made a change in support from S4 objects to S3 objects. A simple fix to this issue will be to pin the version of the package to specific versions in the DESCRIPTION file. However, a lot of developers have always install the latest versions of packages. As I tested Rperform for the first time, I had the issue. This is the error I had :

> plot_metrics(test_path = "tests/testthat/test-join.r", metric = "time", num_commits = 2, save_data = FALSE, save_plots = FALSE)
Error in base::substr(commit_val@summary, start = 1, stop = 15) : 
  trying to get slot "summary" from an object (class "git_commit") that is not an S4 object 

Analyzing the error message suggests that the program is trying to get a property from an object which is not S4 object but rather S3 object. The actual issue was that my git2r version was the latest version. For some reason, the package version which should have been pinned to be <= version 0.21.0 was ignored by my program. The fact is that, the latest git2r package has a lot of new features and fixes. The first task I had to do was updating Rperform to also support the newer versions of git2r so as to make it easier for other developers to use and test the package. I updated the git2r version to (>= 0.30.1) in the DESCRIPTION file so it made some functions in the code obsolete. First of all, the names of some functions were updated. Refactoring the code was not difficult but running through the documentation of the git2r package and checking the newer functions proved to be a little challenging and time consuming. First major change I had to make was replace the head function to repository_head function. This function basically gets the HEAD of a repository.

# From
> repo_head <- git2r::head(repo)

# To
> repo_head <- git2r::repository_head(repo)

Also, I had to check all the functions related to the issue to be compatible to S3 objects. Just to name a few, some of them are :

# From
> base::substr(commit_val@summary, start = 1, stop = 15)
> repo_head@name

# To
> base::substr(commit_val$summary, start = 1, stop = 15)
> repo_head$name

Initially, I used numeric indexes to reference list members. I switched to using named reference approach to reference list members by names instead of numeric indexes because it was less error prone. With most of the changes just like above examples, I just had to replace the @ symbol with the $ symbol. Others were a little more complex than that.

The Animint dependency

The animint package is a forked version of the ggplot2 with improved features. It enables developers to design multi-plot, multi-layer, interactive and animated data visualizations. Unfortunately, the original version has been discontinued. Even though, it was an optional package in the root DESCRIPTION file, it was worth looking into. However, my attention only drew to it when I was running a CMD R Check on the Rperform package and it failed. The Github CI test showed that it is not possible to install it because it could not recognize the package. After digging into the issue, I realized the package has been discontinued. Fast forward, there is an updated version called the animint2 which comes with easier installation and better syntax. Therefore I updated the package to animint2 in the DESCRIPTION file. Then, I checked and tested all the functions that depend on the animint2 package. Most of them had to do with just replacing animint with animint2.

# From
> animint::theme_animint(height = 700)
> animint::animint2dir(plot.list = viz.list, out.dir = paste0(basename(getwd()), "_", "time_animint"))

# To
> animint2::theme_animint(height = 700)
> animint2::animint2dir(plot.list = viz.list, out.dir = paste0(basename(getwd()), "_", "time_animint"))

Fix dependency and related issues

After all the changes and updates, I opened a Pull Request to address the issues. PR#2 and PR#3. As it was a pending issue raised twice on the main repository. These combined PRs close issues #44 and #45.

Conclusion

I have realized that the dependencies that our projects utilize are very crucial to the development of the package and its users. At first, I nearly gave up on the Rperform package because of it’s support for old dependencies which were not already installed and for some reason could not get installed on my computer. Updating dependencies ensures the package gets access to the latest features, improvements and bug fixes. Downgrading the git2r dependency was a temporal fix, updating and ensuring that all related issues are fixed became the permanent fix.

The main goal of the RPerform library is to make performance tests on R packages easier with GitHub. However, currently, the initial version of the RPerform library is outdated in the sense that you would need to improvise before you attempt to test RPerform. This article explains all the potential errors, issues and bugs you might encounter and how to solve them before you run the package.

The test package that is used to test the Rperform library is the stringr (https://github.com/tdhock/stringr) package. Installation of the library has been discussed in the previous articles. Use the library(Rperform) command to load the library. The following command initializes the library by setting the working directory.

setwd(dir= "/Users/newuser/Projects/RProjects/stringr")

DIRECTORY ISSUES

- Working directory error

The first error which might be thrown is :

Error in setwd(dir= "/Users/newuser/Projects/RProjects/stringe") :
 cannot change working directory

The dir argument must be set to the directory of the package or project you want to test.

- test-join directory error

There has been an update with the stringr repository. test-join.r is not in the tests/testthat directory. The new directory is inst/tests/test-join.r . Also, all the files in inst/tests/ directory must be taken note. Misrepresenting them will result in errors. Hence, this implies that running :

plot_metrics(test_path = "tests/testthat/test-join.r", metric = "time", num_commits = 10, save_data = FALSE, save_plots = FALSE)

will throw the error :

Error in file(con , "r") :
    cannot open file "tests/testthat/test-join.r" : No such file or directory    

Instead, run :

plot_metrics(test_path = "inst/tests/test-join.r", metric = "time", num_commits = 10, save_data = FALSE, save_plots = FALSE)

MISSING PACKAGES

These are the packages that Rperform depends on. Not having it pre-installed may prevent you from running the package. We proceed by discussing the relevant dependencies and how to install them.

git2r : The git2r package gives developers access to Git repositories from R. The package uses the libgit2 library which is a C implementation of the Git core methods. Therefore it allows developers to write native speed custom Git applications in any language that supports C bindings.

To install the development version of git2r. You can use :

install.packages("git2r")

it’s easiest to use the devtools package:

# install.packages("devtools")
library(devtools)
install_github("ropensci/git2r")

ggplot2 : The ggplot2 is an R package used for plotting graphs. It provides useful commands that helps to create complex plots from data in a data frame. It enables quality plots to be created with minimal coding, tweaking and adjustments. The package is therefore necessary for Rperform to visualize performance metrics. It provides an interface for determining what variables to plot and how they are shown to the user. Organized data saves massive time when using the package to make figures.

# The easiest way to get ggplot2 is to install the whole tidyverse:
install.packages("tidyverse")

# Alternatively, install just ggplot2:
install.packages("ggplot2")

# Or the development version from GitHub:
install.packages("devtools")
devtools::install_github("tidyverse/ggplot2")

CONCLUSION

It is very crucial to adequately set up the package for testing. Some of the errors have been discussed and solutions have also been proposed. It is advisable to always follow the documentation for latest the updates.

I did a minimal testing of touchstone by using a simple for loop. Therefore, I created a simple R package repository on GitHub called TouchTest . Inside the script.R file, there exists a function touchstone::benchmark_run which helps to benchmark any function call from the package.

To make it simple to test the minimal functionality of touchstone, I modified the function and wrote the for loop inside a random_test function.

Test

On the main branch, inside the script.R file, we basically loop from 1 to 10 printing the current number.

# benchmark a function call from your package (two calls per branch)
touchstone::benchmark_run(
  # expr_before_benchmark = source("dir/data.R"), #<-- TODO OTPIONAL setup before benchmark
  random_test = function() {
    print("Hello, world!")
    # Loop from 1 to 10
    for (i in 1:10) {
      print(i)
    }
  }, #<- TODO put the call you want to benchmark here
  n = 2
)

However, onn the devel branch, there is a minor change in the for loop. We print from 1 to 100 instead.

# benchmark a function call from your package (two calls per branch)
touchstone::benchmark_run(
  # expr_before_benchmark = source("dir/data.R"), #<-- TODO OTPIONAL setup before benchmark
  random_test = function() {
    print("Hello, world!")
    # Loop from 1 to 100
    for (i in 1:100) {
      print(i)
    }
  }, #<- TODO put the call you want to benchmark here
  n = 2
)

Results

It is quite obvious that the 1 to 100 loop will take longer time than the 1 to 10 loop. Inside the touchstone directory, plots and pr-comment folders are created naturally containing the image of the plot and the PR comment text respectively. Below shows the results as expected.

The graph above demonstrates the visualizations of the distribution of the timings for both branches. It can be inferred from the graph above that the density of the devel branch is greater than that of the main branch. This is due to the fact that the devel branch entails looping 100 times whereas the main branch entails looping just 10 times.

Benchmark result is posted as a comment in the Pull Request. An example is shown below :

This is how benchmark results would change (along with a 95% confidence interval in relative change) if b61b5addb59ac69eca91ee97caf7e5f5ef986aed is merged into main:

•   :ballot_box_with_check:random_test: 1.24µs -> 1.3µs [-5.9%, +15.17%] Further explanation regarding interpretation and methodology can be found in the documentation.

The results can be interpreted as an increase in time by approximately 15.17% from the devel branch when compared to the main branch. To avoid any uncertainties, under the hood, the code is run n times based on the confidence interval. The confidence interval shows the certainties about any estimated differences. Hence, a 0.95 confidence interval tells that if the benchmarking experiment were to be repeated 100 times, the estimated change in speed would be in the interval 95 out of 100 times. The graph generally gets noisy when there is no significant speed change.

Therefore, merging a Pull Request of such nature will mean that the code will be a little bit slower. However, checking from the speed decrease implies that it will hardly be noticed practically. There is always a trade-off between time and functionality especially when a new feature is introduced in a code. This is the reason why the Confidence Interval is very crucial with regards to code benchmarking.

Conclusion

Even though, the touchstone package is an experimental R package, it aims on providing the means to perform accurate benchmarking tests for package developers. Due to the fact that Pull requests are natural units for measuring speed, connecting benchmarking to pull requests makes it easier and faster to use.

Touchstone is an R tool which provides accurate benchmarking features for testing other R packages. It provides continuous benchmarking with reliable relative measurement and uncertainty reporting. It is enriched with features which are very useful especially with respect to merging a Pull Request into target branch. Therefore, it is integrated with GitHub Continuous Integration(CI) which helps to automate the whole process.

2. Concept

For your PR branch and the target branch, touchstone will :

• build two versions of the package in isolated libraries.
• measure the accurate relative differences between the branches. The code under experimentation will be run several times in a random order.
• comment the results of the benchmarking on the Pull Request.
• create visualizations to demonstrate the distribution of the timings for both branches.

3. Installation

Installation can be done in two ways :

• CRAN :

  install.packages("touchstone")
  

• GitHub :

   devtools::install_github("lorenzwalthert/touchstone")
  

4. Package Usage

Initialize touchstone by running

touchstone::use_touchstone()

The above line of code will :

• Create a directory known as the touchstone directory in the root of the repository with config.json and script.R. The config.json contains configurations that define how to run your benchmark. The script.R is the script that runs the benchmark. The header.R contains the default PR header whilst the footer.R containing the default PR comment footer.

• Populate the file touchstone-receive.yaml in .github/workflows/. and touchstone-comment.yaml in .github/workflows/ respectively.

• Add the touchstone directory to .Rbuildignore file in the root directory of your repository.

Write the workflow files you need to invoke touchstone in new pull requests into .github/workflows/.

Modify the touchstone script touchstone/script.R to run different benchmarks.

5. Understanding the script

There are three default functions inside the script.R file.

(i) branch_install

touchstone::branch_install()

branch_install installs each branch in a separate library for isolation.

Usage

branch_install(
  branches = c(branch_get_or_fail("GITHUB_BASE_REF"),
    branch_get_or_fail("GITHUB_HEAD_REF")),
  path_pkg = ".",
  install_dependencies = FALSE
)

The arguments :
branches are names of the branches in character vector
path_pkg represents the path to the package which has a default value of "."
install_dependencies is a boolean which enables dependencies to be installed, has a default value of FALSE

(ii) benchmark_run

touchstone::benchmark_run()

benchmark_run runs benchmarks for git branches using function calls from your package.

Usage

benchmark_run(
  expr_before_benchmark = { },
  ...,
  branches = c(branch_get_or_fail("GITHUB_BASE_REF"),
    branch_get_or_fail("GITHUB_HEAD_REF")),
  n = 100,
  path_pkg = "."
)

The arguments :
expr_before_benchmark allows an expression to be executed just before benchmark is run
... is the named expression of length one with code to benchmark
branches are names of the branches in character vector
n is the number of times benchmarks should be run for each of the branches.
path_pkg represents the path to the package

(ii) benchmark_analyze

touchstone::benchmark_analyze()

benchmark_analyze creates artifacts used downstream in the GitHub Action to turn raw benchmark results into text and figures.

Usage

benchmark_analyze(
  branches = c(branch_get_or_fail("GITHUB_BASE_REF"),
    branch_get_or_fail("GITHUB_HEAD_REF")),
  names = NULL,
  ci = 0.95
)

The arguments :
branches are the names of the branches in character vector under consideration
names are the names of the branches which is actually used for the analysis.
ci represents the confidence level which defaults to 0.95 out of 1

6. Running the script

On the R Interactive console, run

touchstone::run_script()

Usage

run_script(
  path = "touchstone/script.R",
  branch = branch_get_or_fail("GITHUB_HEAD_REF")
)

The arguments :
path is the path to the script to run
branch is the name of the branch corresponding to the library

After committing and pushing the workflow files to default branch, Github CI will run the benchmarks on every pull request and on each commit while that pull request is open.

Rperform is a package that allows R developers to track quantitative performance metrics of their code. It helps to provide changes in a package’s performance metrics related to runtime and memory. It is used to access performance metrics over different git versions and across git branches. It can also be integrated with Travis-CI for performance testing during Travis builds by making changes to the repo’s .travis.yml file. It can prove to be particularly useful while measuring the possible changes which can be introduced by a pull request (PR).

The project was initiated as a part of the Google Summer of Code 2015 program and was accepted into the Google Summer of Code 2016 program.

Installation

• You can install the package from github using :

library(devtools)
install_github("analyticalmonk/Rperform")

or

devtools::install_github("analyticalmonk/Rperform")

Basic examples

NOTE: The Rperform package requires you to set the current directory to the concerned git repository before using the functions.

> setwd(dir = "Path/to/repo")

• The following example illustrates the use of the Rperform::plot_metrics() function on the git repository of the package stringr.

> setwd("./stringr")
> library(Rperform)
> plot_metrics(test_path = "tests/testthat/test-join.r", metric = "time", num_commits = 10, save_data = FALSE, save_plots = FALSE)

• The following example illustrates the use of the Rperform::plot_branchmetrics() function on the git repository of the package stringr.

> setwd("./stringr")
> library(Rperform)
> plot_branchmetrics(test_path = "tests/testthat/test-interp.r", metric = "memory", branch1 = "rperform_test", branch2 = "master", save_data = F, save_plots = F)

• The following example illustrates the use of the Rperform::time_compare() and Rperform::mem_compare() functions on the git repository of the package stringr.

> setwd("./stringr")
> library(Rperform)
> time_compare(test_path = "./tests/testthat/test-dup.r", num_commits = 2)
                          test_name metric_name status  metric_val         msg_val           date_time
1           basic duplication works        time   pass 0.001797014 Can now use CRA 2015-01-08 14:09:43
2           basic duplication works        time   pass 0.001539050 Can now use CRA 2015-01-08 14:09:43
3           basic duplication works        time   pass 0.001545034 Can now use CRA 2015-01-08 14:09:43
4  0 duplicates equals empty string        time   pass 0.001019430 Can now use CRA 2015-01-08 14:09:43
5  0 duplicates equals empty string        time   pass 0.000784386 Can now use CRA 2015-01-08 14:09:43
6  0 duplicates equals empty string        time   pass 0.000766279 Can now use CRA 2015-01-08 14:09:43
7                        test-dup.r        time   pass 0.003555478 Can now use CRA 2015-01-08 14:09:43
8                        test-dup.r        time   pass 0.003118946 Can now use CRA 2015-01-08 14:09:43
9                        test-dup.r        time   pass 0.003106017 Can now use CRA 2015-01-08 14:09:43
10          basic duplication works        time   pass 0.001780849 Require latest  2015-01-08 14:03:37
11          basic duplication works        time   pass 0.001545568 Require latest  2015-01-08 14:03:37
12          basic duplication works        time   pass 0.001517300 Require latest  2015-01-08 14:03:37
13 0 duplicates equals empty string        time   pass 0.001028882 Require latest  2015-01-08 14:03:37
14 0 duplicates equals empty string        time   pass 0.000783847 Require latest  2015-01-08 14:03:37
15 0 duplicates equals empty string        time   pass 0.000771520 Require latest  2015-01-08 14:03:37
16                       test-dup.r        time   pass 0.003436051 Require latest  2015-01-08 14:03:37
17                       test-dup.r        time   pass 0.003130397 Require latest  2015-01-08 14:03:37
18                       test-dup.r        time   pass 0.003082713 Require latest  2015-01-08 14:03:37
> 

> Rperform::mem_compare(test_path = "./tests/testthat/test-join.r", num_commits = 1)
             test_name metric_name status metric_val         msg_val           date_time
11.1  basic case works     max_mem_mb   pass      0.040 Can now use CRA 2015-01-08 14:09:43
11.2  basic case works     leak_mb      pass      0.040 Can now use CRA 2015-01-08 14:09:43
11.3 NULLs are dropped     max_mem_mb   pass      0.044 Can now use CRA 2015-01-08 14:09:43
11.4 NULLs are dropped     leak_mb      pass      0.044 Can now use CRA 2015-01-08 14:09:43
11.5       test-join.r     max_mem_mb   pass      0.148 Can now use CRA 2015-01-08 14:09:43
11.6       test-join.r     leak_mb      pass      0.148 Can now use CRA 2015-01-08 14:09:43
12.1  basic case works     max_mem_mb   pass      0.040 Can now use CRA 2015-01-08 14:09:43
12.2  basic case works     leak_mb      pass      0.040 Can now use CRA 2015-01-08 14:09:43
12.3 NULLs are dropped     max_mem_mb   pass      0.044 Can now use CRA 2015-01-08 14:09:43
12.4 NULLs are dropped     leak_mb      pass      0.044 Can now use CRA 2015-01-08 14:09:43
12.5       test-join.r     max_mem_mb   pass      0.144 Can now use CRA 2015-01-08 14:09:43
12.6       test-join.r     leak_mb      pass      0.144 Can now use CRA 2015-01-08 14:09:43
13.1  basic case works     max_mem_mb   pass      0.040 Can now use CRA 2015-01-08 14:09:43
13.2  basic case works     leak_mb      pass      0.040 Can now use CRA 2015-01-08 14:09:43
13.3 NULLs are dropped     max_mem_mb   pass      0.048 Can now use CRA 2015-01-08 14:09:43
13.4 NULLs are dropped     leak_mb      pass      0.048 Can now use CRA 2015-01-08 14:09:43
13.5       test-join.r     max_mem_mb   pass      0.148 Can now use CRA 2015-01-08 14:09:43
13.6       test-join.r     leak_mb      pass      0.148 Can now use CRA 2015-01-08 14:09:43