Plugging the gaps: Lessons learned from implementing the mmrm R package

Kevin Kunzmann on behalf of the ASA BIOP SWE WG

Boehringer Ingelheim

June 13, 2023

Package Authors:

  • Brian Matthew Lang (MSD)
  • Christian Stock (Boehringer-Ingelheim)
  • Craig Gower-Page (Roche)
  • Dan James (AstraZeneca)
  • Daniel Sabanes Bove (Roche, lead)
  • Doug Kelkhoff (Roche)
  • Julia Dedic (Roche)
  • Kevin Kunzmann (Boehringer-Ingelheim)
  • Liming Li (Roche)
  • Ya Wang (Gilead)

Acknowledgments & thanks to:

Ben Bolker (McMaster University), Davide Garolini (Roche), Dinakar Kulkarni (Roche), Gonzalo Duran Pacheco (Roche),
ASA BIOP Software Engineering working group (SWE WG)

Agenda

  • Overview of Open Source in Biostatistics
  • Case Study: {mmrm} Package
    • Mixed Models for Repeated Measures - Why bother?
    • Why this is not “yet another package”
  • Lessons Learned
  • Next Steps

Open Source in Biostatistics

  • Open-source software increasing popularity in Biostatistics
  • Rapid uptake of novel statistical methods
  • Unprecedented opportunities for collaboration
  • Transparency of methods and implementation
  • Variability in software quality
  • No statistical quality assurance on open-source extension package repositories, e.g. CRAN
  • No industry standard for assessing quality of R packages
  • Reliable software for core statistical analysis is paramount

The ASA BIOP SWE Working Group

  • Idea: form industry working group to identify and fill gaps in R package landscape with reliable implementations of crucial methods
  • ASA Biopharmaceutical section software engineering working group (ASA BIOP SWE WG)
  • Foster good coding practices for easier maintenance, review. and transparency:
    • Version control for traceability and structured change management
    • Self-sufficient high-quality documentation
    • Code style standards for easier review
    • Extensive unit testing and CI/CD for test-driven development
    • Reproducible outputs, especially for stochastic algorithms

Why do we need a package for MMRMs?

  • MMRM popular for longitudinal continuous outcomes in randomized clinical trials
  • Think about long term maintenance and responsibility!
  • Try to improve existing package
    • extend glmmTMB with Satterthwaite adjusted degrees of freedom
    • not feasible: glmmTMB only random effects representation, no real unstructured model
  • No tailored R package with sufficient capabilities/reliability
  • lme4 + lmerTest: fails on large data sets (slow, convergence issues)
  • nlme: no Satterthwaite adjusted degrees of freedom, convergence issues, only approximate emmeans

Case Study: The {mmrm} Package

  • Linear model for dependent observations within independent subjects
  • Multiple covariance structures for the dependent observations
  • REML or ML estimation, using multiple optimizers if needed
  • emmeans interface for least square means
  • tidymodels for easy model fitting
  • Satterthwaite and Kenward-Roger adjustments for degrees of freedom
  • Designed for needs of pharma applications from ground up
  • Ongoing maintenance and support from the pharma industry
    • 5 companies involved in the development
    • ambition to become standard package within the field

How we develop {mmrm}

  • Created initial prototype quickly, open-sourced on github.com/openpharma/mmrm
  • In parallel, assembled multiple other companies with shared interest in reliable MMRM for R
  • \(\leadsto\) Start of Software Engineering working group (SWE WG), with >35 members from >25 organizations
  • Focus on core issues and aim for frequent releases of new features instead of waiting for perfect state
  • Do not compromise quality on the way, rather less done well
  • Regular online developer calls to align next steps

Getting started with {mmrm}

  • {mmrm} is on CRAN - use this as a starting point:
install.packages("mmrm")
library(mmrm)
fit <- mmrm(
  formula = FEV1 ~ RACE + SEX + ARMCD * AVISIT + us(AVISIT | USUBJID),
  data = fev_data
)
summary(fit)
library(emmeans)
emmeans(fit, ~ ARMCD | AVISIT)

Comparing {mmrm} and SAS

Comparison to de-facto gold standard in SAS with PROC MIXED:

{mmrm}

  • Documentation: Available online
  • Testing: Transparent + coverage
  • Covariance structures: 10 non-spatial covariance structures
  • Focus on population marginal models

PROC MIXED

  • Documentation: Available online
  • Testing: Hidden
  • Covariance structures: 23 non-spatial covariance structures
  • Marginal and individual level random effects

Deviation from SAS

  • negligible differences from SAS in (non-representative) set of examples
  • {mmrm} orders of magnitude closer than {glmmTMB} or {nlme}

Speed

  • {mmrm} fast due to C++ code via template model builder (TMB) library
  • {mmrm} uses efficient automatic gradients for the ML/REML objective

Lessons Learned: Best Practices

  • User interface design (function names etc.)
    • Difficult to get right
    • Spend enough time on making it convenient/familiar
  • Testing:
    • Unit tests are essential for preventing regression and assuring quality
    • Use continuous integration to catch errors early
  • Documentation:
    • Lots of work but extremely important
    • Needs to be kept up-to-date
    • Testing alone is not sufficient - also use example & vignettes for building trust with users

Lessons Learned: Collaboration

  • Important to go public as soon as possible - you never know who else might be interested/could help
  • Version control with git: cornerstone of effective collaboration
  • Agree on standards:
    • Consistent and readable code style: transparency and easier collaboration
    • How to contribute: Written (!) contribution guidelines + lowering the entry hurdle using developer calls
  • Leverage network of developers in companies to get feedback from real projects as soon as possible
  • Emphasize importance of non-coding contributions like documentation

Long Term Perspective & Next Steps

  • Software engineering is a critical competence in producing high-quality statistical software
  • A lot of work needs to be done regarding the establishment, dissemination and adoption of best practices for engineering open-source software
  • Improving the way software engineering is done will help improve the efficiency, reliability and innovation within Biostatistics
  • Continue to highlight and support cross-industry collaboration through, e.g.: Phuse, R Consortium, openpharma
  • Publicize existing open source projects, e.g.: admiral, rtables, Bayesian mmrm

Thank you! Questions?