Introducing openstatsware and the R Package {mmrm}

ASA-SSC Mini-Symposium on Statistical Computing in Action 2023

Jonathan Sidi on behalf of the ASA Software Engineering Working Group



  • Official working group of the American Statistical Association (ASA) Biopharmaceutical Section
    • Formed on 19 August 2022
    • Cross-industry collaboration (56 members from 35 organizations)
    • Full name: Software Engineering Working Group
    • Short name: openstatsware
    • Homepage:

Working Group Objectives

  • Primary
    • Engineer R packages that implement important statistical methods
      • to fill in gaps in the open-source statistical software landscape
      • focusing on what is needed for biopharmaceutical applications
  • Secondary
    • Develop and disseminate best practices for engineering high-quality open-source statistical software
      • By actively doing the statistical engineering work together, we align on best practices and can communicate these to others

Workstreams in R Package Development

  • Mixed Models for Repeated Measures (MMRM) 🌎
    • Develop mmrm R package for frequentist inference in MMRM (CRAN)
  • Bayesian MMRM 🌎
    • Develop brms.mmrm R package for Bayesian inference in MMRM (CRAN)
  • Health Technology Assessment 🌎
    • Develop open-source R tools to support HTA dossier submission across various countries
    • Particularly topics with unmet needs in R implementation and/or related to EUnetHTA

R Package mmrm


  • MMRM is a popular choice for analyzing longitudinal continuous outcomes in randomized clinical trials
  • No great R package
    • Initially thought that the MMRM problem was solved by using lme4 with lmerTest, learned that this approach failed on large data sets (slow, did not converge)
    • nlme does not give Satterthwaite adjusted degrees of freedom, has convergence issues, and with emmeans it is only approximate
    • Next we tried to extend glmmTMB to calculate Satterthwaite adjusted degrees of freedom, but it did not work


  • We only want to fit a fixed effects model with a structured covariance matrix for each subject
  • The idea is then to use the Template Model Builder (TMB) directly
    • as it is also underlying glmmTMB
    • but code the exact model we want
  • We do this by implementing the log-likelihood in C++ using the TMB provided libraries
  • Provide an R solution that
    • has fast convergence times
    • generates estimates closest to SAS

Advantages of TMB

  • Fast C++ framework for defining objective functions (Rcpp would have been alternative interface)
  • Automatic differentiation of the log-likelihood as a function of the variance parameters
  • We get the gradient and Hessian exactly and without additional coding
  • This can be used from the R side with the TMB interface and plugged into optimizers

Why It’s Not Just Another Package

  • Ongoing maintenance and support from the pharmaceutical industry
    • 5 companies being involved in the development, on track to become standard package
  • Development using best practices as show case for high quality package
    • Thorough unit and integration tests to ensure accurate results
    • In depth documentation of methods, functionality and comparison to other packages and languages
      • nlme, glmmTMB, lme4, SAS PROC GLMMIX

Highlighted Features of mmrm

  • Covariance structures for the dependent observations:
    • Unstructured, Toeplitz, AR1, compound symmetry, ante-dependence, spatial exponential
    • Allows group specific covariance estimates and weights
  • Hypothesis Testing:
    • emmeans interface for least square means
    • Satterthwaite and Kenward-Roger adjustments
    • Robust sandwich estimator for covariance
  • Integrations and extentions
    • tidymodels builtin parsnip engine and recipes for streamlined model fitting workflows
    • teal, tern, rtables integration for post processing and reporting

Comparison with Other Software

  • We have run comparison analyses with other R packages, namely nlme, glmmTMB and lme4
  • Also compared with SAS PROC GLIMMIX
  • Highlights
    • mmrm has faster convergence time
    • mmrm provides closest results to PROC GLIMMIX
  • Detailed results at the online comparison vignette