Following Statistical Analysis Plan Guidelines

Jeremy Albright

Posted on
Statistical Analysis Plans Best Practices for Data Analysis SAP
xkcd study comic

What is a Statistical Analysis Plan?

A statistical analysis plan (SAP) is a document that is authored prior to the start of a clinical or observational study that presents significant detail about how data will be coded and analyzed. It serves three essential roles. First, it furnishes transparency concerning how the analysis will proceed by specifying in advance the methodology that will be applied. Second, it provides clear communication to the statistician involved in the study for how to proceed. Finally, it facilitates replication so that a future research team can follow the same steps to confirm the results on the same or a new sample.

An SAP is not the same as a protocol, although there is a great deal of overlap. A study protocol will provide a description of the statistical methodology, but the detail is far less relative to the SAP. Protocols target clinical audiences, whereas an SAP targets a statistical audience. The SAP can be thought of as a conversation between two or more statisticians.

A company may choose to work with a contract statistician to develop an SAP given the level of detail required. Doing so can save time and resources both in the short term, by working with somebody knowledgeable about the required content and appropriate phrasing, as well as in the long term, by anticipating problems that may emerge during the course of the study and having a transparent, pre-registered description of how they will be handled. The best statisticians to work with will be knowledgeable about relevant methodological and reporting standards, such as those published by the Clinical and Laboratory Standards Institute (CLSI), the International Conference on Harmonization (ICH), or the Consolidated Standards of Reporting Trials (CONSORT) group, among others. In addition, the statistical consultant should also be comfortable identifying the strengths and weaknesses of different study designs, understand different randomization schemes and their consequences for inference, the impact of different statistical methodologies on sample size requirements, and how to minimize false positives when multiple tests are performed.

What is the Importance of a Statistical Analysis Plan?

An SAP should always be completed before the data have been unblinded for the statistician. This enhances the credibility of the study by demonstrating, transparently, a commitment to a specific set of analyses that have not been influenced by the data. Altering a planned statistical analysis after seeing the data increases the risk of bias, because researchers may be tempted to make decisions that capitalize on the idiosyncrasies of the sample at hand. It is well known that even seemingly benign decisions, like changing how a variable is coded, can inflate the probability of Type I errors (incorrectly rejecting the null hypothesis). Consequently, the SAP needs to summarize all steps, including how variables will be coded and how violations of statistical assumptions will be handled, clearly and up front before looking at the data. This also forces communication between the statistician and all stakeholders to verify that the goals of the study are understood and that the analysis will reflect the study objectives. A protocol typically will not have this same level of granularity and hence is insufficient for other researchers to replicate results. Replication requires knowing every single step performed from the time the data are unblinded to completion of the study report.

Until very recently, there has been little guidance for what should be contained in an SAP. A 2017 JAMA article provides a checklist that SAP authors can follow to help ensure that all required elements are included. Even with this checklist, it is essential to work with a knowledgeable statistician who is familiar with ambiguities in data analysis (such as proper variable coding, reliability and validity of outcome measures), appropriate designs, and sample size analysis so that the correct methodology is described using precise language. This further ensures that the three goals of SAPs (transparency, communication, and replicability) are met in a timely fashion.

Opting to work with a contract statistician has several benefits. Hiring a full time statistician within the company can be expensive, and it may be uncertain what role she will play once the study is completed. Statisticians from Contract Research Organizations (CROs) are experienced and administratively less burdensome. Reading through lengthy documents like the FDA’s Guidance for Industry: Statistical Principles for Clinical Trials can be daunting for somebody not well versed in statistical jargon. The same applies for reading guidelines like CLSI, CONSORT, STROBE, SPIRIT, and others.

When one does consider the different guidelines, one notes a common set of considerations that are emphasized throughout. These are the importance of reducing bias, or incorrect estimates of treatment effects, that can be due to poor research design, improper randomization, insufficient blinding, and incomplete adherence to treatment by participants. Any post hoc adjustments to deal with these biases may introduce their own new biases. For example, poor adherence may lead to the decision to analyze only those who adhered. Yet by focusing on this per protocol sample may lead to new biases if adherence is related to treatment assignment. Say treated participants drop out due to a drug’s limited effectiveness and uncomfortable side effects of the drug, while all controls remain in the study. The estimated treatment effect may be biased upwards because only treated participants who benefit and do not experience side effects remain in the study. It is consequently necessary to consider the sensitivity of results when looking at the Intention-to-Treat (ITT) sample, the per protocol sample, or results that use imputation to deal with attrition.

Another common guideline is to carefully choose the primary outcome to minimize multiplicity effects. These occur when there are multiple statistical tests needed to assess the primary outcome, which increases the likelihood of finding a false positive. Methods to deal with this range from a more focused primary outcome definition during the study design to statistical adjustments at the time of analysis. Often, researchers using statistical adjustments default to the Bonferroni method, which is easy to apply but often too conservative (i.e. a true result is not found to be statistically significant). Your statistician should be able to advise on other methods, such False Discovery Rates (FDR) or Holm’s adjustment to the Bonferroni method.

p-value meme

A third point of emphasis in all guidelines is appropriate sample size planning. All studies face the Goldilocks Dilemma of having too few subjects to make statistically meaningful inferences versus having too many subjects that add to the cost and time of the study. Power analysis makes an assumption about a clinically meaningful result and finds the sample size necessary to keep Type-II errors (failing to find a truly significant result) at a minimum. However, identifying this clinically meaningful difference can be a challenge, and a range of options may be considered. Things can become even more complicated for equivalence and non-inferiority studies, where it is necessary to identify a margin that indicates acceptable similarity between treatments. Finally, different statistical tests have different power requirements. As a simple example, using a nonparametric test when parametric assumptions are valid can lead to an unnecessarily inflated, and more costly, sample size. Familiarity with different tests is also important for making sure the study does not cost more than it needs to.

These of course are not the only elements to consider. The guidelines from different organizations tend to be very comprehensive, with each item in the checklist having its own nuance. Working with a statistician, whether in-house or contracted, can expedite the process of getting the SAP into an acceptable format.

Methods Consultants Approach to Statistical Analysis Plans

When we work with clients, we begin with a discussion of the goals of the study to identify proper endpoints (primary versus secondary outcomes) as well as the optimal design to test the client’s claims. Typically, we offer advising on the statistical component of the protocol first so that the broad context of the study and its methodology is documented. Having a protocol also facilitates a division of labor between the subject matter experts, who author components of the SAP relevant to the study motivation, and the statistician, who takes care of the statistical matters related to coding, modeling, and sensitivity analyses. We also discuss with the client a preference for software. We prefer using R because of its tools for version controlling statistical analysis, but we also have facility with SAS, SPSS, Stata, and Python. In some instances, we may use a combination of software options. For example, despite the myriad packages for fitting mixed models in R, none can yet compete with SAS’s comprehensive and intuitive proc glimmix.

All steps of the SAP development are strictly version controlled using private git archives. This allows us to easily compare drafts of files to identify where changes were made and easily develop version history tables if SAP amendments are required.

All analyses are identified up front, including table shells and mock figures. These make it easy for a different statistician to pick up the SAP and know not only what to do but how to present the results. We often generate pseudo-data to mimic the variables that will be collected in order to produce realistic looking figures as templates. The code for generating the figures is then saved so that it does not have to be reconstructed from scratch. The following is an example:

Sample Kaplan-Meier Plot Based on Simulated Data

Figure 1: Sample Kaplan-Meier Plot Based on Simulated Data

When it comes time to perform the analysis, our code is written to create the output in tabular form matching the table shells. In this way, the final deliverable to clients will include all syntax necessary to replicate the tables as they appear in the final report. Every step is documented, version controlled, and made available to the client for full transparency.

All studies, even those not meant for FDA review, can benefit from a careful development of an SAP. If your organization needs assistance, contact us to learn how Methods Consultants can help with your Statistical Analysis Plan.