CLAHRCs are all about developing and testing health service innovations so the CLAHRC WM Director was interested in a recent article on sample sizes for pilot studies. Many of these end-points of a pilot study go under the heading ‘detecting unforeseen problems’. The authors suggest setting the minimum probability of the problem that should be detected and the alpha value depending on the seriousness of the problem for the future trial. To be honest, the CLAHRC WM Director was not highly impressed by the approach. “If the problem occurs more frequently than X, then I want to know a 95% probability of observing at least one instance”, does not make much sense to the context of study design. Why not just calculate the 95% (or 80%) confidence intervals (CIs) of probabilities at given levels. Investigators know that problems can occur, so detailing an instance of the problem is not interesting. It is not like detailing a case of election fraud in a sample of ballot stations. In such a context, the authors’ method, which is nearly equivalent to ‘the rule of three’, would be more useful. The rule of three, to remind you, calculates the 95% CI on zero instances of an event, given n observations by the equation 3/n. So if there were no cases of fraud in 60 polling stations the 95% CI is 3/60 = 0.05. Do others share the CLAHRC WM Director’s concern regarding the underlying logic behind this method in a pilot study?
— Richard Lilford, CLAHRC WM Director
- Veichtbauer W, Smits L, Kotz D, et al. A simple formula for the calculation of sample size in pilot studies. J Clin Epidemiol. 2015; 68: 1375-9.