The analytical choice that decides how precise a meta-analysis is

Two analysts receive exactly the same studies evaluating the effectiveness of a medicine.

They both run a random-effects meta-analysis to pool the study effects.

They report the same pooled effect but different 95% confidence intervals. One interval excludes no effect, suggesting the medicine works. The other crosses the line of no effect and is too wide to conclude whether the medicine works.

That is not an error. The confidence interval is partly an analytical choice.

There are different ways to calculate a confidence interval in a meta-analysis. Two common approaches are the conventional Wald method and the Hartung-Knapp-Sidik-Jonkman method, or HKSJ. Wald uses a standard formula found in many meta-analysis programs. HKSJ adjusts that calculation to allow for more uncertainty when the number of studies is limited.[1,2]

Wang and colleagues reanalysed 88 published meta-analyses of medicines and medical devices using different methods. Changing from a conventional approach to HKSJ changed whether the evidence was judged too imprecise to support a firm conclusion in up to 19% of analyses.[3]

The difference was greatest when the meta-analysis contained fewer studies or more heterogeneity.[3]

A 95% confidence interval around a medicine’s effect is not one calculation

A confidence interval shows the uncertainty around the estimated effect of a medicine. A narrower interval suggests greater precision. A wider interval suggests that a broader range of treatment effects remains plausible.

The 95% does not mean there is a 95% probability that the true effect lies inside one particular interval. It describes how the method performs when used repeatedly. A well-performing method should produce intervals that contain the true average effect in about 95 out of 100 repeated analyses.[1,2]

This is called coverage.

If the true effect is captured less often, the method has undercoverage. Its intervals are too confident and it produces too many false-positive conclusions that a medicine works when it does not. But simply making every interval wider is not the answer. An interval can also be so cautious that it becomes unhelpful.[2,4]

The aim is reliable coverage without making the interval wider than it needs to be.

Random-effects meta-analysis assumes a medicine’s effects are related but not identical

A random-effects meta-analysis allows the studies of a medicine to be estimating related but not identical effects.

The medicine may have a larger effect in one study and a smaller effect in another because the studies recruit different patients, use different doses, provide different standard care alongside the medicine or measure outcomes over different periods.

The standard model assumes that the true intervention effects underlying the included studies follow a normal distribution around an average effect. Some effects will be larger and some smaller. The model estimates the average treatment effect and the variation between the studies.[1,5]

That variation in the medicine’s true effect is called between-study heterogeneity. It is commonly represented by tau squared.

The value of tau squared is not known. It has to be estimated from the included studies. When only a few studies are available, that estimate can be highly uncertain.[1,6]

This is where the confidence interval becomes vulnerable.

Estimating heterogeneity and calculating precision are separate choices

A random-effects meta-analysis involves two connected but separate decisions:

1. How should the variation between the medicine’s true effects across the studies be estimated?

2. How should the confidence interval around the pooled treatment effect be calculated?

Methods such as DerSimonian-Laird, restricted maximum likelihood, Paule-Mandel and Sidik-Jonkman estimate heterogeneity. Wald, HKSJ and profile likelihood are approaches to calculating the confidence interval.[1,2]

The two choices interact, but they are not the same. Selecting restricted maximum likelihood does not decide whether the confidence interval should use Wald or HKSJ.

Software can quietly make both decisions through its default settings.

The conventional method can make a medicine’s effect look too certain

The conventional Wald confidence interval inserts the estimated heterogeneity into a standard formula. It does not fully carry the uncertainty in that estimate into the confidence interval.[2,6-8]

When many consistent studies of a medicine contribute, the impact may be small. When there are few studies, substantial heterogeneity or one study that is much more precise than the others, the interval can be too narrow.[2,7,8]

In simulations by IntHout and colleagues, a method intended to produce false-positive findings 5% of the time exceeded 30% in some settings when the conventional DerSimonian-Laird and Wald approach was used.[9]

The practical impact was also clear. Across 689 Cochrane meta-analyses of healthcare interventions, 25.1% of findings that were statistically significant using the conventional approach were no longer significant when HKSJ was used.[9]

The studies had not changed. The method used to describe their uncertainty had.

HKSJ often improves coverage, but it is not a universal fix

HKSJ uses a different estimate of uncertainty around the pooled effect. It also uses a t distribution, which recognises that there may be only a small number of studies.[10,11]

It often produces confidence intervals with coverage closer to the intended 95%, particularly when heterogeneity is present and the studies contribute similar amounts of information.[1,8-10]

But HKSJ is not simply “the conservative method”.

With only two or three studies, its interval can become extremely wide. When the observed heterogeneity is very low, HKSJ can sometimes produce a narrower interval than the conventional method.[1,4,12]

Modified HKSJ was developed to prevent that counterintuitive narrowing. It can improve false-positive control when study precision differs greatly, but it may also produce an excessively wide interval, particularly when only two studies are available.[4,6,8]

Two studies remain a genuinely difficult evidence base. Conventional methods can look too certain. HKSJ and modified HKSJ can become uninformative. Bayesian and newer small-sample methods may help, but their performance depends on assumptions that must be justified.[1,2,13-15]

Figure 1 shows why the choice matters. The pooled odds ratio is identical under Wald and HKSJ in each example, but the confidence intervals differ. In Example 2, HKSJ produces an extremely wide interval with only two studies. In Example 3, the Wald interval excludes no effect while the HKSJ interval crosses the line of no effect.[1]

The solution is not to select whichever method gives the neatest answer.

Wider does not automatically mean better

A confidence-interval method should be judged by more than its width.

Does it capture the true effect close to 95% of the time? Does it control false-positive conclusions? Is the interval unnecessarily wide? Does the method remain stable when there are few studies or large differences in study precision?[2,4]

A narrow interval with poor coverage can make the evidence for a medicine look more certain than it is. A huge interval may be statistically cautious but useless for deciding whether the medicine provides a credible benefit.

No single method performs best in every setting.[2]

The medicine’s evidence base should decide the method

The table below compares methods used to calculate confidence intervals when pooling treatment effects. It does not compare methods for estimating heterogeneity. That is a separate analytical choice.

Confidence-interval approach	Most defensible use
Conventional Wald interval [5]	Most defensible when many studies of the medicine contribute, heterogeneity is limited and study precision is not severely unbalanced. It should not be the only analysis when the evidence base is sparse or heterogeneous.[1,2,8]
HKSJ [10]	Often a strong primary or sensitivity analysis when more than two studies contribute, particularly when heterogeneity is present and study precision is reasonably similar.[1,2,8,14]
Modified HKSJ [4]	A useful sensitivity analysis when ordinary HKSJ becomes narrower than Wald or when a few studies have very different precision. It can be excessively cautious with very few studies.[2,4,6]
Profile likelihood [7]	A specialist alternative that incorporates more uncertainty in heterogeneity and permits asymmetric intervals. Check convergence, especially with small evidence bases.[2,3,7]
Higher-order likelihood methods [16]	A specialist sensitivity analysis for moderate numbers of studies. Evidence is less reassuring when five or fewer studies contribute.[2,16]
Kenward-Roger [2]	A specialist small-sample sensitivity analysis. It can become overly wide when heterogeneity is low or study precision differs greatly.[2,8]
Bayesian credible interval [14]	Particularly useful with two or very few studies when the prior assumptions about heterogeneity are clinically justified, prespecified and tested in sensitivity analyses.[13-15]
Bootstrap or permutation interval [14]	Best reserved for specialist sensitivity analyses. These methods add complexity and do not have a consistent advantage across settings.[2,14]

The number of studies is not the only consideration. Outcome type, heterogeneity and differences in study precision also matter. A drug evidence base containing one large phase III trial and several small phase II studies may behave differently from one in which every study contributes a similar amount of information.[2,4,8,9]

A confidence interval does not predict the medicine’s effect in the next setting

A confidence interval describes uncertainty around the average effect across the included studies.

It does not show the range of treatment effects that might occur when the medicine is used in a new hospital, patient population or healthcare system. That is the role of a prediction interval.[1,8]

When heterogeneity is substantial, clinical development teams, HTA bodies and payers may need both. The confidence interval describes uncertainty around the average treatment effect. The prediction interval shows how far the medicine’s effect might vary in another setting.

Prediction intervals also become unreliable when there are few studies or large differences in study precision. They are not an escape from the same small-sample problem.[8]

Prespecify the method before seeing which answer favours the medicine

The primary method should be selected before anyone sees which interval is most favourable.

When a medicine’s evidence base is small, heterogeneous or unbalanced, the analysis plan should include sensitivity analyses using other defensible approaches. If those methods change the conclusion about precision, clinical importance or statistical significance, report that difference.[1-3]

Wang and colleagues describe selective reporting of the most desirable result as a form of reporting bias.[3]

At Evidax, our methodologists treat the confidence-interval method as part of the evidence generation plan, not an unexamined software setting. The method should match the structure of the medicine’s evidence base and the decision the analysis needs to support.

The confidence interval is part of the result. How it was calculated should never be an afterthought.

Take home

• A 95% confidence interval around a pooled treatment effect is not one fixed calculation.

• Random-effects meta-analysis assumes that studies of a medicine estimate related but not identical effects, usually distributed normally around an average treatment effect.

• Estimating between-study heterogeneity and calculating the confidence interval are separate analytical choices.

• Conventional Wald intervals can make the evidence for a medicine look too precise when there are few studies, meaningful heterogeneity or large differences in study precision.

• HKSJ often improves coverage, but it can become extremely wide or occasionally narrower than the conventional interval.

• Select and prespecify the method based on the evidence structure. If defensible methods change the conclusion, show that sensitivity rather than hiding it.

References

1. Deeks JJ, Higgins JPT, Altman DG, McKenzie JE, Veroniki AA. Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions. Version 6.5. Cochrane; 2024. https://training.cochrane.org/handbook/current/chapter-10

2. Veroniki AA, Jackson D, Bender R, et al. Methods to calculate uncertainty in the estimated overall effect size from a random-effects meta-analysis. Research Synthesis Methods. 2019;10(1):23-43. https://doi.org/10.1002/jrsm.1319

3. Wang Z, Alzuabi MA, Morgan RL, et al. Different meta-analysis methods can change judgements about imprecision of effect estimates: a meta-epidemiological study. BMJ Evidence-Based Medicine. 2023;28(2):126-132. https://doi.org/10.1136/bmjebm-2022-112053

4. Röver C, Knapp G, Friede T. Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies. BMC Medical Research Methodology. 2015;15:99. https://doi.org/10.1186/s12874-015-0091-1

5. DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials. 1986;7(3):177-188. https://doi.org/10.1016/0197-2456(86)90046-2

6. Jackson D, Law M, Rücker G, Schwarzer G. The Hartung-Knapp modification for random-effects meta-analysis: a useful refinement but are there any residual concerns? Statistics in Medicine. 2017;36(25):3923-3934. https://doi.org/10.1002/sim.7411

7. Hardy RJ, Thompson SG. A likelihood approach to meta-analysis with random effects. Statistics in Medicine. 1996;15(6):619-629. https://doi.org/10.1002/(SICI)1097-0258(19960330)15:6%3C619::AID-SIM188%3E3.0.CO;2-A

8. Partlett C, Riley RD. Random effects meta-analysis: coverage performance of 95% confidence and prediction intervals following REML estimation. Statistics in Medicine. 2017;36(2):301-317. https://doi.org/10.1002/sim.7140

9. IntHout J, Ioannidis JPA, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Medical Research Methodology. 2014;14:25. https://doi.org/10.1186/1471-2288-14-25

10. Hartung J, Knapp G. On tests of the overall treatment effect in meta-analysis with normally distributed responses. Statistics in Medicine. 2001;20(12):1771-1782. https://doi.org/10.1002/sim.791

11. Sidik K, Jonkman JN. A simple confidence interval for meta-analysis. Statistics in Medicine. 2002;21(21):3153-3159. https://doi.org/10.1002/sim.1262

12. Wiksten A, Rücker G, Schwarzer G. Hartung-Knapp method is not always conservative compared with fixed-effect meta-analysis. Statistics in Medicine. 2016;35(15):2503-2515. https://doi.org/10.1002/sim.6879

13. Seide SE, Röver C, Friede T. Likelihood-based random-effects meta-analysis with few studies: empirical and simulation studies. BMC Medical Research Methodology. 2019;19:16. https://doi.org/10.1186/s12874-018-0618-3

14. Weber F, Knapp G, Glass Ä, Kundt G, Ickstadt K. Interval estimation of the overall treatment effect in random-effects meta-analyses: recommendations from a simulation study comparing frequentist, Bayesian, and bootstrap methods. Research Synthesis Methods. 2021;12(3):291-315. https://doi.org/10.1002/jrsm.1471

15. Duan Y, Mathew T, Alemayehu D, Cheng G. Novel approaches for random-effects meta-analysis of a small number of studies under normality. Research Synthesis Methods. 2025;16:922-936. https://doi.org/10.1017/rsm.2025.10022

16. Noma H. Confidence intervals for a random-effects meta-analysis based on Bartlett-type corrections. Statistics in Medicine. 2011;30(28):3304-3312. https://doi.org/10.1002/sim.4350