First published March 2007, updated November 2021

Key points:

  • The randomised trial is the gold standard method for evaluating the effectiveness of medical interventions.
  • Inclusion/exclusion criteria that are widely used in trials may limit generalisation of the results to patients seen in day-to-day clinical practice.
  • Data from large prospective cohorts may provide additional and complementary evidence about the effectiveness and safety of medical therapies and contribute to the clinical decision making process.
  • There are methodological limitations that should be considered in interpreting the results from such studies.

The randomised clinical trial, an intervention study in which subjects are assigned treatment by random allocation, is regarded as the standard method for evaluating the effectiveness and safety of medical interventions. Results from randomised trials provide clinicians with evidence to guide clinical practice; however, data from trials are not always available and the strict inclusion/ exclusion criteria often used raises questions about how the results can be generally applied to patients — especially the elderly, for whom few trials are focused — seen in day-to-day clinical practice. In the absence of published randomised controlled trials, data from noninterventional (observational studies) may be available and help inform clinical practice.

Randomised controlled clinical trials The first trial with a properly randomised control group was for streptomycin in the treatment of pulmonary tuberculosis1 . Since then the randomised clinical trial has evolved into the standard methodological approach in the evaluation of new treatments. Randomisation ensures differences between those assigned the treatment under investigation and those not, are distributed independently of the treatment and thus any observed differences in outcome are likely to be related to it. The use of blinding or masked  assessment, so that neither the researchers nor the patient is aware of who is taking the treatment, helps reduce bias. Because patient-related factors may potentially influence treatment response (eg, age, gender, racial group and the presence of comorbidity) inclusion and exclusion criteria are widely used to restrict participation. Typically trials will also use strict protocols, both to supervise and monitor patients. The effect of these measures is to maximise participation, and minimise or avoid selection and other types of bias. The consequence, though, may be to limit generalisation of the results to potential recipients seen in daily clinical practice.

Dowd2 studied 120 consecutive female patients seeking healthcare at a tertiary referral centre with a reduced bone mineral density (t-score <-2.0) and/ or with one or more low trauma fractures, to see whether they would have been eligible for participation in one of four large multicentre randomised controlled clinical trials of osteoporosis therapies. Based on the predefined inclusion and exclusion criteria, between four and 25 would have been eligible for entry into the various studies. Thus, even the most liberal trial, with respect to inclusion criteria, would only have included 21 per cent of patients. The main reasons for ineligibility were age, co-morbid conditions, prior treatment with bone active therapy and current medications. The small proportion of eligible patients raises questions about the applicability of the results from such trials to the majority of ordinary (noneligible) patients.

Randomised controlled trials also provide information about adverse effects of medical interventions. However, because they tend to be relatively small in size and duration of follow-up, serious side effects which may take time to develop or are rare may not be detected. Patients in trials tend to be healthier, in part because of exclusion criteria used, making them less likely to experience side effects. Also, adverse effects which occur, particularly in those with pre-existing co-morbidity, may not be seen in trials with the co-morbidity as an exclusion criteria.

Observational data: broadening the evidence

In the absence of trial data, evidence about effectiveness and safety of medical therapies may be available from non-interventional (observational) studies. Methodologically the most powerful design is the prospective cohort study. In this type of study individuals who did or did not receive an intervention are followed over time and outcomes assessed. Examples of cohorts include large scale research studies (eg, the Norfolk Arthritis Register3 ) and those in which data has been collected as part of routine care (eg, the UK General Practice Research Database). The last decade has seen also developments in the use of large databases for claims for payment, particularly in countries where payment is provided by insurance. Such cohorts provide opportunities to study clinical outcomes. Because interventions are typically administered as part of routine clinical practice, the data usually provides a more realistic representation of the impact of treatment in everyday clinical setting. They also have the potential to establish utility in subgroups of patients — for example, defined on the basis of age or ethnic group — and also may provide data concerning the effectiveness of different drug therapies. A case in point are observational studies that have compared antihypertensives and statins for reducing myocardial infarction and the effectiveness of different bisphosphonates in reducing fracture risk4,5,6. Because of their size many of these databases provide opportunities to look at safety. There are, however, limitations which need to be considered in interpretation of the results.

Effectiveness

In a prospective cohort study, allocation of the treatment or intervention is made because of clinical considerations. If the decision to intervene is influenced by factors that are independently related to the outcome (eg, age, duration or severity of the disease), it is possible any observed treatment effect may be due to these factors. There are a number of statistical approaches that can be used to try to adjust for such confounding. Such techniques may attenuate any bias and provide more accurate estimates of a treatment effect. The techniques, however, can only allow for factors that are measured and limited by the available data about them. Further, they can not allow for unmeasured (or unknown) confounding characteristics. Additional limitations to be considered include errors in coding of the disease and non-compliance with the treatment7 . In interpreting data from such studies, clinicians should attempt to determine whether any improvement in patient outcome following treatment is such that it can not be explained by these limitations.

Safety

Rare or uncommon side effects may only become apparent after an intervention has been licensed and in use in clinical practice. National pharmacovigilance systems may capture potential side effects of treatments, though spontaneous notification by prescribers (or patients) may be linked with significant under-reporting — particularly if the event occurs after the treatment has ended. More systematic surveillance systems include patient registers, such as the British Society of Rheumatology biologics register where individuals undergoing treatment with biological therapy are registered and monitored for the occurrence of side effects — including infection and cancer8 . Data may also be available from large administrative databases or databases of patients monitored for routine care. As discussed earlier, if factors influencing allocation of the therapy (including physician or patient preference) are also linked with the adverse outcome this may potentially bias the result. One of the reasons why observational data suggested women taking hormone replacement therapy were at reduced risk of coronary heart disease is thought to be that women who received the therapy were healthier and therefore at lower risk of the disease: no protective effect was observed in subsequent randomised controlled trials.

Results from randomised clinical trials and observational data

McKee9 undertook a systematic review comparing results of randomised and non-randomised studies where a single intervention was evaluated by both methods. He found treatment effects may differ, but one method did not give a consistently greater effect than the other. The results of the non-randomised studies best approximated to the randomised trials when both used the same exclusion criteria and where potential prognostic factors were well understood, measured and appropriately controlled for in the non-randomised studies. More recent reviews provide additional support, though there are conflicting views10,11,12. Observational data can provide evidence of side effects that may not be apparent in randomised controlled trials. Differences between patients seen in day-to-day practice and study subjects may be the reason for the apparent discordance, possibly due to the exclusion criteria used, or that patients in clinical practice were less informed about how to take the treatment or to interactions with other medications/comorbidity

Is there evidence side effects observed in observational studies are similar to those seen in randomised controlled trials of the same therapies? Papanikolaou13 looked at adverse effects of various medical interventions for which data were available from systematic reviews of randomised trials and compared them with the same harms reported in non-randomised studies of the same interventions. In most instances the observational studies estimated smaller risks than the randomised trials. Thus — if anything — where there is evidence from a randomised trial for an adverse effect, the observational studies tended to be more conservative. The mechanism is likely to be due in part to the greater surveillance possible in randomised controlled trials than in observational research, particularly when using data based on routine patient care14.

Conclusion

The randomised controlled clinical trial is the standard method for evaluating the effectiveness and safety of medical interventions. Data from large prospective cohorts, including claims and administrative databases, may provide additional and complementary evidence about safety and effectiveness, thus contributing to the clinical decision-making process; however, it is important that clinicians are aware of the limitations of this type of data.

Conflict of interest: Dr O’Neill has received an education grant from The Alliance for Better Bone Health, a collaboration between Procter and Gamble Pharmaceuticals Limited and sanofi -aventis.

References

  1. Medical Research Council. Clinical trials of new remedies (annotations). Lancet 1931;2:304
  2. Dowd R, Recker RR, Heaney RP. Study subjects and ordinary patients. Osteoporos Int Osteoporos Int 2000; 11: 533-6
  3. Symmons DPM, Silman AJ. The Norfolk Arthritis Register (NOAR). Clin Exp Rheumatol 2003; 21: S94-99
  4. Psaty BM, Heckbert SR, Koepsell TD et al. The risk of myocardial infarction associated with antihypertensive drug therapies. JAMA 1995; 274: 620-5
  5. Zhou Z, Rahme E, Abrahamowicz M et al. Effectiveness of statins for secondary prevention in elderly patients after acute myocardial infarction: an evaluation of class effect. CMAJ 2005; 172: 1187-94
  6. Silverman SL, Watts NB, Delmas PD et al. Effectiveness of bisphosphonates on nonvertebral fractures and hip fractures in the fi rst year of therapy: The risedronate and alendronate (REAL) cohort study. Osteoporos Int 2007 January; 2007 January; 18(1): 25–34
  7. Motheral BR, Fairman KA. The use of claims databases for outcomes research: Rationale, Challenges and Strategies. Clin Therapeutics 1997; 19: 346-66
  8. Silman A, Symmons D, Scott DGI, Griffi ths I. British Society for Rheumatology Biologics Register. Ann Rheum Dis 2003; 62: 28-29
  9. McKee M, Britton A, Black N et al. Interpreting the evidence: choosing between randomised and non-randomised studies. Br Med J 1999;319: 312-5
  10. Concato J, Shah N, Horwitz RI. Randomised controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med Engl J Med 2000;342:1887-92
  11. Benson K, Hartz AJ. A comparison of observational studies and randomised controlled trials. N Engl J Med 2000; 342: 1878-86
  12. Pocock SJ, Elbourne DR. Randomised trials of observational tribulations? N Eng J Med 2000;342:1907-9
  13. Papanikolaou PN, Christidi GD, Ioannidis JPA. Comparison of evidence on harms of medical interventions in randomised and nonrandomised studies. CMAJ 2006; 174: 635-41
  14. Vandenbroucke JP. What is the best evidence for determining harms of medical treatment?. CMAJ 2006;174.5: 645