posts: 48378

This table contains the full text for all our old WORDPRESS content in html form. The format is a bit odd as it represents a direct copy from wordpress content that includes a lot of wordpress specific comments that carry some meaning in terms of layout instructions etc.. The html in the content column can thus be rendered as is but will not look identical to the way it looks on our website as several post processing steps are not applied yet. The content should work though as a readable approximation for every post. For easy searching etc, a markdown copy of the content is available in the markdown column. If you want to query all posts (both wordpress and gdocs), use the posts_unified table or the posts_unified_with_content view.

Data license: CC-BY

This data as json

id	title	slug	type	status	content	archieml	archieml_update_statistics	published_at	updated_at	gdocSuccessorId	authors	excerpt	created_at_in_wordpress	updated_at_in_wordpress	featured_image	formattingOptions	markdown	wpApiSnapshot
48378	Why randomized controlled trials matter and the procedures that strengthen them	randomized-controlled-trials	post	publish	<!-- wp:owid/summary --> <!-- wp:paragraph --> <p>In order to tackle the world's largest problems, we need to be able to identify their causes and effective ways to solve them. How would we know about the effects of a new idea, treatment or policy? Answering this question can be difficult. Some research is poorly done and there are incentives for scientists and funders to exaggerate their findings.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>In this post, I look at a key tool to study causal effects reliably: randomized controlled trials (RCTs). What makes RCTs useful is the set of procedures they are expected to follow, such as placebo controls, concealment, randomization and pre-registration. These procedures are layers of protection against challenges that scientists usually face in research. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>To be able to read published research, we need to understand why these procedures matter and why they are not always followed in practice. With this knowledge, we have a greater ability to identify the causes and solutions to the largest problems that affect us.</p> <!-- /wp:paragraph --> <!-- /wp:owid/summary --> <!-- wp:paragraph --> <p>At Our World in Data, we bring attention to the world's largest problems. We explore what these problems are, why they matter and how large they are. Whenever possible, we try to explain why these problems exist, and how we might solve them.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>To make progress, we need to be able to identify real solutions and evaluate them carefully. But doing this well is not simple. It's difficult for scientists to collect good evidence on the costs and benefits of a new idea, policy, or treatment. And it's challenging for decision-makers to scrutinize the evidence that exists, to make better decisions.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>What we need are reliable ways to distinguish between ideas that work and those that don't.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>In this post, I will explain a crucial tool that helps us do this – randomized controlled trials (RCTs). We will see that RCTs matter for three reasons: when we don't know about the effects of interventions, when we don't know how to study them, and when scientific research is affected by biases.</p> <!-- /wp:paragraph --> <!-- wp:heading {"level":4} --> <h4>What are randomized controlled trials?</h4> <!-- /wp:heading --> <!-- wp:paragraph --> <p>To begin with, what are RCTs? These are experiments where people are given, at random, either an intervention (such as a drug) or a control and then followed up to see how they fare on various outcomes.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>RCTs are conducted by researchers around the world. In the map, you can see how many RCTs have ever been published in high-ranked medical journals, by the country where the first author was based. Over 18,000 of these were from the United States, but most countries have had fewer than a hundred.{ref}Catalá-López, F., Aleixandre-Benavent, R., Caulley, L., Hutton, B., Tabarés-Seisdedos, R., Moher, D., & Alonso-Arroyo, A. (2020). Global mapping of randomised trials related articles published in high-impact-factor medical journals: a cross-sectional analysis. <em>Trials</em>, <em>21</em>(1), 1-24.{/ref} RCTs have also become more common over time.{ref}Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., & Otte, W. M. (2021). The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. <em>PLOS Biology</em>, <em>19</em>(4), e3001162. <a href="https://doi.org/10.1371/journal.pbio.3001162">https://doi.org/10.1371/journal.pbio.3001162</a>{/ref}</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>It's easy to take RCTs for granted, but these trials have transformed our understanding of cause and effect. They are a powerful tool to illuminate what is unknown or uncertain; to discern whether something works and how well it works.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>But it's also important to recognize that these trials are not always perfect, and understand why.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>The strengths of RCTs are subtle: they are powerful because of the set of procedures that they are expected to follow. This includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>In this post, we will explore why these procedures matter – how each one adds a layer of protection against complications that scientists face when they do research. </p> <!-- /wp:paragraph --> <!-- wp:html --> <iframe src="https://ourworldindata.org/grapher/total-number-of-rcts-published-in-high-ranked-medical-journals?tab=map&country=GBR~USA~IND~DNK~NGA~BOL~MEX~FRA~JPN~CHN" loading="lazy" style="width: 100%; height: 600px; border: 0px none;"></iframe> <!-- /wp:html --> <!-- wp:heading {"level":4} --> <h4>The fundamental problem of causal inference</h4> <!-- /wp:heading --> <!-- wp:paragraph --> <p>We make decisions based on our understanding of how things work – we try to predict the consequences of our actions. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>But understanding cause and effect is not just crucial for personal decisions: our knowledge of what works can have large consequences for people around us. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>An example is antiretroviral therapy (ART), which is used to treat <a href="http://ourworldindata.org/hiv-aids">HIV/AIDS</a>. The benefits of these drugs were surprising. One of the first ART drugs discovered was azidothymidine, which had previously been abandoned as an ineffective treatment for cancer.{ref}National Research Council (U.S.) (1993). The social impact of AIDS in the United States. 4. Clinical Research and Drug Regulation. <em>National Academy Press.</em>{/ref} The discovery that azidothymidine and other antiretroviral therapies worked, and their use worldwide, has prevented millions of deaths from HIV/AIDS, as the chart shows.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Discovering what works can save lives. But discovering what <em>doesn’t</em> work can do the same. It means we can redirect our time and resources away from things that don't work, towards things that do.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Even when we already know that something works, understanding how <em>well</em> it works can help us make better decisions. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>An example for this is <a href="https://ourworldindata.org/grapher/bcg-immunization-coverage-for-tb-among-1-year-olds?country=~OWID_WRL">the BCG vaccine</a>, which reduces the risk of tuberculosis. The efficacy of this vaccine is different for different people around the world, and the reasons for this are unclear.{ref}Dockrell, H. M., & Smith, S. G. (2017). What Have We Learnt about BCG Vaccination in the Last 20 Years? <em>Frontiers in Immunology</em>, <em>8</em>, 1134. <a href="https://doi.org/10.3389/fimmu.2017.01134">https://doi.org/10.3389/fimmu.2017.01134</a> <br>Mangtani, P., Abubakar, I., Ariti, C., Beynon, R., Pimpin, L., Fine, P. E. M., Rodrigues, L. C., Smith, P. G., Lipman, M., Whiting, P. F., & Sterne, J. A. (2014). Protection by BCG Vaccine Against Tuberculosis: A Systematic Review of Randomized Controlled Trials. <em>Clinical Infectious Diseases</em>, <em>58</em>(4), 470–480. <a href="https://doi.org/10.1093/cid/cit790%7B/ref">https://doi.org/10.1093/cid/cit790</a>{/ref} Still, knowing this is informative, because it tells us that more effort is needed to protect people against the disease in places where the benefit of the vaccine is low.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>If there was a reliable way to know about the effects of the measures we took, we could prioritize solutions that are most effective.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>So, how would we understand cause and effect without trials?</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>One way is through observation: we can observe what different people did and track their outcomes afterwards. But it's possible that the events that followed their actions were simply a coincidence, or that they would have happened anyway.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>The biggest challenge in trying to understand causes and effects is that we are only able to see one version of history. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>When someone makes a particular decision, we are able to see what follows, but we cannot see what would have happened if they had made a different decision. This is known as “the fundamental problem of causal inference."{ref}Holland, P. W. (1986). Statistics and Causal Inference. <em>Journal of the American Statistical Association</em>, <em>81</em>(396), 945–960. <a href="https://doi.org/10.1080/01621459.1986.10478354">https://doi.org/10.1080/01621459.1986.10478354</a><br>Imbens, G. W., & Rubin, D. B. (2010). Rubin causal model. In <em>Microeconometrics</em> (pp. 229–241). Springer.<br>Rubin, D. B. (1977). Assignment to Treatment Group on the Basis of a Covariate. <em>Journal of Educational Statistics</em>, <em>2</em>(1), 1–26. <a href="https://doi.org/10.3102/10769986002001001%7B/ref">https://doi.org/10.3102/10769986002001001</a>{/ref}</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>What this means is that it is impossible to predict the effects that an action will have for an individual person, but we can try to predict the effects it would have on average.</p> <!-- /wp:paragraph --> <!-- wp:html --> <iframe src="https://ourworldindata.org/grapher/hivaids-deaths-and-averted-due-to-art?country=~OWID_WRL" loading="lazy" style="width: 100%; height: 600px; border: 0px none;"></iframe> <!-- /wp:html --> <!-- wp:heading {"level":4} --> <h4>Why randomized controlled trials matter</h4> <!-- /wp:heading --> <!-- wp:columns --> <div class="wp-block-columns"><!-- wp:column --> <div class="wp-block-column"><!-- wp:paragraph --> <p>Sometimes we do not need an RCT to identify causes and effects. The effect of smoking on lung cancer is one example, where scientists could be confident as early as the 1960s that the large increase in lung cancer rates could not have been caused by other factors. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>This was because there was already knowledge from many different lines of evidence.{ref}Hill, G., Millar, W., & Connelly, J. (2003). “The Great Debate”: Smoking, Lung Cancer, and Cancer Epidemiology. <em>Canadian Bulletin of Medical History</em>, <em>20</em>(2), 367-386.{/ref} Experiments, biopsies and population studies all showed that cigarette smoke was associated with specific changes in lung tissue and with the incidence of lung cancer. The association was so large and consistent that it could not be explained by other factors.{ref}Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B., & Wynder, E. L. (1959). Smoking and Lung Cancer: Recent Evidence and a Discussion of Some Questions. <em>JNCI: Journal of the National Cancer Institute</em>. <a href="https://doi.org/10.1093/jnci/22.1.173">https://doi.org/10.1093/jnci/22.1.173</a>{/ref}</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Even now, smoking is estimated to cause a large proportion of cancer deaths in many countries, as you can see in the chart.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>But in other situations, RCTs have made a huge difference to our understanding of the world. Prioritizing the drugs that were shown to be effective in trials saved lives, as we saw with the example of antiretroviral therapy to treat HIV/AIDS.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>People sometimes refer to RCTs as the "gold standard of evidence", but it's more useful to recognize that their strengths emerge from a <em>set of procedures</em> that they aim to follow.</p> <!-- /wp:paragraph --></div> <!-- /wp:column --> <!-- wp:column --> <div class="wp-block-column"><!-- wp:html --> <iframe src="https://ourworldindata.org/grapher/share-of-cancer-deaths-attributed-to-tobacco" loading="lazy" style="width: 100%; height: 600px; border: 0px none;"></iframe> <!-- /wp:html --></div> <!-- /wp:column --></div> <!-- /wp:columns --> <!-- wp:columns --> <div class="wp-block-columns"><!-- wp:column --> <div class="wp-block-column"><!-- wp:heading {"level":4} --> <h4>Why do RCTs matter? </h4> <!-- /wp:heading --> <!-- wp:paragraph --> <p>In my opinion, even though we do have lots of knowledge about various topics, these trials matter for three reasons. They matter because we may not know enough, because we can be wrong, and because we might see what we want to see.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>First, they matter when <em>we don't know</em>. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Randomized controlled trials can illuminate our understanding of effects, especially when there is uncertainty around them. They can help resolve disagreements between experts.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>In some cases, scientists can use other methods to investigate these topics. But when there is insufficient knowledge, other methods may not be enough. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>This is because, in a typical study, scientists need to be able to account for the possibility that other factors are causing the outcome. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>They need to have knowledge about why an effect usually occurs and whether it will happen anyway. They need to know about which other risk factors can cause the outcome and what makes it likely that someone will receive the treatment. They also need to know how to measure these factors properly, and how to account for them.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>The second reason they matter is when <em>we are wrong</em>. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Even when scientists think they know which risk factors affect the outcome, they might be incorrect. They might not account for the right risk factors or they might account for the wrong ones.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>In an RCT where scientists use randomization, blinding and concealment, they can minimize both of these problems. Despite people's risks before the trial, the reason that someone will receive the treatment is the random number they are given. The reason that participants in each group differ at the end of the study is the group they were randomized to. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Even if we don't know about other risk factors, or even if we are wrong about them, these procedures mean that we can still find out whether a treatment has an effect and how large the effect is.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>The third reason is when <em>we see what we want to see</em>. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>When participants expect to feel different after a treatment, they might report that they feel different, even if they didn't receive the treatment at all. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>When scientists want to see people improve after a treatment, they might decide to allocate the treatment to healthier participants. They might measure their data differently or re-analyze their data until they can find an improvement. They might even decide against publishing their findings when they don't find a benefit.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>If scientists use concealment, blinding, and pre-registration, they can reduce these biases. They can protect research against their own prejudices and the expectations of participants. They can also defend scientific knowledge from poor incentives, such as the desires of scientists to hype up their findings and the incentives for pharmaceutical companies to claim that their drugs work.</p> <!-- /wp:paragraph --></div> <!-- /wp:column --> <!-- wp:column --> <div class="wp-block-column"></div> <!-- /wp:column --></div> <!-- /wp:columns --> <!-- wp:heading {"level":3} --> <h3>The layers of protection against bias</h3> <!-- /wp:heading --> <!-- wp:paragraph --> <p>Previously, I described how the strengths of RCTs emerge from the <em>set of procedures</em> that they aim to follow.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>In medicine, this commonly includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.{ref}Although these procedures are also used outside of medicine, it can be difficult to apply them elsewhere. For example, in a trial that tests the effectiveness of talking therapy, it would be known to the participants that they are receiving it; it may not be possible to find a placebo control version to disguise the procedure. Due to constraints in length and focus, I will not detail the advantages of intention-to-treat analysis or experimentation.{/ref} Each of these helps to protect the evidence against distortions, which can come from many sources. But they are not always enforced. Below, we will explore them in more detail.</p> <!-- /wp:paragraph --> <!-- wp:heading {"level":4} --> <h4>The control group gives us a comparison to see what would have happened otherwise</h4> <!-- /wp:heading --> <!-- wp:paragraph --> <p>The most crucial element to understand causes and effects is a control group. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Let's use antidepressants as an example to see why we need them.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>If no one in the world received antidepressants, we wouldn’t know what their effects were. But the opposite is also true. If <em>everyone</em> received antidepressants, we wouldn't know how things would be without them. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>In order to understand the effect of an antidepressant, we need to see which other outcomes are possible. The most important thing we need to understand effects is a <em>comparison</em> – we need some people who receive the treatment and some who don’t. The people who don’t could be our control group.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>An ideal control group should allow us to <em>control</em> the biases and measurement errors in a study completely. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>For example, different doctors may diagnose depression differently. If someone had symptoms of depression, then the chances they get diagnosed should be equal in the antidepressant group and the control group. If they weren't equal, we could mistake differences between the groups for an effect of antidepressants, even if they didn't have any effect.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>In fact, an ideal control group should allow us to control the total effects of <em>all of the other factors</em> that could affect people's mood, not just control for the biases and errors in studies. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>As an example, we know that the symptoms of depression tend to change over time, as I explained in <a href="https://ourworldindata.org/what-is-depression">this earlier post</a>. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>You can see this in the chart. This shows that the symptoms of depression tend to decline over time, among people who are diagnosed with depression but not treated for it. This is measured by seeing how many patients would still meet the threshold for a diagnosis of depression later on. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Almost a quarter of patients with depression (23%) would no longer meet the threshold for depression after three months, despite receiving no treatment. Just over half (53%) would no longer meet the threshold after one year.{ref}Whiteford, H. A., Harris, M. G., McKeon, G., Baxter, A., Pennell, C., Barendregt, J. J., & Wang, J. (2013). Estimating remission from untreated major depression: A systematic review and meta-analysis. <em>Psychological Medicine</em>, <em>43</em>(8), 1569–1585. <a href="https://doi.org/10.1017/S0033291712001717">https://doi.org/10.1017/S0033291712001717</a>{/ref} This change is known as "regression to the mean."</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>If we didn’t have a control group in our study, we might misattribute such an improvement to the antidepressant. We need a control group to know how much their symptoms would improve anyway.</p> <!-- /wp:paragraph --> <!-- wp:image --> <figure class="wp-block-image"><img src="https://lh5.googleusercontent.com/khK-H11avMDGkTpCLAqCJIsIyAJSX63_Blm-15GL-0fkGYoDK-47Q1Ancs-i06algU5S3GRWTTeL4hNMQgfezt2_DlxwUm1kTykSx6oc3Unzbnz0UNaKUbGZdQF4fagJFIwrXvCu" alt=""/></figure> <!-- /wp:image --> <!-- wp:heading {"level":4} --> <h4>Placebos allow us to account for placebo effects</h4> <!-- /wp:heading --> <!-- wp:paragraph --> <p>Some types of controls are special because they resemble the treatment without actually being it – these controls are called placebos. An ideal placebo has all the qualities above and also allows us to account for "placebo effects." In this case, the placebo effect refers to when people's moods improve from the mere <em>procedure</em> of receiving the antidepressant. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>For example, taking a pill might improve people's mood because they believe that taking a pill will give them some benefit, even if the pill does not actually contain any active ingredient. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>How large is the placebo effect? </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Some clinical trials have tried to estimate this by comparing patients who received a placebo to patients who received no treatment at all. Overall, these studies have found that placebo effects are small in clinical trials for many conditions, but are larger for physical treatments, such as acupuncture to treat pain.{ref}Hróbjartsson, A., & Gøtzsche, P. C. (2010). Placebo interventions for all clinical conditions. <em>Cochrane Database of Systematic Reviews</em>. https://doi.org/10.1002/14651858.CD003974.pub3  <br>Hróbjartsson, A., & Gøtzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. <em>New England Journal of Medicine</em>, <em>344</em>(21), 1594–1602. <a href="https://doi.org/10.1056/NEJM200105243442106">https://doi.org/10.1056/NEJM200105243442106</a>{/ref} </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Placebo effects tend to be larger when scientists are recording symptoms reported by patients, rather than when they are measuring something that can be verified by others, such as results from a blood test or death. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>For depression, the placebo effect is non-significant. Studies do not detect a difference in the moods of patients who receive a placebo and those who do not receive any treatment at all.{ref}Hróbjartsson, A., & Gøtzsche, P. C. (2010). Placebo interventions for all clinical conditions. <em>Cochrane Database of Systematic Reviews</em>. <a href="https://doi.org/10.1002/14651858.CD003974.pub3">https://doi.org/10.1002/14651858.CD003974.pub3</a>  <br>Hróbjartsson, A., & Gøtzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. <em>New England Journal of Medicine</em>, <em>344</em>(21), 1594–1602. <a href="https://doi.org/10.1056/NEJM200105243442106%7B/ref">https://doi.org/10.1056/NEJM200105243442106</a>{/ref} Instead, the placebo group serves as a typical control group, to show what would happen to these patients even if they did not receive treatment.</p> <!-- /wp:paragraph --> <!-- wp:heading {"level":4} --> <h4>Randomization ensures that there are no differences between control and treatment group apart from whether they received the treatment</h4> <!-- /wp:heading --> <!-- wp:paragraph --> <p>Before participants are enrolled in a trial, they might already have different levels of risk of developing the outcomes. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Let's look at an example to illustrate this point. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Statins are a type of drug commonly used to prevent stroke. But strokes are more common among people who use statins. Does that mean that statins <em>caused</em> an increase in the rates of stroke? </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>No – people who are prescribed statins are more likely to have cardiovascular problems to begin with, which increases the chances of having a stroke later on. When this is accounted for, researchers find that people who take statins are actually <em>less likely</em> to develop a stroke.{ref}Orkaby, A. R., Gaziano, J. M., Djousse, L., & Driver, J. A. (2017). Statins for Primary Prevention of Cardiovascular Events and Mortality in Older Men. <em>Journal of the American Geriatrics Society</em>, <em>65</em>(11), 2362–2368. <a href="https://doi.org/10.1111/jgs.14993">https://doi.org/10.1111/jgs.14993</a><br>Makihara, N., Kamouchi, M., Hata, J., Matsuo, R., Ago, T., Kuroda, J., Kuwashiro, T., Sugimori, H., & Kitazono, T. (2013). Statins and the risks of stroke recurrence and death after ischemic stroke: The Fukuoka Stroke Registry. <em>Atherosclerosis</em>, <em>231</em>(2), 211–215. <a href="https://doi.org/10.1016/j.atherosclerosis.2013.09.017">https://doi.org/10.1016/j.atherosclerosis.2013.09.017</a><br>Ní Chróinín, D., Asplund, K., Åsberg, S., Callaly, E., Cuadrado-Godia, E., Díez-Tejedor, E., Di Napoli, M., Engelter, S. T., Furie, K. L., Giannopoulos, S., Gotto, A. M., Hannon, N., Jonsson, F., Kapral, M. K., Martí-Fàbregas, J., Martínez-Sánchez, P., Milionis, H. J., Montaner, J., Muscari, A., … Kelly, P. J. (2013). Statin Therapy and Outcome After Ischemic Stroke: Systematic Review and Meta-Analysis of Observational Studies and Randomized Trials. <em>Stroke</em>, <em>44</em>(2), 448–456. <a href="https://doi.org/10.1161/STROKEAHA.112.668277">https://doi.org/10.1161/STROKEAHA.112.668277</a>{/ref}</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>If researchers simply compared the rates of stroke in those who used statins with those who did not, they would miss the fact that there were other differences between the two groups, which could have caused differences in their rates of stroke. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Important differences such as these – which affect people's likelihood of receiving the treatment (using statins) <em>and also</em> affect the outcome (the risk of a stroke) – are called ‘confounders’.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>But it can be difficult to know what all of these confounders might be. It can also be difficult to measure and account for them properly. In fact, scientists can actually <em>worsen</em> a study by accounting for the wrong factors.{ref}Cinelli, C., Forney, A., & Pearl, J. (2020). A crash course in good and bad controls. <em>Available at SSRN, 3689437:</em> <a href="https://dx.doi.org/10.2139/ssrn.3689437">http://dx.doi.org/10.2139/ssrn.3689437</a>{/ref}</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>What happens when participants are randomized in a trial? </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Randomization is the procedure of allocating them into one of two groups at random.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>With randomization, the problems above are minimized: everyone has the possibility of receiving the treatment. Whether people receive the treatment is not determined by the risks they have, but whether they are randomly selected to receive the treatment.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>So, the overall risks of developing the outcome in one group become comparable to the risks in the other group.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Randomization means that it is not a problem when there are confounders that are not known or not measured. Researchers don't have to know about why or how the outcome usually occurs.{ref}Aronow, P., Robins, J. M., Saarinen, T., Sävje, F., & Sekhon, J. (2021). Nonparametric identification is not enough, but randomized controlled trials are. <em>ArXiv Preprint ArXiv:2108.11342</em>.{/ref}</p> <!-- /wp:paragraph --> <!-- wp:heading {"level":4} --> <h4>Concealment and blinding limit the biases of researchers and expectations of participants</h4> <!-- /wp:heading --> <!-- wp:paragraph --> <p>In a clinical trial, participants or scientists might realize which groups they are assigned to. For example, the drug might smell, taste or look different from the placebo. Or it might have obvious benefits or different side effects compared to the placebo.{ref}Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., Cochrane Bias Methods Group, & Cochrane Statistical Methods Group. (2011). The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. <em>BMJ</em>, <em>343</em>(oct18 2), d5928–d5928.<a href="https://doi.org/10.1136/bmj.d5928"> https://doi.org/10.1136/bmj.d5928</a>{/ref}</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Concealment is a first step in preventing this: this is the procedure of preventing scientists from knowing which treatment people <em>will be</em> assigned to. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Blinding is a second step: this is the procedure of preventing participants and scientists from finding out which treatment group people <em>have been</em> assigned to.{ref}Schulz, K. F., Chalmers, I., & Altman, D. G. (2002). The Landscape and Lexicon of Blinding in Randomized Trials. <em>Annals of Internal Medicine</em>, <em>136</em>(3), 254. <a href="https://doi.org/10.7326/0003-4819-136-3-200202050-00022">https://doi.org/10.7326/0003-4819-136-3-200202050-00022</a>{/ref} </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>When blinding is incomplete, it can partly reverse the benefits of randomization. For example, in clinical trials for oral health, the benefits of treatments appear larger when patients and the scientists who assess their health are not blinded sufficiently.{ref}Saltaji, H. <em>et al.</em> Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions. <em>BMC Med Res Methodol</em> <strong>18</strong>, 42 (2018).{/ref} </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>If randomization was maintained, the only reason that groups would differ on their outcomes was the treatment they received. However, if the treatments were not hidden from scientists and participants, other factors could cause differences between them.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Sometimes, blinding is not possible – there might not be something that is safe and closely resembles the treatment, which could be used as a placebo. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Fortunately, this does not necessarily mean that these trials cannot be useful. Researchers can measure verifiable outcomes (such as changes in blood levels or even deaths) to avoid some placebo effects.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>But even when blinding occurs, participants and researchers might still make different decisions, because of the effects of the treatment or placebo. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>For example, some participants might decide to withdraw from the trial or not follow the protocols of the trial closely. Similarly, scientists might guess which groups people are in and therefore treat them differently or measure their outcomes in a biased way.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>It's difficult to predict how this might affect the results of a trial. It could cause us to overestimate or underestimate the benefit of a treatment. For example, in the clinical trials for <a href="https://ourworldindata.org/covid-vaccinations">Covid-19 vaccines</a>, some participants may have guessed that they received the vaccine because they experienced side effects. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>So, they may have believed that they were more protected and took fewer precautions. This means they may have increased their risk of catching Covid-19. This would make it appear as if the vaccines gave less protection than they actually did: it would result in an underestimate of their efficacy.</p> <!-- /wp:paragraph --> <!-- wp:heading {"level":4} --> <h4>Preregistration allows us to hold researchers accountable to their original study plans</h4> <!-- /wp:heading --> <!-- wp:columns --> <div class="wp-block-columns"><!-- wp:column --> <div class="wp-block-column"><!-- wp:paragraph --> <p>Some of the procedures we've explored so far are used to safeguard research against errors and biases that scientists can have. Pre-registration is another procedure that contributes to the same goal. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>After the data in a study is analyzed, scientists have some choice in which results they present to their colleagues and the wider community. This opens up research to the possibility of cherry-picking.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>This problem unfortunately often arises in studies that are sponsored by industry. If a pharmaceutical company is testing their new drugs in trials, disappointing results can lead to financial losses. So, they may decide not to publish them. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>But this problem is not limited to trials conducted by pharmaceutical companies.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>For many reasons, scientists may decide not to publish some of their studies. Or they might re-analyze their data in a different way to find more positive results. Even if scientists want to publish their findings, journals may decide not to publish them because they may be seen as disappointing, controversial or uninteresting. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>To counter these incentives, scientists can follow the practice of "pre-registration." This is when they publicly declare which analyses they plan to do in advance of collecting the data.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>In 2000, the United States Food and Drug Administration (FDA) established an online registry for the details of clinical trials. The FDA required that scientists who were studying therapies for serious diseases needed to provide some details of their study designs and the outcomes that they planned to investigate.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. <em>Jama</em>, <em>307</em>(17), 1861–1864.{/ref}</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>This requirement reduced the bias towards positive findings in published research. We see evidence of this in the chart, showing clinical trials funded by the National Heart, Lung and Blood Institute (NHLBI), which adopted this requirement. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Before 2000 – when pre-registration of studies was not required – most candidate drugs to treat or prevent cardiovascular disease showed large benefits. But most trials published after 2000 showed no benefit.{ref}Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. <em>PLOS ONE</em>, <em>10</em>(8), e0132382. <a href="https://doi.org/10.1371/journal.pone.0132382">https://doi.org/10.1371/journal.pone.0132382</a>{/ref}</p> <!-- /wp:paragraph --></div> <!-- /wp:column --> <!-- wp:column --> <div class="wp-block-column"><!-- wp:image {"id":48462,"sizeSlug":"full","linkDestination":"none"} --> <figure class="wp-block-image size-full"><img src="https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2.jpg" alt="" class="wp-image-48462"/></figure> <!-- /wp:image --></div> <!-- /wp:column --></div> <!-- /wp:columns --> <!-- wp:columns {"className":"is-style-sticky-right"} --> <div class="wp-block-columns is-style-sticky-right"><!-- wp:column --> <div class="wp-block-column"><!-- wp:paragraph --> <p>Over the last two decades, this practice was strengthened and expanded. In 2007, the FDA required that most approved trials must be registered when people are enrolled into the study. They introduced notices and civil penalties for not doing so. Now, similar requirements are in place in Europe, Canada and some other parts of the world.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. <em>Jama</em>, <em>307</em>(17), 1861–1864. <a href="https://doi.org/10.1001/jama.2012.4230">https://doi.org/10.1001/jama.2012.4230</a>{/ref}</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Importantly, sponsors of clinical trials are required to share the results online after the trial is completed. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>But unfortunately many still violate these requirements. For example, less than half (40%) of trials that were conducted in the US since 2017 actually reported their results on time.{ref}DeVito, N. J., & Goldacre, B. (2021). Evaluation of Compliance With Legal Requirements Under the FDA Amendments Act of 2007 for Timely Registration of Clinical Trials, Data Verification, Delayed Reporting, and Trial Document Submission. JAMA Internal Medicine, 181(8), 1128. <a href="https://doi.org/10.1001/jamainternmed.2021.2036">https://doi.org/10.1001/jamainternmed.2021.2036</a>{/ref} </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Here, you can see patterns in reporting results in Europe. According to EU regulations, all clinical trials in the European Economic Area are required to report their results to the European Clinical Trials Register (EU-CTR) within a year of completing the trial. But reporting rates vary a lot by country. By January 2021, most clinical trials by non-commercial sponsors in the UK (94%) reported their results in time, while nearly none (4%) in France had.{ref}Note: Data is shown until January 2021 for trials. After the UK left the European Union in January 2021, clinical trials in the UK were then no longer required to report their results to the EU-CTR. Only data from trials by non-commercial sponsors is shown. This includes trials sponsored by institutions such as universities, hospitals, research foundations and so on.<br>Dal-Ré, R., Goldacre, B., Mahillo-Fernández, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. <em>Journal of Clinical Epidemiology</em>, S0895435621003577. <a href="https://doi.org/10.1016/j.jclinepi.2021.11.005">https://doi.org/10.1016/j.jclinepi.2021.11.005</a>{/ref}</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Although a large share of clinical trials fail to report their results when they are due, this has improved recently: only half (50%) of clinical trials by non-commercial sponsors reported their results in time in 2018, while more than two-thirds (69%) did in 2021.{ref}Dal-Ré, R., Goldacre, B., Mahillo-Fernández, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. <em>Journal of Clinical Epidemiology</em>, S0895435621003577. <a href="https://doi.org/10.1016/j.jclinepi.2021.11.005">https://doi.org/10.1016/j.jclinepi.2021.11.005</a>{/ref}</p> <!-- /wp:paragraph --></div> <!-- /wp:column --> <!-- wp:column --> <div class="wp-block-column"></div> <!-- /wp:column --></div> <!-- /wp:columns --> <!-- wp:columns {"className":"is-style-side-by-side"} --> <div class="wp-block-columns is-style-side-by-side"><!-- wp:column {"width":"50%"} --> <div class="wp-block-column" style="flex-basis:50%"><!-- wp:html --> <iframe src="https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-in-time-by-country?country=GBR~FRA~BEL~NLD~DEU~ITA~ESP~SWE~AUT~DNK" loading="lazy" style="width: 100%; height: 600px; border: 0px none;"></iframe> <!-- /wp:html --></div> <!-- /wp:column --> <!-- wp:column {"width":"50%"} --> <div class="wp-block-column" style="flex-basis:50%"><!-- wp:html --> <iframe src="https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-within-a-year-over-time?country=~European+Economic+Area" loading="lazy" style="width: 100%; height: 600px; border: 0px none;"></iframe> <!-- /wp:html --></div> <!-- /wp:column --></div> <!-- /wp:columns --> <!-- wp:heading {"level":4} --> <h4>Pulling the layers together</h4> <!-- /wp:heading --> <!-- wp:paragraph --> <p>Together, these procedures give us a far more reliable way to distinguish between what works and what doesn't – even when we don't know enough, when we're wrong and when we see what we want to see.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Unfortunately, we have seen that many clinical trials do not follow them. They tend not to report the details of their methods or test whether their participants were blinded to what they received. These standards cannot just be expected from scientists – they require cooperation from funders and journals, and they need to be actively enforced with penalties and incentives.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>We've also seen that trials can still suffer from remaining problems: that participants may drop out of the study and not adhere to the treatment. And clinical trials tend to not share their data or the code that they used to analyze it.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>So, despite all the benefits they provide, we shouldn't see these layers as a fixed checklist. Just as some of them were introduced recently, they may still evolve in the future – open data sharing, registered reports and procedures to reduce dropouts may be next on the list.{ref}Chambers, C., & Tzavella, L. (2020). The past, present, and future of Registered Reports.{/ref}</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>The procedures used in these trials are not a silver bullet. But when they are upheld, they can make these trials a more powerful source of evidence for understanding cause and effect. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>To make progress, we need to be able to understand the problems that affect us, their causes and solutions. Randomized controlled trials can give scientists a reliable way to collect evidence on these important questions, and give us the ability to make better decisions. At Our World in Data, this is why they matter most of all.</p> <!-- /wp:paragraph --> <!-- wp:separator --> <hr class="wp-block-separator"/> <!-- /wp:separator --> <!-- wp:heading {"level":5} --> <h5><em>Keep reading on Our World in Data</em></h5> <!-- /wp:heading --> <!-- wp:owid/prominent-link {"title":"","linkUrl":"https://ourworldindata.org/what-is-depression","className":"is-style-thin"} /--> <!-- wp:owid/prominent-link {"title":"","linkUrl":"https://ourworldindata.org/mental-health","className":"is-style-thin"} /--> <!-- wp:paragraph --> <p><strong>Acknowledgements:</strong> I would like to thank Darren Dahly, Nathaniel Bechhofer, Hannah Ritchie and Max Roser for reading drafts of this post and their very helpful guidance and suggestions to improve it.</p> <!-- /wp:paragraph --> <!-- wp:separator --> <hr class="wp-block-separator"/> <!-- /wp:separator --> <!-- wp:owid/additional-information --> <!-- wp:heading {"level":3} --> <h3>Appendix: When are randomized controls ethical?</h3> <!-- /wp:heading --> <!-- wp:columns {"className":"is-style-sticky-right"} --> <div class="wp-block-columns is-style-sticky-right"><!-- wp:column --> <div class="wp-block-column"><!-- wp:paragraph --> <p>Trials are useful to test new ideas or treatments, when people do not have enough knowledge to recommend them on a larger scale. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>But people also have ethical concerns surrounding trials of treatments that are already used by doctors. For one, they might believe that it isn't justified to give patients a placebo when a drug is available, even if it is unclear how well the drug works. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>How can we balance these concerns with the benefits of trials? When are trials important?</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>A leading view is that they are justified when there is disagreement or uncertainty about the benefits of an intervention.{ref}Wendler, D. The Ethics of Clinical Research. in The Stanford Encyclopedia of Philosophy (ed. Zalta, E. N.) (Metaphysics Research Lab, Stanford University, 2021).<br>London, A. J. Equipoise in Research: Integrating Ethics and Science in Human Research. JAMA 317, 525 (2017).{/ref}</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>Outside of a trial, some doctors might prescribe a particular treatment, while other doctors would not, because they believe that same treatment is ineffective. In this case, the options in a trial – an intervention or control – would not be so different from what people could already encounter. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>When experts disagree on a topic, an RCT is useful because it can resolve these conflicts and inform us about which option is better. </p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p>So, to make the most of trials, they should be planned in line with the questions that people have. For example, if we wanted to understand whether vaccines gave less protection to people who were immunocompromised, then we should plan a trial with enough participants with these conditions.</p> <!-- /wp:paragraph --> <!-- wp:paragraph --> <p></p> <!-- /wp:paragraph --></div> <!-- /wp:column --> <!-- wp:column --> <div class="wp-block-column"></div> <!-- /wp:column --></div> <!-- /wp:columns --> <!-- /wp:owid/additional-information --> <!-- wp:paragraph --> <p></p> <!-- /wp:paragraph -->	{ "id": "wp-48378", "slug": "randomized-controlled-trials", "content": { "toc": [], "body": [ { "text": [ { "type": "text", "value": [ { "text": "In order to tackle the world's largest problems, we need to be able to identify their causes and effective ways to solve them. How would we know about the effects of a new idea, treatment or policy? Answering this question can be difficult. Some research is poorly done and there are incentives for scientists and funders to exaggerate their findings.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "In this post, I look at a key tool to study causal effects reliably: randomized controlled trials (RCTs). What makes RCTs useful is the set of procedures they are expected to follow, such as placebo controls, concealment, randomization and pre-registration. These procedures are layers of protection against challenges that scientists usually face in research. ", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "To be able to read published research, we need to understand why these procedures matter and why they are not always followed in practice. With this knowledge, we have a greater ability to identify the causes and solutions to the largest problems that affect us.", "spanType": "span-simple-text" } ], "parseErrors": [] } ], "type": "callout", "title": "Summary", "parseErrors": [] }, { "type": "text", "value": [ { "text": "At Our World in Data, we bring attention to the world's largest problems. We explore what these problems are, why they matter and how large they are. Whenever possible, we try to explain why these problems exist, and how we might solve them.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "To make progress, we need to be able to identify real solutions and evaluate them carefully. But doing this well is not simple. It's difficult for scientists to collect good evidence on the costs and benefits of a new idea, policy, or treatment. And it's challenging for decision-makers to scrutinize the evidence that exists, to make better decisions.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "What we need are reliable ways to distinguish between ideas that work and those that don't.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "In this post, I will explain a crucial tool that helps us do this \u2013 randomized controlled trials (RCTs). We will see that RCTs matter for three reasons: when we don't know about the effects of interventions, when we don't know how to study them, and when scientific research is affected by biases.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "text": [ { "text": "What are randomized controlled trials?", "spanType": "span-simple-text" } ], "type": "heading", "level": 3, "parseErrors": [] }, { "type": "text", "value": [ { "text": "To begin with, what are RCTs? These are experiments where people are given, at random, either an intervention (such as a drug) or a control and then followed up to see how they fare on various outcomes.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "RCTs are conducted by researchers around the world. In the map, you can see how many RCTs have ever been published in high-ranked medical journals, by the country where the first author was based. Over 18,000 of these were from the United States, but most countries have had fewer than a hundred.{ref}Catal\u00e1-L\u00f3pez, F., Aleixandre-Benavent, R., Caulley, L., Hutton, B., Tabar\u00e9s-Seisdedos, R., Moher, D., & Alonso-Arroyo, A. (2020). Global mapping of randomised trials related articles published in high-impact-factor medical journals: a cross-sectional analysis. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Trials", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "21", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(1), 1-24.{/ref} RCTs have also become more common over time.{ref}Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., & Otte, W. M. (2021). The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. ", "spanType": "span-simple-text" }, { "children": [ { "text": "PLOS Biology", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "19", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(4), e3001162. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1371/journal.pbio.3001162", "children": [ { "text": "https://doi.org/10.1371/journal.pbio.3001162", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "It's easy to take RCTs for granted, but these trials have transformed our understanding of cause and effect. They are a powerful tool to illuminate what is unknown or uncertain; to discern whether something works and how well it works.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "But it's also important to recognize that these trials are not always perfect, and understand why.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "The strengths of RCTs are subtle: they are powerful because of the set of procedures that they are expected to follow. This includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "In this post, we will explore why these procedures matter \u2013 how each one adds a layer of protection against complications that scientists face when they do research.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "url": "https://ourworldindata.org/grapher/total-number-of-rcts-published-in-high-ranked-medical-journals?tab=map&country=GBR~USA~IND~DNK~NGA~BOL~MEX~FRA~JPN~CHN", "type": "chart", "parseErrors": [] }, { "text": [ { "text": "The fundamental problem of causal inference", "spanType": "span-simple-text" } ], "type": "heading", "level": 3, "parseErrors": [] }, { "type": "text", "value": [ { "text": "We make decisions based on our understanding of how things work \u2013 we try to predict the consequences of our actions.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "But understanding cause and effect is not just crucial for personal decisions: our knowledge of what works can have large consequences for people around us.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "An example is antiretroviral therapy (ART), which is used to treat ", "spanType": "span-simple-text" }, { "url": "http://ourworldindata.org/hiv-aids", "children": [ { "text": "HIV/AIDS", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": ". The benefits of these drugs were surprising. One of the first ART drugs discovered was azidothymidine, which had previously been abandoned as an ineffective treatment for cancer.{ref}National Research Council (U.S.) (1993). The social impact of AIDS in the United States. 4. Clinical Research and Drug Regulation. ", "spanType": "span-simple-text" }, { "children": [ { "text": "National Academy Press.", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "{/ref} The discovery that azidothymidine and other antiretroviral therapies worked, and their use worldwide, has prevented millions of deaths from HIV/AIDS, as the chart shows.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Discovering what works can save lives. But discovering what ", "spanType": "span-simple-text" }, { "children": [ { "text": "doesn\u2019t", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " work can do the same. It means we can redirect our time and resources away from things that don't work, towards things that do.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Even when we already know that something works, understanding how ", "spanType": "span-simple-text" }, { "children": [ { "text": "well", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " it works can help us make better decisions.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "An example for this is ", "spanType": "span-simple-text" }, { "url": "https://ourworldindata.org/grapher/bcg-immunization-coverage-for-tb-among-1-year-olds?country=~OWID_WRL", "children": [ { "text": "the BCG vaccine", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": ", which reduces the risk of tuberculosis. The efficacy of this vaccine is different for different people around the world, and the reasons for this are unclear.{ref}Dockrell, H. M., & Smith, S. G. (2017). What Have We Learnt about BCG Vaccination in the Last 20 Years? ", "spanType": "span-simple-text" }, { "children": [ { "text": "Frontiers in Immunology", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "8", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", 1134. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.3389/fimmu.2017.01134", "children": [ { "text": "https://doi.org/10.3389/fimmu.2017.01134", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "spanType": "span-newline" }, { "text": "Mangtani, P., Abubakar, I., Ariti, C., Beynon, R., Pimpin, L., Fine, P. E. M., Rodrigues, L. C., Smith, P. G., Lipman, M., Whiting, P. F., & Sterne, J. A. (2014). Protection by BCG Vaccine Against Tuberculosis: A Systematic Review of Randomized Controlled Trials. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Clinical Infectious Diseases", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "58", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(4), 470\u2013480. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1093/cid/cit790%7B/ref", "children": [ { "text": "https://doi.org/10.1093/cid/cit790", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref} Still, knowing this is informative, because it tells us that more effort is needed to protect people against the disease in places where the benefit of the vaccine is low.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "If there was a reliable way to know about the effects of the measures we took, we could prioritize solutions that are most effective.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "So, how would we understand cause and effect without trials?", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "One way is through observation: we can observe what different people did and track their outcomes afterwards. But it's possible that the events that followed their actions were simply a coincidence, or that they would have happened anyway.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "The biggest challenge in trying to understand causes and effects is that we are only able to see one version of history.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "When someone makes a particular decision, we are able to see what follows, but we cannot see what would have happened if they had made a different decision. This is known as \u201cthe fundamental problem of causal inference.\"{ref}Holland, P. W. (1986). Statistics and Causal Inference. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Journal of the American Statistical Association", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "81", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(396), 945\u2013960. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1080/01621459.1986.10478354", "children": [ { "text": "https://doi.org/10.1080/01621459.1986.10478354", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "spanType": "span-newline" }, { "text": "Imbens, G. W., & Rubin, D. B. (2010). Rubin causal model. In ", "spanType": "span-simple-text" }, { "children": [ { "text": "Microeconometrics", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " (pp. 229\u2013241). Springer.", "spanType": "span-simple-text" }, { "spanType": "span-newline" }, { "text": "Rubin, D. B. (1977). Assignment to Treatment Group on the Basis of a Covariate. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Journal of Educational Statistics", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "2", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(1), 1\u201326. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.3102/10769986002001001%7B/ref", "children": [ { "text": "https://doi.org/10.3102/10769986002001001", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "What this means is that it is impossible to predict the effects that an action will have for an individual person, but we can try to predict the effects it would have on average.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "url": "https://ourworldindata.org/grapher/hivaids-deaths-and-averted-due-to-art?country=~OWID_WRL", "type": "chart", "parseErrors": [] }, { "text": [ { "text": "Why randomized controlled trials matter", "spanType": "span-simple-text" } ], "type": "heading", "level": 3, "parseErrors": [] }, { "left": [ { "type": "text", "value": [ { "text": "Sometimes we do not need an RCT to identify causes and effects. The effect of smoking on lung cancer is one example, where scientists could be confident as early as the 1960s that the large increase in lung cancer rates could not have been caused by other factors.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "This was because there was already knowledge from many different lines of evidence.{ref}Hill, G., Millar, W., & Connelly, J. (2003). \u201cThe Great Debate\u201d: Smoking, Lung Cancer, and Cancer Epidemiology. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Canadian Bulletin of Medical History", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "20", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(2), 367-386.{/ref} Experiments, biopsies and population studies all showed that cigarette smoke was associated with specific changes in lung tissue and with the incidence of lung cancer. The association was so large and consistent that it could not be explained by other factors.{ref}Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B., & Wynder, E. L. (1959). Smoking and Lung Cancer: Recent Evidence and a Discussion of Some Questions. ", "spanType": "span-simple-text" }, { "children": [ { "text": "JNCI: Journal of the National Cancer Institute", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ". ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1093/jnci/22.1.173", "children": [ { "text": "https://doi.org/10.1093/jnci/22.1.173", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Even now, smoking is estimated to cause a large proportion of cancer deaths in many countries, as you can see in the chart.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "But in other situations, RCTs have made a huge difference to our understanding of the world. Prioritizing the drugs that were shown to be effective in trials saved lives, as we saw with the example of antiretroviral therapy to treat HIV/AIDS.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "People sometimes refer to RCTs as the \"gold standard of evidence\", but it's more useful to recognize that their strengths emerge from a ", "spanType": "span-simple-text" }, { "children": [ { "text": "set of procedures", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " that they aim to follow.", "spanType": "span-simple-text" } ], "parseErrors": [] } ], "type": "sticky-right", "right": [ { "url": "https://ourworldindata.org/grapher/share-of-cancer-deaths-attributed-to-tobacco", "type": "chart", "parseErrors": [] } ], "parseErrors": [] }, { "text": [ { "text": "Why do RCTs matter?\u00a0", "spanType": "span-simple-text" } ], "type": "heading", "level": 3, "parseErrors": [] }, { "type": "text", "value": [ { "text": "In my opinion, even though we do have lots of knowledge about various topics, these trials matter for three reasons. They matter because we may not know enough, because we can be wrong, and because we might see what we want to see.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "First, they matter when ", "spanType": "span-simple-text" }, { "children": [ { "text": "we don't know", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ".\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Randomized controlled trials can illuminate our understanding of effects, especially when there is uncertainty around them. They can help resolve disagreements between experts.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "In some cases, scientists can use other methods to investigate these topics. But when there is insufficient knowledge, other methods may not be enough.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "This is because, in a typical study, scientists need to be able to account for the possibility that other factors are causing the outcome.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "They need to have knowledge about why an effect usually occurs and whether it will happen anyway. They need to know about which other risk factors can cause the outcome and what makes it likely that someone will receive the treatment. They also need to know how to measure these factors properly, and how to account for them.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "The second reason they matter is when ", "spanType": "span-simple-text" }, { "children": [ { "text": "we are wrong", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ".\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Even when scientists think they know which risk factors affect the outcome, they might be incorrect. They might not account for the right risk factors or they might account for the wrong ones.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "In an RCT where scientists use randomization, blinding and concealment, they can minimize both of these problems. Despite people's risks before the trial, the reason that someone will receive the treatment is the random number they are given. The reason that participants in each group differ at the end of the study is the group they were randomized to.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Even if we don't know about other risk factors, or even if we are wrong about them, these procedures mean that we can still find out whether a treatment has an effect and how large the effect is.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "The third reason is when ", "spanType": "span-simple-text" }, { "children": [ { "text": "we see what we want to see", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ".\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "When participants expect to feel different after a treatment, they might report that they feel different, even if they didn't receive the treatment at all.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "When scientists want to see people improve after a treatment, they might decide to allocate the treatment to healthier participants. They might measure their data differently or re-analyze their data until they can find an improvement. They might even decide against publishing their findings when they don't find a benefit.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "If scientists use concealment, blinding, and pre-registration, they can reduce these biases. They can protect research against their own prejudices and the expectations of participants. They can also defend scientific knowledge from poor incentives, such as the desires of scientists to hype up their findings and the incentives for pharmaceutical companies to claim that their drugs work.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "text": [ { "text": "The layers of protection against bias", "spanType": "span-simple-text" } ], "type": "heading", "level": 2, "parseErrors": [] }, { "type": "text", "value": [ { "text": "Previously, I described how the strengths of RCTs emerge from the ", "spanType": "span-simple-text" }, { "children": [ { "text": "set of procedures", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " that they aim to follow.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "In medicine, this commonly includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.{ref}Although these procedures are also used outside of medicine, it can be difficult to apply them elsewhere. For example, in a trial that tests the effectiveness of talking therapy, it would be known to the participants that they are receiving it; it may not be possible to find a placebo control version to disguise the procedure. Due to constraints in length and focus, I will not detail the advantages of intention-to-treat analysis or experimentation.{/ref} Each of these helps to protect the evidence against distortions, which can come from many sources. But they are not always enforced. Below, we will explore them in more detail.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "text": [ { "text": "The control group gives us a comparison to see what would have happened otherwise", "spanType": "span-simple-text" } ], "type": "heading", "level": 3, "parseErrors": [] }, { "type": "text", "value": [ { "text": "The most crucial element to understand causes and effects is a control group.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Let's use antidepressants as an example to see why we need them.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "If no one in the world received antidepressants, we wouldn\u2019t know what their effects were. But the opposite is also true. If ", "spanType": "span-simple-text" }, { "children": [ { "text": "everyone", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " received antidepressants, we wouldn't know how things would be without them.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "In order to understand the effect of an antidepressant, we need to see which other outcomes are possible. The most important thing we need to understand effects is a ", "spanType": "span-simple-text" }, { "children": [ { "text": "comparison", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " \u2013 we need some people who receive the treatment and some who don\u2019t. The people who don\u2019t could be our control group.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "An ideal control group should allow us to ", "spanType": "span-simple-text" }, { "children": [ { "text": "control", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " the biases and measurement errors in a study completely.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "For example, different doctors may diagnose depression differently. If someone had symptoms of depression, then the chances they get diagnosed should be equal in the antidepressant group and the control group. If they weren't equal, we could mistake differences between the groups for an effect of antidepressants, even if they didn't have any effect.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "In fact, an ideal control group should allow us to control the total effects of ", "spanType": "span-simple-text" }, { "children": [ { "text": "all of the other factors", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " that could affect people's mood, not just control for the biases and errors in studies.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "As an example, we know that the symptoms of depression tend to change over time, as I explained in ", "spanType": "span-simple-text" }, { "url": "https://ourworldindata.org/what-is-depression", "children": [ { "text": "this earlier post", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": ".\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "You can see this in the chart. This shows that the symptoms of depression tend to decline over time, among people who are diagnosed with depression but not treated for it. This is measured by seeing how many patients would still meet the threshold for a diagnosis of depression later on.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Almost a quarter of patients with depression (23%) would no longer meet the threshold for depression after three months, despite receiving no treatment. Just over half (53%) would no longer meet the threshold after one year.{ref}Whiteford, H. A., Harris, M. G., McKeon, G., Baxter, A., Pennell, C., Barendregt, J. J., & Wang, J. (2013). Estimating remission from untreated major depression: A systematic review and meta-analysis. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Psychological Medicine", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "43", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(8), 1569\u20131585. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1017/S0033291712001717", "children": [ { "text": "https://doi.org/10.1017/S0033291712001717", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref} This change is known as \"regression to the mean.\"", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "If we didn\u2019t have a control group in our study, we might misattribute such an improvement to the antidepressant. We need a control group to know how much their symptoms would improve anyway.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "alt": "", "size": "wide", "type": "image", "filename": "khK-H11avMDGkTpCLAqCJIsIyAJSX63_Blm-15GL-0fkGYoDK-47Q1Ancs-i06algU5S3GRWTTeL4hNMQgfezt2_DlxwUm1kTykSx6oc3Unzbnz0UNaKUbGZdQF4fagJFIwrXvCu", "parseErrors": [] }, { "text": [ { "text": "Placebos allow us to account for placebo effects", "spanType": "span-simple-text" } ], "type": "heading", "level": 3, "parseErrors": [] }, { "type": "text", "value": [ { "text": "Some types of controls are special because they resemble the treatment without actually being it \u2013 these controls are called placebos. An ideal placebo has all the qualities above and also allows us to account for \"placebo effects.\" In this case, the placebo effect refers to when people's moods improve from the mere ", "spanType": "span-simple-text" }, { "children": [ { "text": "procedure", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " of receiving the antidepressant.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "For example, taking a pill might improve people's mood because they believe that taking a pill will give them some benefit, even if the pill does not actually contain any active ingredient.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "How large is the placebo effect?\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Some clinical trials have tried to estimate this by comparing patients who received a placebo to patients who received no treatment at all. Overall, these studies have found that placebo effects are small in clinical trials for many conditions, but are larger for physical treatments, such as acupuncture to treat pain.{ref}Hr\u00f3bjartsson, A., & G\u00f8tzsche, P. C. (2010). Placebo interventions for all clinical conditions. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Cochrane Database of Systematic Reviews", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ". https://doi.org/10.1002/14651858.CD003974.pub3 \u00a0", "spanType": "span-simple-text" }, { "spanType": "span-newline" }, { "text": "Hr\u00f3bjartsson, A., & G\u00f8tzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. ", "spanType": "span-simple-text" }, { "children": [ { "text": "New England Journal of Medicine", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "344", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(21), 1594\u20131602. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1056/NEJM200105243442106", "children": [ { "text": "https://doi.org/10.1056/NEJM200105243442106", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Placebo effects tend to be larger when scientists are recording symptoms reported by patients, rather than when they are measuring something that can be verified by others, such as results from a blood test or death.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "For depression, the placebo effect is non-significant. Studies do not detect a difference in the moods of patients who receive a placebo and those who do not receive any treatment at all.{ref}Hr\u00f3bjartsson, A., & G\u00f8tzsche, P. C. (2010). Placebo interventions for all clinical conditions. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Cochrane Database of Systematic Reviews", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ". ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1002/14651858.CD003974.pub3", "children": [ { "text": "https://doi.org/10.1002/14651858.CD003974.pub3", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "spanType": "span-newline" }, { "text": "Hr\u00f3bjartsson, A., & G\u00f8tzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. ", "spanType": "span-simple-text" }, { "children": [ { "text": "New England Journal of Medicine", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "344", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(21), 1594\u20131602. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1056/NEJM200105243442106%7B/ref", "children": [ { "text": "https://doi.org/10.1056/NEJM200105243442106", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref} Instead, the placebo group serves as a typical control group, to show what would happen to these patients even if they did not receive treatment.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "text": [ { "text": "Randomization ensures that there are no differences between control and treatment group apart from whether they received the treatment", "spanType": "span-simple-text" } ], "type": "heading", "level": 3, "parseErrors": [] }, { "type": "text", "value": [ { "text": "Before participants are enrolled in a trial, they might already have different levels of risk of developing the outcomes.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Let's look at an example to illustrate this point.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Statins are a type of drug commonly used to prevent stroke. But strokes are more common among people who use statins. Does that mean that statins ", "spanType": "span-simple-text" }, { "children": [ { "text": "caused", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " an increase in the rates of stroke?\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "No \u2013 people who are prescribed statins are more likely to have cardiovascular problems to begin with, which increases the chances of having a stroke later on. When this is accounted for, researchers find that people who take statins are actually ", "spanType": "span-simple-text" }, { "children": [ { "text": "less likely", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " to develop a stroke.{ref}Orkaby, A. R., Gaziano, J. M., Djousse, L., & Driver, J. A. (2017). Statins for Primary Prevention of Cardiovascular Events and Mortality in Older Men. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Journal of the American Geriatrics Society", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "65", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(11), 2362\u20132368. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1111/jgs.14993", "children": [ { "text": "https://doi.org/10.1111/jgs.14993", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "spanType": "span-newline" }, { "text": "Makihara, N., Kamouchi, M., Hata, J., Matsuo, R., Ago, T., Kuroda, J., Kuwashiro, T., Sugimori, H., & Kitazono, T. (2013). Statins and the risks of stroke recurrence and death after ischemic stroke: The Fukuoka Stroke Registry. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Atherosclerosis", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "231", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(2), 211\u2013215. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1016/j.atherosclerosis.2013.09.017", "children": [ { "text": "https://doi.org/10.1016/j.atherosclerosis.2013.09.017", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "spanType": "span-newline" }, { "text": "N\u00ed Chr\u00f3in\u00edn, D., Asplund, K., \u00c5sberg, S., Callaly, E., Cuadrado-Godia, E., D\u00edez-Tejedor, E., Di Napoli, M., Engelter, S. T., Furie, K. L., Giannopoulos, S., Gotto, A. M., Hannon, N., Jonsson, F., Kapral, M. K., Mart\u00ed-F\u00e0bregas, J., Mart\u00ednez-S\u00e1nchez, P., Milionis, H. J., Montaner, J., Muscari, A., \u2026 Kelly, P. J. (2013). Statin Therapy and Outcome After Ischemic Stroke: Systematic Review and Meta-Analysis of Observational Studies and Randomized Trials. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Stroke", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "44", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(2), 448\u2013456. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1161/STROKEAHA.112.668277", "children": [ { "text": "https://doi.org/10.1161/STROKEAHA.112.668277", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "If researchers simply compared the rates of stroke in those who used statins with those who did not, they would miss the fact that there were other differences between the two groups, which could have caused differences in their rates of stroke.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Important differences such as these \u2013 which affect people's likelihood of receiving the treatment (using statins) ", "spanType": "span-simple-text" }, { "children": [ { "text": "and also", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " affect the outcome (the risk of a stroke) \u2013 are called \u2018confounders\u2019.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "But it can be difficult to know what all of these confounders might be. It can also be difficult to measure and account for them properly. In fact, scientists can actually ", "spanType": "span-simple-text" }, { "children": [ { "text": "worsen", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " a study by accounting for the wrong factors.{ref}Cinelli, C., Forney, A., & Pearl, J. (2020). A crash course in good and bad controls. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Available at SSRN, 3689437:", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "url": "https://dx.doi.org/10.2139/ssrn.3689437", "children": [ { "text": "http://dx.doi.org/10.2139/ssrn.3689437", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "What happens when participants are randomized in a trial?\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Randomization is the procedure of allocating them into one of two groups at random.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "With randomization, the problems above are minimized: everyone has the possibility of receiving the treatment. Whether people receive the treatment is not determined by the risks they have, but whether they are randomly selected to receive the treatment.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "So, the overall risks of developing the outcome in one group become comparable to the risks in the other group.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Randomization means that it is not a problem when there are confounders that are not known or not measured. Researchers don't have to know about why or how the outcome usually occurs.{ref}Aronow, P., Robins, J. M., Saarinen, T., S\u00e4vje, F., & Sekhon, J. (2021). Nonparametric identification is not enough, but randomized controlled trials are. ", "spanType": "span-simple-text" }, { "children": [ { "text": "ArXiv Preprint ArXiv:2108.11342", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ".{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "text": [ { "text": "Concealment and blinding limit the biases of researchers and expectations of participants", "spanType": "span-simple-text" } ], "type": "heading", "level": 3, "parseErrors": [] }, { "type": "text", "value": [ { "text": "In a clinical trial, participants or scientists might realize which groups they are assigned to. For example, the drug might smell, taste or look different from the placebo. Or it might have obvious benefits or different side effects compared to the placebo.{ref}Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., Cochrane Bias Methods Group, & Cochrane Statistical Methods Group. (2011). The Cochrane Collaboration\u2019s tool for assessing risk of bias in randomised trials. ", "spanType": "span-simple-text" }, { "children": [ { "text": "BMJ", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "343", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(oct18 2), d5928\u2013d5928.", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1136/bmj.d5928", "children": [ { "text": " https://doi.org/10.1136/bmj.d5928", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Concealment is a first step in preventing this: this is the procedure of preventing scientists from knowing which treatment people ", "spanType": "span-simple-text" }, { "children": [ { "text": "will be", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " assigned to.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Blinding is a second step: this is the procedure of preventing participants and scientists from finding out which treatment group people ", "spanType": "span-simple-text" }, { "children": [ { "text": "have been", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " assigned to.{ref}Schulz, K. F., Chalmers, I., & Altman, D. G. (2002). The Landscape and Lexicon of Blinding in Randomized Trials. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Annals of Internal Medicine", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "136", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(3), 254. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.7326/0003-4819-136-3-200202050-00022", "children": [ { "text": "https://doi.org/10.7326/0003-4819-136-3-200202050-00022", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "When blinding is incomplete, it can partly reverse the benefits of randomization. For example, in clinical trials for oral health, the benefits of treatments appear larger when patients and the scientists who assess their health are not blinded sufficiently.{ref}Saltaji, H. ", "spanType": "span-simple-text" }, { "children": [ { "text": "et al.", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": " Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions. ", "spanType": "span-simple-text" }, { "children": [ { "text": "BMC Med Res Methodol", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "children": [ { "text": "18", "spanType": "span-simple-text" } ], "spanType": "span-bold" }, { "text": ", 42 (2018).{/ref}\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "If randomization was maintained, the only reason that groups would differ on their outcomes was the treatment they received. However, if the treatments were not hidden from scientists and participants, other factors could cause differences between them.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Sometimes, blinding is not possible \u2013 there might not be something that is safe and closely resembles the treatment, which could be used as a placebo.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Fortunately, this does not necessarily mean that these trials cannot be useful. Researchers can measure verifiable outcomes (such as changes in blood levels or even deaths) to avoid some placebo effects.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "But even when blinding occurs, participants and researchers might still make different decisions, because of the effects of the treatment or placebo.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "For example, some participants might decide to withdraw from the trial or not follow the protocols of the trial closely. Similarly, scientists might guess which groups people are in and therefore treat them differently or measure their outcomes in a biased way.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "It's difficult to predict how this might affect the results of a trial. It could cause us to overestimate or underestimate the benefit of a treatment. For example, in the clinical trials for ", "spanType": "span-simple-text" }, { "url": "https://ourworldindata.org/covid-vaccinations", "children": [ { "text": "Covid-19 vaccines", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": ", some participants may have guessed that they received the vaccine because they experienced side effects.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "So, they may have believed that they were more protected and took fewer precautions. This means they may have increased their risk of catching Covid-19. This would make it appear as if the vaccines gave less protection than they actually did: it would result in an underestimate of their efficacy.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "text": [ { "text": "Preregistration allows us to hold researchers accountable to their original study plans", "spanType": "span-simple-text" } ], "type": "heading", "level": 3, "parseErrors": [] }, { "left": [ { "type": "text", "value": [ { "text": "Some of the procedures we've explored so far are used to safeguard research against errors and biases that scientists can have. Pre-registration is another procedure that contributes to the same goal.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "After the data in a study is analyzed, scientists have some choice in which results they present to their colleagues and the wider community. This opens up research to the possibility of cherry-picking.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "This problem unfortunately often arises in studies that are sponsored by industry. If a pharmaceutical company is testing their new drugs in trials, disappointing results can lead to financial losses. So, they may decide not to publish them.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "But this problem is not limited to trials conducted by pharmaceutical companies.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "For many reasons, scientists may decide not to publish some of their studies. Or they might re-analyze their data in a different way to find more positive results. Even if scientists want to publish their findings, journals may decide not to publish them because they may be seen as disappointing, controversial or uninteresting.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "To counter these incentives, scientists can follow the practice of \"pre-registration.\" This is when they publicly declare which analyses they plan to do in advance of collecting the data.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "In 2000, the United States Food and Drug Administration (FDA) established an online registry for the details of clinical trials. The FDA required that scientists who were studying therapies for serious diseases needed to provide some details of their study designs and the outcomes that they planned to investigate.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Jama", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "307", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(17), 1861\u20131864.{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "This requirement reduced the bias towards positive findings in published research. We see evidence of this in the chart, showing clinical trials funded by the National Heart, Lung and Blood Institute (NHLBI), which adopted this requirement. ", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Before 2000 \u2013 when pre-registration of studies was not required \u2013 most candidate drugs to treat or prevent cardiovascular disease showed large benefits. But most trials published after 2000 showed no benefit.{ref}Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. ", "spanType": "span-simple-text" }, { "children": [ { "text": "PLOS ONE", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "10", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(8), e0132382. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1371/journal.pone.0132382", "children": [ { "text": "https://doi.org/10.1371/journal.pone.0132382", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] } ], "type": "sticky-right", "right": [ { "alt": "", "size": "wide", "type": "image", "filename": "Efficacy-in-trials-before-and-after-registration-requirement2.jpg", "parseErrors": [] } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Over the last two decades, this practice was strengthened and expanded. In 2007, the FDA required that most approved trials must be registered when people are enrolled into the study. They introduced notices and civil penalties for not doing so. Now, similar requirements are in place in Europe, Canada and some other parts of the world.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Jama", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", ", "spanType": "span-simple-text" }, { "children": [ { "text": "307", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": "(17), 1861\u20131864. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1001/jama.2012.4230", "children": [ { "text": "https://doi.org/10.1001/jama.2012.4230", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Importantly, sponsors of clinical trials are required to share the results online after the trial is completed.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "But unfortunately many still violate these requirements. For example, less than half (40%) of trials that were conducted in the US since 2017 actually reported their results on time.{ref}DeVito, N. J., & Goldacre, B. (2021). Evaluation of Compliance With Legal Requirements Under the FDA Amendments Act of 2007 for Timely Registration of Clinical Trials, Data Verification, Delayed Reporting, and Trial Document Submission. JAMA Internal Medicine, 181(8), 1128. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1001/jamainternmed.2021.2036", "children": [ { "text": "https://doi.org/10.1001/jamainternmed.2021.2036", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Here, you can see patterns in reporting results in Europe. According to EU regulations, all clinical trials in the European Economic Area are required to report their results to the European Clinical Trials Register (EU-CTR) within a year of completing the trial. But reporting rates vary a lot by country. By January 2021, most clinical trials by non-commercial sponsors in the UK (94%) reported their results in time, while nearly none (4%) in France had.{ref}Note: Data is shown until January 2021 for trials. After the UK left the European Union in January 2021, clinical trials in the UK were then no longer required to report their results to the EU-CTR. Only data from trials by non-commercial sponsors is shown. This includes trials sponsored by institutions such as universities, hospitals, research foundations and so on.", "spanType": "span-simple-text" }, { "spanType": "span-newline" }, { "text": "Dal-R\u00e9, R., Goldacre, B., Mahillo-Fern\u00e1ndez, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Journal of Clinical Epidemiology", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", S0895435621003577. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1016/j.jclinepi.2021.11.005", "children": [ { "text": "https://doi.org/10.1016/j.jclinepi.2021.11.005", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Although a large share of clinical trials fail to report their results when they are due, this has improved recently: only half (50%) of clinical trials by non-commercial sponsors reported their results in time in 2018, while more than two-thirds (69%) did in 2021.{ref}Dal-R\u00e9, R., Goldacre, B., Mahillo-Fern\u00e1ndez, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. ", "spanType": "span-simple-text" }, { "children": [ { "text": "Journal of Clinical Epidemiology", "spanType": "span-simple-text" } ], "spanType": "span-italic" }, { "text": ", S0895435621003577. ", "spanType": "span-simple-text" }, { "url": "https://doi.org/10.1016/j.jclinepi.2021.11.005", "children": [ { "text": "https://doi.org/10.1016/j.jclinepi.2021.11.005", "spanType": "span-simple-text" } ], "spanType": "span-link" }, { "text": "{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "left": [ { "url": "https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-in-time-by-country?country=GBR~FRA~BEL~NLD~DEU~ITA~ESP~SWE~AUT~DNK", "type": "chart", "parseErrors": [] } ], "type": "sticky-right", "right": [ { "url": "https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-within-a-year-over-time?country=~European+Economic+Area", "type": "chart", "parseErrors": [] } ], "parseErrors": [] }, { "text": [ { "text": "Pulling the layers together", "spanType": "span-simple-text" } ], "type": "heading", "level": 3, "parseErrors": [] }, { "type": "text", "value": [ { "text": "Together, these procedures give us a far more reliable way to distinguish between what works and what doesn't \u2013 even when we don't know enough, when we're wrong and when we see what we want to see.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Unfortunately, we have seen that many clinical trials do not follow them. They tend not to report the details of their methods or test whether their participants were blinded to what they received. These standards cannot just be expected from scientists \u2013 they require cooperation from funders and journals, and they need to be actively enforced with penalties and incentives.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "We've also seen that trials can still suffer from remaining problems: that participants may drop out of the study and not adhere to the treatment. And clinical trials tend to not share their data or the code that they used to analyze it.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "So, despite all the benefits they provide, we shouldn't see these layers as a fixed checklist. Just as some of them were introduced recently, they may still evolve in the future \u2013 open data sharing, registered reports and procedures to reduce dropouts may be next on the list.{ref}Chambers, C., & Tzavella, L. (2020). The past, present, and future of Registered Reports.{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "The procedures used in these trials are not a silver bullet. But when they are upheld, they can make these trials a more powerful source of evidence for understanding cause and effect.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "To make progress, we need to be able to understand the problems that affect us, their causes and solutions. Randomized controlled trials can give scientists a reliable way to collect evidence on these important questions, and give us the ability to make better decisions. At Our World in Data, this is why they matter most of all.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "text": [ { "children": [ { "text": "Keep reading on Our World in Data", "spanType": "span-simple-text" } ], "spanType": "span-italic" } ], "type": "heading", "level": 4, "parseErrors": [] }, { "url": "https://ourworldindata.org/what-is-depression", "type": "prominent-link", "title": "", "description": "", "parseErrors": [] }, { "url": "https://ourworldindata.org/mental-health", "type": "prominent-link", "title": "", "description": "", "parseErrors": [] }, { "type": "text", "value": [ { "children": [ { "text": "Acknowledgements:", "spanType": "span-simple-text" } ], "spanType": "span-bold" }, { "text": " I would like to thank Darren Dahly, Nathaniel Bechhofer, Hannah Ritchie and Max Roser for reading drafts of this post and their very helpful guidance and suggestions to improve it.", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "gray-section", "items": [ { "text": [ { "text": "Additional information", "spanType": "span-simple-text" } ], "type": "heading", "level": 2, "parseErrors": [] }, { "type": "expandable-paragraph", "items": [ { "type": "text", "value": [ { "text": "Trials are useful to test new ideas or treatments, when people do not have enough knowledge to recommend them on a larger scale.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "But people also have ethical concerns surrounding trials of treatments that are already used by doctors. For one, they might believe that it isn't justified to give patients a placebo when a drug is available, even if it is unclear how well the drug works.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "How can we balance these concerns with the benefits of trials? When are trials important?", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "A leading view is that they are justified when there is disagreement or uncertainty about the benefits of an intervention.{ref}Wendler, D. The Ethics of Clinical Research. in The Stanford Encyclopedia of Philosophy (ed. Zalta, E. N.) (Metaphysics Research Lab, Stanford University, 2021).", "spanType": "span-simple-text" }, { "spanType": "span-newline" }, { "text": "London, A. J. Equipoise in Research: Integrating Ethics and Science in Human Research. JAMA 317, 525 (2017).{/ref}", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "Outside of a trial, some doctors might prescribe a particular treatment, while other doctors would not, because they believe that same treatment is ineffective. In this case, the options in a trial \u2013 an intervention or control\u00a0\u2013 would not be so different from what people could already encounter.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "When experts disagree on a topic, an RCT is useful because it can resolve these conflicts and inform us about which option is better.\u00a0", "spanType": "span-simple-text" } ], "parseErrors": [] }, { "type": "text", "value": [ { "text": "So, to make the most of trials, they should be planned in line with the questions that people have. For example, if we wanted to understand whether vaccines gave less protection to people who were immunocompromised, then we should plan a trial with enough participants with these conditions.", "spanType": "span-simple-text" } ], "parseErrors": [] } ], "parseErrors": [] } ], "parseErrors": [] } ], "type": "article", "title": "Why randomized controlled trials matter and the procedures that strengthen them", "authors": [ "Saloni Dattani" ], "excerpt": "Randomized controlled trials are a key tool to study cause and effect. Why do they matter and how do they work?", "dateline": "February 10, 2022", "subtitle": "Randomized controlled trials are a key tool to study cause and effect. Why do they matter and how do they work?", "sidebar-toc": false, "featured-image": "RCT-thumbnail-1.jpg" }, "createdAt": "2022-02-07T11:56:11.000Z", "published": false, "updatedAt": "2023-04-12T12:57:29.000Z", "revisionId": null, "publishedAt": "2022-02-10T11:00:00.000Z", "relatedCharts": [], "publicationContext": "listed" }	{ "errors": [ { "name": "unexpected wp component tag", "details": "Found unhandled wp:comment tag image" }, { "name": "unexpected wp component tag", "details": "Found unhandled wp:comment tag image" }, { "name": "unhandled html tag found", "details": "Encountered the unhandled tag hr" }, { "name": "unexpected wp component tag", "details": "Found unhandled wp:comment tag separator" }, { "name": "unhandled html tag found", "details": "Encountered the unhandled tag hr" }, { "name": "unexpected wp component tag", "details": "Found unhandled wp:comment tag separator" } ], "numBlocks": 112, "numErrors": 6, "wpTagCounts": { "html": 5, "image": 2, "column": 12, "columns": 6, "heading": 13, "paragraph": 116, "separator": 2, "owid/summary": 1, "owid/prominent-link": 2, "owid/additional-information": 1 }, "htmlTagCounts": { "p": 116, "h3": 2, "h4": 10, "h5": 1, "hr": 2, "div": 18, "figure": 2, "iframe": 5 } }	2022-02-10 11:00:00	2024-02-16 14:22:53	1Ugd_lwpRxqPgf1-l4X0TWnJUBC7V-cYJEJTJWTj7KfU	[ "Saloni Dattani" ]	Randomized controlled trials are a key tool to study cause and effect. Why do they matter and how do they work?	2022-02-07 11:56:11	2023-04-12 12:57:29	https://ourworldindata.org/wp-content/uploads/2022/02/RCT-thumbnail-1.jpg	{}	<Callout title="Summary"/> At Our World in Data, we bring attention to the world's largest problems. We explore what these problems are, why they matter and how large they are. Whenever possible, we try to explain why these problems exist, and how we might solve them. To make progress, we need to be able to identify real solutions and evaluate them carefully. But doing this well is not simple. It's difficult for scientists to collect good evidence on the costs and benefits of a new idea, policy, or treatment. And it's challenging for decision-makers to scrutinize the evidence that exists, to make better decisions. What we need are reliable ways to distinguish between ideas that work and those that don't. In this post, I will explain a crucial tool that helps us do this – randomized controlled trials (RCTs). We will see that RCTs matter for three reasons: when we don't know about the effects of interventions, when we don't know how to study them, and when scientific research is affected by biases. ### What are randomized controlled trials? To begin with, what are RCTs? These are experiments where people are given, at random, either an intervention (such as a drug) or a control and then followed up to see how they fare on various outcomes. RCTs are conducted by researchers around the world. In the map, you can see how many RCTs have ever been published in high-ranked medical journals, by the country where the first author was based. Over 18,000 of these were from the United States, but most countries have had fewer than a hundred.{ref}Catalá-López, F., Aleixandre-Benavent, R., Caulley, L., Hutton, B., Tabarés-Seisdedos, R., Moher, D., & Alonso-Arroyo, A. (2020). Global mapping of randomised trials related articles published in high-impact-factor medical journals: a cross-sectional analysis. _Trials_, _21_(1), 1-24.{/ref} RCTs have also become more common over time.{ref}Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., & Otte, W. M. (2021). The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. _PLOS Biology_, _19_(4), e3001162. [https://doi.org/10.1371/journal.pbio.3001162](https://doi.org/10.1371/journal.pbio.3001162){/ref} It's easy to take RCTs for granted, but these trials have transformed our understanding of cause and effect. They are a powerful tool to illuminate what is unknown or uncertain; to discern whether something works and how well it works. But it's also important to recognize that these trials are not always perfect, and understand why. The strengths of RCTs are subtle: they are powerful because of the set of procedures that they are expected to follow. This includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration. In this post, we will explore why these procedures matter – how each one adds a layer of protection against complications that scientists face when they do research. <Chart url="https://ourworldindata.org/grapher/total-number-of-rcts-published-in-high-ranked-medical-journals?tab=map&country=GBR~USA~IND~DNK~NGA~BOL~MEX~FRA~JPN~CHN"/> ### The fundamental problem of causal inference We make decisions based on our understanding of how things work – we try to predict the consequences of our actions. But understanding cause and effect is not just crucial for personal decisions: our knowledge of what works can have large consequences for people around us. An example is antiretroviral therapy (ART), which is used to treat [HIV/AIDS](http://ourworldindata.org/hiv-aids). The benefits of these drugs were surprising. One of the first ART drugs discovered was azidothymidine, which had previously been abandoned as an ineffective treatment for cancer.{ref}National Research Council (U.S.) (1993). The social impact of AIDS in the United States. 4. Clinical Research and Drug Regulation. _National Academy Press._{/ref} The discovery that azidothymidine and other antiretroviral therapies worked, and their use worldwide, has prevented millions of deaths from HIV/AIDS, as the chart shows. Discovering what works can save lives. But discovering what _doesn’t_ work can do the same. It means we can redirect our time and resources away from things that don't work, towards things that do. Even when we already know that something works, understanding how _well_ it works can help us make better decisions. An example for this is [the BCG vaccine](https://ourworldindata.org/grapher/bcg-immunization-coverage-for-tb-among-1-year-olds?country=~OWID_WRL), which reduces the risk of tuberculosis. The efficacy of this vaccine is different for different people around the world, and the reasons for this are unclear.{ref}Dockrell, H. M., & Smith, S. G. (2017). What Have We Learnt about BCG Vaccination in the Last 20 Years? _Frontiers in Immunology_, _8_, 1134. [https://doi.org/10.3389/fimmu.2017.01134](https://doi.org/10.3389/fimmu.2017.01134) Mangtani, P., Abubakar, I., Ariti, C., Beynon, R., Pimpin, L., Fine, P. E. M., Rodrigues, L. C., Smith, P. G., Lipman, M., Whiting, P. F., & Sterne, J. A. (2014). Protection by BCG Vaccine Against Tuberculosis: A Systematic Review of Randomized Controlled Trials. _Clinical Infectious Diseases_, _58_(4), 470–480. [https://doi.org/10.1093/cid/cit790](https://doi.org/10.1093/cid/cit790%7B/ref){/ref} Still, knowing this is informative, because it tells us that more effort is needed to protect people against the disease in places where the benefit of the vaccine is low. If there was a reliable way to know about the effects of the measures we took, we could prioritize solutions that are most effective. So, how would we understand cause and effect without trials? One way is through observation: we can observe what different people did and track their outcomes afterwards. But it's possible that the events that followed their actions were simply a coincidence, or that they would have happened anyway. The biggest challenge in trying to understand causes and effects is that we are only able to see one version of history. When someone makes a particular decision, we are able to see what follows, but we cannot see what would have happened if they had made a different decision. This is known as “the fundamental problem of causal inference."{ref}Holland, P. W. (1986). Statistics and Causal Inference. _Journal of the American Statistical Association_, _81_(396), 945–960. [https://doi.org/10.1080/01621459.1986.10478354](https://doi.org/10.1080/01621459.1986.10478354) Imbens, G. W., & Rubin, D. B. (2010). Rubin causal model. In _Microeconometrics_ (pp. 229–241). Springer. Rubin, D. B. (1977). Assignment to Treatment Group on the Basis of a Covariate. _Journal of Educational Statistics_, _2_(1), 1–26. [https://doi.org/10.3102/10769986002001001](https://doi.org/10.3102/10769986002001001%7B/ref){/ref} What this means is that it is impossible to predict the effects that an action will have for an individual person, but we can try to predict the effects it would have on average. <Chart url="https://ourworldindata.org/grapher/hivaids-deaths-and-averted-due-to-art?country=~OWID_WRL"/> ### Why randomized controlled trials matter Sometimes we do not need an RCT to identify causes and effects. The effect of smoking on lung cancer is one example, where scientists could be confident as early as the 1960s that the large increase in lung cancer rates could not have been caused by other factors. This was because there was already knowledge from many different lines of evidence.{ref}Hill, G., Millar, W., & Connelly, J. (2003). “The Great Debate”: Smoking, Lung Cancer, and Cancer Epidemiology. _Canadian Bulletin of Medical History_, _20_(2), 367-386.{/ref} Experiments, biopsies and population studies all showed that cigarette smoke was associated with specific changes in lung tissue and with the incidence of lung cancer. The association was so large and consistent that it could not be explained by other factors.{ref}Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B., & Wynder, E. L. (1959). Smoking and Lung Cancer: Recent Evidence and a Discussion of Some Questions. _JNCI: Journal of the National Cancer Institute_. [https://doi.org/10.1093/jnci/22.1.173](https://doi.org/10.1093/jnci/22.1.173){/ref} Even now, smoking is estimated to cause a large proportion of cancer deaths in many countries, as you can see in the chart. But in other situations, RCTs have made a huge difference to our understanding of the world. Prioritizing the drugs that were shown to be effective in trials saved lives, as we saw with the example of antiretroviral therapy to treat HIV/AIDS. People sometimes refer to RCTs as the "gold standard of evidence", but it's more useful to recognize that their strengths emerge from a _set of procedures_ that they aim to follow. <Chart url="https://ourworldindata.org/grapher/share-of-cancer-deaths-attributed-to-tobacco"/> ### Why do RCTs matter? In my opinion, even though we do have lots of knowledge about various topics, these trials matter for three reasons. They matter because we may not know enough, because we can be wrong, and because we might see what we want to see. First, they matter when _we don't know_. Randomized controlled trials can illuminate our understanding of effects, especially when there is uncertainty around them. They can help resolve disagreements between experts. In some cases, scientists can use other methods to investigate these topics. But when there is insufficient knowledge, other methods may not be enough. This is because, in a typical study, scientists need to be able to account for the possibility that other factors are causing the outcome. They need to have knowledge about why an effect usually occurs and whether it will happen anyway. They need to know about which other risk factors can cause the outcome and what makes it likely that someone will receive the treatment. They also need to know how to measure these factors properly, and how to account for them. The second reason they matter is when _we are wrong_. Even when scientists think they know which risk factors affect the outcome, they might be incorrect. They might not account for the right risk factors or they might account for the wrong ones. In an RCT where scientists use randomization, blinding and concealment, they can minimize both of these problems. Despite people's risks before the trial, the reason that someone will receive the treatment is the random number they are given. The reason that participants in each group differ at the end of the study is the group they were randomized to. Even if we don't know about other risk factors, or even if we are wrong about them, these procedures mean that we can still find out whether a treatment has an effect and how large the effect is. The third reason is when _we see what we want to see_. When participants expect to feel different after a treatment, they might report that they feel different, even if they didn't receive the treatment at all. When scientists want to see people improve after a treatment, they might decide to allocate the treatment to healthier participants. They might measure their data differently or re-analyze their data until they can find an improvement. They might even decide against publishing their findings when they don't find a benefit. If scientists use concealment, blinding, and pre-registration, they can reduce these biases. They can protect research against their own prejudices and the expectations of participants. They can also defend scientific knowledge from poor incentives, such as the desires of scientists to hype up their findings and the incentives for pharmaceutical companies to claim that their drugs work. ## The layers of protection against bias Previously, I described how the strengths of RCTs emerge from the _set of procedures_ that they aim to follow. In medicine, this commonly includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.{ref}Although these procedures are also used outside of medicine, it can be difficult to apply them elsewhere. For example, in a trial that tests the effectiveness of talking therapy, it would be known to the participants that they are receiving it; it may not be possible to find a placebo control version to disguise the procedure. Due to constraints in length and focus, I will not detail the advantages of intention-to-treat analysis or experimentation.{/ref} Each of these helps to protect the evidence against distortions, which can come from many sources. But they are not always enforced. Below, we will explore them in more detail. ### The control group gives us a comparison to see what would have happened otherwise The most crucial element to understand causes and effects is a control group. Let's use antidepressants as an example to see why we need them. If no one in the world received antidepressants, we wouldn’t know what their effects were. But the opposite is also true. If _everyone_ received antidepressants, we wouldn't know how things would be without them. In order to understand the effect of an antidepressant, we need to see which other outcomes are possible. The most important thing we need to understand effects is a _comparison_ – we need some people who receive the treatment and some who don’t. The people who don’t could be our control group. An ideal control group should allow us to _control_ the biases and measurement errors in a study completely. For example, different doctors may diagnose depression differently. If someone had symptoms of depression, then the chances they get diagnosed should be equal in the antidepressant group and the control group. If they weren't equal, we could mistake differences between the groups for an effect of antidepressants, even if they didn't have any effect. In fact, an ideal control group should allow us to control the total effects of _all of the other factors_ that could affect people's mood, not just control for the biases and errors in studies. As an example, we know that the symptoms of depression tend to change over time, as I explained in [this earlier post](https://ourworldindata.org/what-is-depression). You can see this in the chart. This shows that the symptoms of depression tend to decline over time, among people who are diagnosed with depression but not treated for it. This is measured by seeing how many patients would still meet the threshold for a diagnosis of depression later on. Almost a quarter of patients with depression (23%) would no longer meet the threshold for depression after three months, despite receiving no treatment. Just over half (53%) would no longer meet the threshold after one year.{ref}Whiteford, H. A., Harris, M. G., McKeon, G., Baxter, A., Pennell, C., Barendregt, J. J., & Wang, J. (2013). Estimating remission from untreated major depression: A systematic review and meta-analysis. _Psychological Medicine_, _43_(8), 1569–1585. [https://doi.org/10.1017/S0033291712001717](https://doi.org/10.1017/S0033291712001717){/ref} This change is known as "regression to the mean." If we didn’t have a control group in our study, we might misattribute such an improvement to the antidepressant. We need a control group to know how much their symptoms would improve anyway. <Image filename="khK-H11avMDGkTpCLAqCJIsIyAJSX63_Blm-15GL-0fkGYoDK-47Q1Ancs-i06algU5S3GRWTTeL4hNMQgfezt2_DlxwUm1kTykSx6oc3Unzbnz0UNaKUbGZdQF4fagJFIwrXvCu" alt=""/> ### Placebos allow us to account for placebo effects Some types of controls are special because they resemble the treatment without actually being it – these controls are called placebos. An ideal placebo has all the qualities above and also allows us to account for "placebo effects." In this case, the placebo effect refers to when people's moods improve from the mere _procedure_ of receiving the antidepressant. For example, taking a pill might improve people's mood because they believe that taking a pill will give them some benefit, even if the pill does not actually contain any active ingredient. How large is the placebo effect? Some clinical trials have tried to estimate this by comparing patients who received a placebo to patients who received no treatment at all. Overall, these studies have found that placebo effects are small in clinical trials for many conditions, but are larger for physical treatments, such as acupuncture to treat pain.{ref}Hróbjartsson, A., & Gøtzsche, P. C. (2010). Placebo interventions for all clinical conditions. _Cochrane Database of Systematic Reviews_. https://doi.org/10.1002/14651858.CD003974.pub3 Hróbjartsson, A., & Gøtzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. _New England Journal of Medicine_, _344_(21), 1594–1602. [https://doi.org/10.1056/NEJM200105243442106](https://doi.org/10.1056/NEJM200105243442106){/ref} Placebo effects tend to be larger when scientists are recording symptoms reported by patients, rather than when they are measuring something that can be verified by others, such as results from a blood test or death. For depression, the placebo effect is non-significant. Studies do not detect a difference in the moods of patients who receive a placebo and those who do not receive any treatment at all.{ref}Hróbjartsson, A., & Gøtzsche, P. C. (2010). Placebo interventions for all clinical conditions. _Cochrane Database of Systematic Reviews_. [https://doi.org/10.1002/14651858.CD003974.pub3](https://doi.org/10.1002/14651858.CD003974.pub3) Hróbjartsson, A., & Gøtzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. _New England Journal of Medicine_, _344_(21), 1594–1602. [https://doi.org/10.1056/NEJM200105243442106](https://doi.org/10.1056/NEJM200105243442106%7B/ref){/ref} Instead, the placebo group serves as a typical control group, to show what would happen to these patients even if they did not receive treatment. ### Randomization ensures that there are no differences between control and treatment group apart from whether they received the treatment Before participants are enrolled in a trial, they might already have different levels of risk of developing the outcomes. Let's look at an example to illustrate this point. Statins are a type of drug commonly used to prevent stroke. But strokes are more common among people who use statins. Does that mean that statins _caused_ an increase in the rates of stroke? No – people who are prescribed statins are more likely to have cardiovascular problems to begin with, which increases the chances of having a stroke later on. When this is accounted for, researchers find that people who take statins are actually _less likely_ to develop a stroke.{ref}Orkaby, A. R., Gaziano, J. M., Djousse, L., & Driver, J. A. (2017). Statins for Primary Prevention of Cardiovascular Events and Mortality in Older Men. _Journal of the American Geriatrics Society_, _65_(11), 2362–2368. [https://doi.org/10.1111/jgs.14993](https://doi.org/10.1111/jgs.14993) Makihara, N., Kamouchi, M., Hata, J., Matsuo, R., Ago, T., Kuroda, J., Kuwashiro, T., Sugimori, H., & Kitazono, T. (2013). Statins and the risks of stroke recurrence and death after ischemic stroke: The Fukuoka Stroke Registry. _Atherosclerosis_, _231_(2), 211–215. [https://doi.org/10.1016/j.atherosclerosis.2013.09.017](https://doi.org/10.1016/j.atherosclerosis.2013.09.017) Ní Chróinín, D., Asplund, K., Åsberg, S., Callaly, E., Cuadrado-Godia, E., Díez-Tejedor, E., Di Napoli, M., Engelter, S. T., Furie, K. L., Giannopoulos, S., Gotto, A. M., Hannon, N., Jonsson, F., Kapral, M. K., Martí-Fàbregas, J., Martínez-Sánchez, P., Milionis, H. J., Montaner, J., Muscari, A., … Kelly, P. J. (2013). Statin Therapy and Outcome After Ischemic Stroke: Systematic Review and Meta-Analysis of Observational Studies and Randomized Trials. _Stroke_, _44_(2), 448–456. [https://doi.org/10.1161/STROKEAHA.112.668277](https://doi.org/10.1161/STROKEAHA.112.668277){/ref} If researchers simply compared the rates of stroke in those who used statins with those who did not, they would miss the fact that there were other differences between the two groups, which could have caused differences in their rates of stroke. Important differences such as these – which affect people's likelihood of receiving the treatment (using statins) _and also_ affect the outcome (the risk of a stroke) – are called ‘confounders’. But it can be difficult to know what all of these confounders might be. It can also be difficult to measure and account for them properly. In fact, scientists can actually _worsen_ a study by accounting for the wrong factors.{ref}Cinelli, C., Forney, A., & Pearl, J. (2020). A crash course in good and bad controls. _Available at SSRN, 3689437:_[http://dx.doi.org/10.2139/ssrn.3689437](https://dx.doi.org/10.2139/ssrn.3689437){/ref} What happens when participants are randomized in a trial? Randomization is the procedure of allocating them into one of two groups at random. With randomization, the problems above are minimized: everyone has the possibility of receiving the treatment. Whether people receive the treatment is not determined by the risks they have, but whether they are randomly selected to receive the treatment. So, the overall risks of developing the outcome in one group become comparable to the risks in the other group. Randomization means that it is not a problem when there are confounders that are not known or not measured. Researchers don't have to know about why or how the outcome usually occurs.{ref}Aronow, P., Robins, J. M., Saarinen, T., Sävje, F., & Sekhon, J. (2021). Nonparametric identification is not enough, but randomized controlled trials are. _ArXiv Preprint ArXiv:2108.11342_.{/ref} ### Concealment and blinding limit the biases of researchers and expectations of participants In a clinical trial, participants or scientists might realize which groups they are assigned to. For example, the drug might smell, taste or look different from the placebo. Or it might have obvious benefits or different side effects compared to the placebo.{ref}Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., Cochrane Bias Methods Group, & Cochrane Statistical Methods Group. (2011). The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. _BMJ_, _343_(oct18 2), d5928–d5928.[ https://doi.org/10.1136/bmj.d5928](https://doi.org/10.1136/bmj.d5928){/ref} Concealment is a first step in preventing this: this is the procedure of preventing scientists from knowing which treatment people _will be_ assigned to. Blinding is a second step: this is the procedure of preventing participants and scientists from finding out which treatment group people _have been_ assigned to.{ref}Schulz, K. F., Chalmers, I., & Altman, D. G. (2002). The Landscape and Lexicon of Blinding in Randomized Trials. _Annals of Internal Medicine_, _136_(3), 254. [https://doi.org/10.7326/0003-4819-136-3-200202050-00022](https://doi.org/10.7326/0003-4819-136-3-200202050-00022){/ref} When blinding is incomplete, it can partly reverse the benefits of randomization. For example, in clinical trials for oral health, the benefits of treatments appear larger when patients and the scientists who assess their health are not blinded sufficiently.{ref}Saltaji, H. _et al._ Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions. _BMC Med Res Methodol_18, 42 (2018).{/ref} If randomization was maintained, the only reason that groups would differ on their outcomes was the treatment they received. However, if the treatments were not hidden from scientists and participants, other factors could cause differences between them. Sometimes, blinding is not possible – there might not be something that is safe and closely resembles the treatment, which could be used as a placebo. Fortunately, this does not necessarily mean that these trials cannot be useful. Researchers can measure verifiable outcomes (such as changes in blood levels or even deaths) to avoid some placebo effects. But even when blinding occurs, participants and researchers might still make different decisions, because of the effects of the treatment or placebo. For example, some participants might decide to withdraw from the trial or not follow the protocols of the trial closely. Similarly, scientists might guess which groups people are in and therefore treat them differently or measure their outcomes in a biased way. It's difficult to predict how this might affect the results of a trial. It could cause us to overestimate or underestimate the benefit of a treatment. For example, in the clinical trials for [Covid-19 vaccines](https://ourworldindata.org/covid-vaccinations), some participants may have guessed that they received the vaccine because they experienced side effects. So, they may have believed that they were more protected and took fewer precautions. This means they may have increased their risk of catching Covid-19. This would make it appear as if the vaccines gave less protection than they actually did: it would result in an underestimate of their efficacy. ### Preregistration allows us to hold researchers accountable to their original study plans Some of the procedures we've explored so far are used to safeguard research against errors and biases that scientists can have. Pre-registration is another procedure that contributes to the same goal. After the data in a study is analyzed, scientists have some choice in which results they present to their colleagues and the wider community. This opens up research to the possibility of cherry-picking. This problem unfortunately often arises in studies that are sponsored by industry. If a pharmaceutical company is testing their new drugs in trials, disappointing results can lead to financial losses. So, they may decide not to publish them. But this problem is not limited to trials conducted by pharmaceutical companies. For many reasons, scientists may decide not to publish some of their studies. Or they might re-analyze their data in a different way to find more positive results. Even if scientists want to publish their findings, journals may decide not to publish them because they may be seen as disappointing, controversial or uninteresting. To counter these incentives, scientists can follow the practice of "pre-registration." This is when they publicly declare which analyses they plan to do in advance of collecting the data. In 2000, the United States Food and Drug Administration (FDA) established an online registry for the details of clinical trials. The FDA required that scientists who were studying therapies for serious diseases needed to provide some details of their study designs and the outcomes that they planned to investigate.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. _Jama_, _307_(17), 1861–1864.{/ref} This requirement reduced the bias towards positive findings in published research. We see evidence of this in the chart, showing clinical trials funded by the National Heart, Lung and Blood Institute (NHLBI), which adopted this requirement. Before 2000 – when pre-registration of studies was not required – most candidate drugs to treat or prevent cardiovascular disease showed large benefits. But most trials published after 2000 showed no benefit.{ref}Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. _PLOS ONE_, _10_(8), e0132382. [https://doi.org/10.1371/journal.pone.0132382](https://doi.org/10.1371/journal.pone.0132382){/ref} <Image filename="Efficacy-in-trials-before-and-after-registration-requirement2.jpg" alt=""/> Over the last two decades, this practice was strengthened and expanded. In 2007, the FDA required that most approved trials must be registered when people are enrolled into the study. They introduced notices and civil penalties for not doing so. Now, similar requirements are in place in Europe, Canada and some other parts of the world.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. _Jama_, _307_(17), 1861–1864. [https://doi.org/10.1001/jama.2012.4230](https://doi.org/10.1001/jama.2012.4230){/ref} Importantly, sponsors of clinical trials are required to share the results online after the trial is completed. But unfortunately many still violate these requirements. For example, less than half (40%) of trials that were conducted in the US since 2017 actually reported their results on time.{ref}DeVito, N. J., & Goldacre, B. (2021). Evaluation of Compliance With Legal Requirements Under the FDA Amendments Act of 2007 for Timely Registration of Clinical Trials, Data Verification, Delayed Reporting, and Trial Document Submission. JAMA Internal Medicine, 181(8), 1128. [https://doi.org/10.1001/jamainternmed.2021.2036](https://doi.org/10.1001/jamainternmed.2021.2036){/ref} Here, you can see patterns in reporting results in Europe. According to EU regulations, all clinical trials in the European Economic Area are required to report their results to the European Clinical Trials Register (EU-CTR) within a year of completing the trial. But reporting rates vary a lot by country. By January 2021, most clinical trials by non-commercial sponsors in the UK (94%) reported their results in time, while nearly none (4%) in France had.{ref}Note: Data is shown until January 2021 for trials. After the UK left the European Union in January 2021, clinical trials in the UK were then no longer required to report their results to the EU-CTR. Only data from trials by non-commercial sponsors is shown. This includes trials sponsored by institutions such as universities, hospitals, research foundations and so on. Dal-Ré, R., Goldacre, B., Mahillo-Fernández, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. _Journal of Clinical Epidemiology_, S0895435621003577. [https://doi.org/10.1016/j.jclinepi.2021.11.005](https://doi.org/10.1016/j.jclinepi.2021.11.005){/ref} Although a large share of clinical trials fail to report their results when they are due, this has improved recently: only half (50%) of clinical trials by non-commercial sponsors reported their results in time in 2018, while more than two-thirds (69%) did in 2021.{ref}Dal-Ré, R., Goldacre, B., Mahillo-Fernández, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. _Journal of Clinical Epidemiology_, S0895435621003577. [https://doi.org/10.1016/j.jclinepi.2021.11.005](https://doi.org/10.1016/j.jclinepi.2021.11.005){/ref} <Chart url="https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-in-time-by-country?country=GBR~FRA~BEL~NLD~DEU~ITA~ESP~SWE~AUT~DNK"/> <Chart url="https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-within-a-year-over-time?country=~European+Economic+Area"/> ### Pulling the layers together Together, these procedures give us a far more reliable way to distinguish between what works and what doesn't – even when we don't know enough, when we're wrong and when we see what we want to see. Unfortunately, we have seen that many clinical trials do not follow them. They tend not to report the details of their methods or test whether their participants were blinded to what they received. These standards cannot just be expected from scientists – they require cooperation from funders and journals, and they need to be actively enforced with penalties and incentives. We've also seen that trials can still suffer from remaining problems: that participants may drop out of the study and not adhere to the treatment. And clinical trials tend to not share their data or the code that they used to analyze it. So, despite all the benefits they provide, we shouldn't see these layers as a fixed checklist. Just as some of them were introduced recently, they may still evolve in the future – open data sharing, registered reports and procedures to reduce dropouts may be next on the list.{ref}Chambers, C., & Tzavella, L. (2020). The past, present, and future of Registered Reports.{/ref} The procedures used in these trials are not a silver bullet. But when they are upheld, they can make these trials a more powerful source of evidence for understanding cause and effect. To make progress, we need to be able to understand the problems that affect us, their causes and solutions. Randomized controlled trials can give scientists a reliable way to collect evidence on these important questions, and give us the ability to make better decisions. At Our World in Data, this is why they matter most of all. #### _Keep reading on Our World in Data_ ### https://ourworldindata.org/what-is-depression ### https://ourworldindata.org/mental-health Acknowledgements: I would like to thank Darren Dahly, Nathaniel Bechhofer, Hannah Ritchie and Max Roser for reading drafts of this post and their very helpful guidance and suggestions to improve it. ## Additional information Trials are useful to test new ideas or treatments, when people do not have enough knowledge to recommend them on a larger scale. But people also have ethical concerns surrounding trials of treatments that are already used by doctors. For one, they might believe that it isn't justified to give patients a placebo when a drug is available, even if it is unclear how well the drug works. How can we balance these concerns with the benefits of trials? When are trials important? A leading view is that they are justified when there is disagreement or uncertainty about the benefits of an intervention.{ref}Wendler, D. The Ethics of Clinical Research. in The Stanford Encyclopedia of Philosophy (ed. Zalta, E. N.) (Metaphysics Research Lab, Stanford University, 2021). London, A. J. Equipoise in Research: Integrating Ethics and Science in Human Research. JAMA 317, 525 (2017).{/ref} Outside of a trial, some doctors might prescribe a particular treatment, while other doctors would not, because they believe that same treatment is ineffective. In this case, the options in a trial – an intervention or control – would not be so different from what people could already encounter. When experts disagree on a topic, an RCT is useful because it can resolve these conflicts and inform us about which option is better. So, to make the most of trials, they should be planned in line with the questions that people have. For example, if we wanted to understand whether vaccines gave less protection to people who were immunocompromised, then we should plan a trial with enough participants with these conditions.	{ "id": 48378, "date": "2022-02-10T11:00:00", "guid": { "rendered": "https://owid.cloud/?p=48378" }, "link": "https://owid.cloud/randomized-controlled-trials", "meta": { "owid_publication_context_meta_field": { "latest": true, "homepage": true, "immediate_newsletter": true } }, "slug": "randomized-controlled-trials", "tags": [], "type": "post", "title": { "rendered": "Why randomized controlled trials matter and the procedures that strengthen them" }, "_links": { "self": [ { "href": "https://owid.cloud/wp-json/wp/v2/posts/48378" } ], "about": [ { "href": "https://owid.cloud/wp-json/wp/v2/types/post" } ], "author": [ { "href": "https://owid.cloud/wp-json/wp/v2/users/47", "embeddable": true } ], "curies": [ { "href": "https://api.w.org/{rel}", "name": "wp", "templated": true } ], "replies": [ { "href": "https://owid.cloud/wp-json/wp/v2/comments?post=48378", "embeddable": true } ], "wp:term": [ { "href": "https://owid.cloud/wp-json/wp/v2/categories?post=48378", "taxonomy": "category", "embeddable": true }, { "href": "https://owid.cloud/wp-json/wp/v2/tags?post=48378", "taxonomy": "post_tag", "embeddable": true } ], "collection": [ { "href": "https://owid.cloud/wp-json/wp/v2/posts" } ], "wp:attachment": [ { "href": "https://owid.cloud/wp-json/wp/v2/media?parent=48378" } ], "version-history": [ { "href": "https://owid.cloud/wp-json/wp/v2/posts/48378/revisions", "count": 29 } ], "wp:featuredmedia": [ { "href": "https://owid.cloud/wp-json/wp/v2/media/48379", "embeddable": true } ], "predecessor-version": [ { "id": 56713, "href": "https://owid.cloud/wp-json/wp/v2/posts/48378/revisions/56713" } ] }, "author": 47, "format": "standard", "status": "publish", "sticky": false, "content": { "rendered": "\t<div class=\"wp-block-owid-summary\">\n\t\t<h2>Summary</h2>\n\t\t\n\n<p>In order to tackle the world’s largest problems, we need to be able to identify their causes and effective ways to solve them. How would we know about the effects of a new idea, treatment or policy? Answering this question can be difficult. Some research is poorly done and there are incentives for scientists and funders to exaggerate their findings.</p>\n\n\n\n<p>In this post, I look at a key tool to study causal effects reliably: randomized controlled trials (RCTs). What makes RCTs useful is the set of procedures they are expected to follow, such as placebo controls, concealment, randomization and pre-registration. These procedures are layers of protection against challenges that scientists usually face in research. </p>\n\n\n\n<p>To be able to read published research, we need to understand why these procedures matter and why they are not always followed in practice. With this knowledge, we have a greater ability to identify the causes and solutions to the largest problems that affect us.</p>\n\n\n\t</div>\n\n\n<p>At Our World in Data, we bring attention to the world’s largest problems. We explore what these problems are, why they matter and how large they are. Whenever possible, we try to explain why these problems exist, and how we might solve them.</p>\n\n\n\n<p>To make progress, we need to be able to identify real solutions and evaluate them carefully. But doing this well is not simple. It’s difficult for scientists to collect good evidence on the costs and benefits of a new idea, policy, or treatment. And it’s challenging for decision-makers to scrutinize the evidence that exists, to make better decisions.</p>\n\n\n\n<p>What we need are reliable ways to distinguish between ideas that work and those that don’t.</p>\n\n\n\n<p>In this post, I will explain a crucial tool that helps us do this \u2013 randomized controlled trials (RCTs). We will see that RCTs matter for three reasons: when we don’t know about the effects of interventions, when we don’t know how to study them, and when scientific research is affected by biases.</p>\n\n\n\n<h4>What are randomized controlled trials?</h4>\n\n\n\n<p>To begin with, what are RCTs? These are experiments where people are given, at random, either an intervention (such as a drug) or a control and then followed up to see how they fare on various outcomes.</p>\n\n\n\n<p>RCTs are conducted by researchers around the world. In the map, you can see how many RCTs have ever been published in high-ranked medical journals, by the country where the first author was based. Over 18,000 of these were from the United States, but most countries have had fewer than a hundred.{ref}Catal\u00e1-L\u00f3pez, F., Aleixandre-Benavent, R., Caulley, L., Hutton, B., Tabar\u00e9s-Seisdedos, R., Moher, D., & Alonso-Arroyo, A. (2020). Global mapping of randomised trials related articles published in high-impact-factor medical journals: a cross-sectional analysis. <em>Trials</em>, <em>21</em>(1), 1-24.{/ref} RCTs have also become more common over time.{ref}Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., & Otte, W. M. (2021). The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. <em>PLOS Biology</em>, <em>19</em>(4), e3001162. <a href=\"https://doi.org/10.1371/journal.pbio.3001162\">https://doi.org/10.1371/journal.pbio.3001162</a>{/ref}</p>\n\n\n\n<p>It’s easy to take RCTs for granted, but these trials have transformed our understanding of cause and effect. They are a powerful tool to illuminate what is unknown or uncertain; to discern whether something works and how well it works.</p>\n\n\n\n<p>But it’s also important to recognize that these trials are not always perfect, and understand why.</p>\n\n\n\n<p>The strengths of RCTs are subtle: they are powerful because of the set of procedures that they are expected to follow. This includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.</p>\n\n\n\n<p>In this post, we will explore why these procedures matter \u2013 how each one adds a layer of protection against complications that scientists face when they do research. </p>\n\n\n\n<iframe src=\"https://ourworldindata.org/grapher/total-number-of-rcts-published-in-high-ranked-medical-journals?tab=map&country=GBR~USA~IND~DNK~NGA~BOL~MEX~FRA~JPN~CHN\" loading=\"lazy\" style=\"width: 100%; height: 600px; border: 0px none;\"></iframe>\n\n\n\n<h4>The fundamental problem of causal inference</h4>\n\n\n\n<p>We make decisions based on our understanding of how things work \u2013 we try to predict the consequences of our actions. </p>\n\n\n\n<p>But understanding cause and effect is not just crucial for personal decisions: our knowledge of what works can have large consequences for people around us. </p>\n\n\n\n<p>An example is antiretroviral therapy (ART), which is used to treat <a href=\"http://ourworldindata.org/hiv-aids\">HIV/AIDS</a>. The benefits of these drugs were surprising. One of the first ART drugs discovered was azidothymidine, which had previously been abandoned as an ineffective treatment for cancer.{ref}National Research Council (U.S.) (1993). The social impact of AIDS in the United States. 4. Clinical Research and Drug Regulation. <em>National Academy Press.</em>{/ref} The discovery that azidothymidine and other antiretroviral therapies worked, and their use worldwide, has prevented millions of deaths from HIV/AIDS, as the chart shows.</p>\n\n\n\n<p>Discovering what works can save lives. But discovering what <em>doesn\u2019t</em> work can do the same. It means we can redirect our time and resources away from things that don’t work, towards things that do.</p>\n\n\n\n<p>Even when we already know that something works, understanding how <em>well</em> it works can help us make better decisions. </p>\n\n\n\n<p>An example for this is <a href=\"https://ourworldindata.org/grapher/bcg-immunization-coverage-for-tb-among-1-year-olds?country=~OWID_WRL\">the BCG vaccine</a>, which reduces the risk of tuberculosis. The efficacy of this vaccine is different for different people around the world, and the reasons for this are unclear.{ref}Dockrell, H. M., & Smith, S. G. (2017). What Have We Learnt about BCG Vaccination in the Last 20 Years? <em>Frontiers in Immunology</em>, <em>8</em>, 1134. <a href=\"https://doi.org/10.3389/fimmu.2017.01134\">https://doi.org/10.3389/fimmu.2017.01134</a> <br>Mangtani, P., Abubakar, I., Ariti, C., Beynon, R., Pimpin, L., Fine, P. E. M., Rodrigues, L. C., Smith, P. G., Lipman, M., Whiting, P. F., & Sterne, J. A. (2014). Protection by BCG Vaccine Against Tuberculosis: A Systematic Review of Randomized Controlled Trials. <em>Clinical Infectious Diseases</em>, <em>58</em>(4), 470\u2013480. <a href=\"https://doi.org/10.1093/cid/cit790%7B/ref\">https://doi.org/10.1093/cid/cit790</a>{/ref} Still, knowing this is informative, because it tells us that more effort is needed to protect people against the disease in places where the benefit of the vaccine is low.</p>\n\n\n\n<p>If there was a reliable way to know about the effects of the measures we took, we could prioritize solutions that are most effective.</p>\n\n\n\n<p>So, how would we understand cause and effect without trials?</p>\n\n\n\n<p>One way is through observation: we can observe what different people did and track their outcomes afterwards. But it’s possible that the events that followed their actions were simply a coincidence, or that they would have happened anyway.</p>\n\n\n\n<p>The biggest challenge in trying to understand causes and effects is that we are only able to see one version of history. </p>\n\n\n\n<p>When someone makes a particular decision, we are able to see what follows, but we cannot see what would have happened if they had made a different decision. This is known as \u201cthe fundamental problem of causal inference.”{ref}Holland, P. W. (1986). Statistics and Causal Inference. <em>Journal of the American Statistical Association</em>, <em>81</em>(396), 945\u2013960. <a href=\"https://doi.org/10.1080/01621459.1986.10478354\">https://doi.org/10.1080/01621459.1986.10478354</a><br>Imbens, G. W., & Rubin, D. B. (2010). Rubin causal model. In <em>Microeconometrics</em> (pp. 229\u2013241). Springer.<br>Rubin, D. B. (1977). Assignment to Treatment Group on the Basis of a Covariate. <em>Journal of Educational Statistics</em>, <em>2</em>(1), 1\u201326. <a href=\"https://doi.org/10.3102/10769986002001001%7B/ref\">https://doi.org/10.3102/10769986002001001</a>{/ref}</p>\n\n\n\n<p>What this means is that it is impossible to predict the effects that an action will have for an individual person, but we can try to predict the effects it would have on average.</p>\n\n\n\n<iframe src=\"https://ourworldindata.org/grapher/hivaids-deaths-and-averted-due-to-art?country=~OWID_WRL\" loading=\"lazy\" style=\"width: 100%; height: 600px; border: 0px none;\"></iframe>\n\n\n\n<h4>Why randomized controlled trials matter</h4>\n\n\n\n<div class=\"wp-block-columns\">\n<div class=\"wp-block-column\">\n<p>Sometimes we do not need an RCT to identify causes and effects. The effect of smoking on lung cancer is one example, where scientists could be confident as early as the 1960s that the large increase in lung cancer rates could not have been caused by other factors. </p>\n\n\n\n<p>This was because there was already knowledge from many different lines of evidence.{ref}Hill, G., Millar, W., & Connelly, J. (2003). \u201cThe Great Debate\u201d: Smoking, Lung Cancer, and Cancer Epidemiology. <em>Canadian Bulletin of Medical History</em>, <em>20</em>(2), 367-386.{/ref} Experiments, biopsies and population studies all showed that cigarette smoke was associated with specific changes in lung tissue and with the incidence of lung cancer. The association was so large and consistent that it could not be explained by other factors.{ref}Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B., & Wynder, E. L. (1959). Smoking and Lung Cancer: Recent Evidence and a Discussion of Some Questions. <em>JNCI: Journal of the National Cancer Institute</em>. <a href=\"https://doi.org/10.1093/jnci/22.1.173\">https://doi.org/10.1093/jnci/22.1.173</a>{/ref}</p>\n\n\n\n<p>Even now, smoking is estimated to cause a large proportion of cancer deaths in many countries, as you can see in the chart.</p>\n\n\n\n<p>But in other situations, RCTs have made a huge difference to our understanding of the world. Prioritizing the drugs that were shown to be effective in trials saved lives, as we saw with the example of antiretroviral therapy to treat HIV/AIDS.</p>\n\n\n\n<p>People sometimes refer to RCTs as the “gold standard of evidence”, but it’s more useful to recognize that their strengths emerge from a <em>set of procedures</em> that they aim to follow.</p>\n</div>\n\n\n\n<div class=\"wp-block-column\">\n<iframe src=\"https://ourworldindata.org/grapher/share-of-cancer-deaths-attributed-to-tobacco\" loading=\"lazy\" style=\"width: 100%; height: 600px; border: 0px none;\"></iframe>\n</div>\n</div>\n\n\n\n<div class=\"wp-block-columns\">\n<div class=\"wp-block-column\">\n<h4>Why do RCTs matter? </h4>\n\n\n\n<p>In my opinion, even though we do have lots of knowledge about various topics, these trials matter for three reasons. They matter because we may not know enough, because we can be wrong, and because we might see what we want to see.</p>\n\n\n\n<p>First, they matter when <em>we don’t know</em>. </p>\n\n\n\n<p>Randomized controlled trials can illuminate our understanding of effects, especially when there is uncertainty around them. They can help resolve disagreements between experts.</p>\n\n\n\n<p>In some cases, scientists can use other methods to investigate these topics. But when there is insufficient knowledge, other methods may not be enough. </p>\n\n\n\n<p>This is because, in a typical study, scientists need to be able to account for the possibility that other factors are causing the outcome. </p>\n\n\n\n<p>They need to have knowledge about why an effect usually occurs and whether it will happen anyway. They need to know about which other risk factors can cause the outcome and what makes it likely that someone will receive the treatment. They also need to know how to measure these factors properly, and how to account for them.</p>\n\n\n\n<p>The second reason they matter is when <em>we are wrong</em>. </p>\n\n\n\n<p>Even when scientists think they know which risk factors affect the outcome, they might be incorrect. They might not account for the right risk factors or they might account for the wrong ones.</p>\n\n\n\n<p>In an RCT where scientists use randomization, blinding and concealment, they can minimize both of these problems. Despite people’s risks before the trial, the reason that someone will receive the treatment is the random number they are given. The reason that participants in each group differ at the end of the study is the group they were randomized to. </p>\n\n\n\n<p>Even if we don’t know about other risk factors, or even if we are wrong about them, these procedures mean that we can still find out whether a treatment has an effect and how large the effect is.</p>\n\n\n\n<p>The third reason is when <em>we see what we want to see</em>. </p>\n\n\n\n<p>When participants expect to feel different after a treatment, they might report that they feel different, even if they didn’t receive the treatment at all. </p>\n\n\n\n<p>When scientists want to see people improve after a treatment, they might decide to allocate the treatment to healthier participants. They might measure their data differently or re-analyze their data until they can find an improvement. They might even decide against publishing their findings when they don’t find a benefit.</p>\n\n\n\n<p>If scientists use concealment, blinding, and pre-registration, they can reduce these biases. They can protect research against their own prejudices and the expectations of participants. They can also defend scientific knowledge from poor incentives, such as the desires of scientists to hype up their findings and the incentives for pharmaceutical companies to claim that their drugs work.</p>\n</div>\n\n\n\n<div class=\"wp-block-column\"></div>\n</div>\n\n\n\n<h3>The layers of protection against bias</h3>\n\n\n\n<p>Previously, I described how the strengths of RCTs emerge from the <em>set of procedures</em> that they aim to follow.</p>\n\n\n\n<p>In medicine, this commonly includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.{ref}Although these procedures are also used outside of medicine, it can be difficult to apply them elsewhere. For example, in a trial that tests the effectiveness of talking therapy, it would be known to the participants that they are receiving it; it may not be possible to find a placebo control version to disguise the procedure. Due to constraints in length and focus, I will not detail the advantages of intention-to-treat analysis or experimentation.{/ref} Each of these helps to protect the evidence against distortions, which can come from many sources. But they are not always enforced. Below, we will explore them in more detail.</p>\n\n\n\n<h4>The control group gives us a comparison to see what would have happened otherwise</h4>\n\n\n\n<p>The most crucial element to understand causes and effects is a control group. </p>\n\n\n\n<p>Let’s use antidepressants as an example to see why we need them.</p>\n\n\n\n<p>If no one in the world received antidepressants, we wouldn\u2019t know what their effects were. But the opposite is also true. If <em>everyone</em> received antidepressants, we wouldn’t know how things would be without them. </p>\n\n\n\n<p>In order to understand the effect of an antidepressant, we need to see which other outcomes are possible. The most important thing we need to understand effects is a <em>comparison</em> \u2013 we need some people who receive the treatment and some who don\u2019t. The people who don\u2019t could be our control group.</p>\n\n\n\n<p>An ideal control group should allow us to <em>control</em> the biases and measurement errors in a study completely. </p>\n\n\n\n<p>For example, different doctors may diagnose depression differently. If someone had symptoms of depression, then the chances they get diagnosed should be equal in the antidepressant group and the control group. If they weren’t equal, we could mistake differences between the groups for an effect of antidepressants, even if they didn’t have any effect.</p>\n\n\n\n<p>In fact, an ideal control group should allow us to control the total effects of <em>all of the other factors</em> that could affect people’s mood, not just control for the biases and errors in studies. </p>\n\n\n\n<p>As an example, we know that the symptoms of depression tend to change over time, as I explained in <a href=\"https://ourworldindata.org/what-is-depression\">this earlier post</a>. </p>\n\n\n\n<p>You can see this in the chart. This shows that the symptoms of depression tend to decline over time, among people who are diagnosed with depression but not treated for it. This is measured by seeing how many patients would still meet the threshold for a diagnosis of depression later on. </p>\n\n\n\n<p>Almost a quarter of patients with depression (23%) would no longer meet the threshold for depression after three months, despite receiving no treatment. Just over half (53%) would no longer meet the threshold after one year.{ref}Whiteford, H. A., Harris, M. G., McKeon, G., Baxter, A., Pennell, C., Barendregt, J. J., & Wang, J. (2013). Estimating remission from untreated major depression: A systematic review and meta-analysis. <em>Psychological Medicine</em>, <em>43</em>(8), 1569\u20131585. <a href=\"https://doi.org/10.1017/S0033291712001717\">https://doi.org/10.1017/S0033291712001717</a>{/ref} This change is known as “regression to the mean.”</p>\n\n\n\n<p>If we didn\u2019t have a control group in our study, we might misattribute such an improvement to the antidepressant. We need a control group to know how much their symptoms would improve anyway.</p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https://lh5.googleusercontent.com/khK-H11avMDGkTpCLAqCJIsIyAJSX63_Blm-15GL-0fkGYoDK-47Q1Ancs-i06algU5S3GRWTTeL4hNMQgfezt2_DlxwUm1kTykSx6oc3Unzbnz0UNaKUbGZdQF4fagJFIwrXvCu\" alt=\"\"/></figure>\n\n\n\n<h4>Placebos allow us to account for placebo effects</h4>\n\n\n\n<p>Some types of controls are special because they resemble the treatment without actually being it \u2013 these controls are called placebos. An ideal placebo has all the qualities above and also allows us to account for “placebo effects.” In this case, the placebo effect refers to when people’s moods improve from the mere <em>procedure</em> of receiving the antidepressant. </p>\n\n\n\n<p>For example, taking a pill might improve people’s mood because they believe that taking a pill will give them some benefit, even if the pill does not actually contain any active ingredient. </p>\n\n\n\n<p>How large is the placebo effect? </p>\n\n\n\n<p>Some clinical trials have tried to estimate this by comparing patients who received a placebo to patients who received no treatment at all. Overall, these studies have found that placebo effects are small in clinical trials for many conditions, but are larger for physical treatments, such as acupuncture to treat pain.{ref}Hr\u00f3bjartsson, A., & G\u00f8tzsche, P. C. (2010). Placebo interventions for all clinical conditions. <em>Cochrane Database of Systematic Reviews</em>. https://doi.org/10.1002/14651858.CD003974.pub3  <br>Hr\u00f3bjartsson, A., & G\u00f8tzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. <em>New England Journal of Medicine</em>, <em>344</em>(21), 1594\u20131602. <a href=\"https://doi.org/10.1056/NEJM200105243442106\">https://doi.org/10.1056/NEJM200105243442106</a>{/ref} </p>\n\n\n\n<p>Placebo effects tend to be larger when scientists are recording symptoms reported by patients, rather than when they are measuring something that can be verified by others, such as results from a blood test or death. </p>\n\n\n\n<p>For depression, the placebo effect is non-significant. Studies do not detect a difference in the moods of patients who receive a placebo and those who do not receive any treatment at all.{ref}Hr\u00f3bjartsson, A., & G\u00f8tzsche, P. C. (2010). Placebo interventions for all clinical conditions. <em>Cochrane Database of Systematic Reviews</em>. <a href=\"https://doi.org/10.1002/14651858.CD003974.pub3\">https://doi.org/10.1002/14651858.CD003974.pub3</a>  <br>Hr\u00f3bjartsson, A., & G\u00f8tzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. <em>New England Journal of Medicine</em>, <em>344</em>(21), 1594\u20131602. <a href=\"https://doi.org/10.1056/NEJM200105243442106%7B/ref\">https://doi.org/10.1056/NEJM200105243442106</a>{/ref} Instead, the placebo group serves as a typical control group, to show what would happen to these patients even if they did not receive treatment.</p>\n\n\n\n<h4>Randomization ensures that there are no differences between control and treatment group apart from whether they received the treatment</h4>\n\n\n\n<p>Before participants are enrolled in a trial, they might already have different levels of risk of developing the outcomes. </p>\n\n\n\n<p>Let’s look at an example to illustrate this point. </p>\n\n\n\n<p>Statins are a type of drug commonly used to prevent stroke. But strokes are more common among people who use statins. Does that mean that statins <em>caused</em> an increase in the rates of stroke? </p>\n\n\n\n<p>No \u2013 people who are prescribed statins are more likely to have cardiovascular problems to begin with, which increases the chances of having a stroke later on. When this is accounted for, researchers find that people who take statins are actually <em>less likely</em> to develop a stroke.{ref}Orkaby, A. R., Gaziano, J. M., Djousse, L., & Driver, J. A. (2017). Statins for Primary Prevention of Cardiovascular Events and Mortality in Older Men. <em>Journal of the American Geriatrics Society</em>, <em>65</em>(11), 2362\u20132368. <a href=\"https://doi.org/10.1111/jgs.14993\">https://doi.org/10.1111/jgs.14993</a><br>Makihara, N., Kamouchi, M., Hata, J., Matsuo, R., Ago, T., Kuroda, J., Kuwashiro, T., Sugimori, H., & Kitazono, T. (2013). Statins and the risks of stroke recurrence and death after ischemic stroke: The Fukuoka Stroke Registry. <em>Atherosclerosis</em>, <em>231</em>(2), 211\u2013215. <a href=\"https://doi.org/10.1016/j.atherosclerosis.2013.09.017\">https://doi.org/10.1016/j.atherosclerosis.2013.09.017</a><br>N\u00ed Chr\u00f3in\u00edn, D., Asplund, K., \u00c5sberg, S., Callaly, E., Cuadrado-Godia, E., D\u00edez-Tejedor, E., Di Napoli, M., Engelter, S. T., Furie, K. L., Giannopoulos, S., Gotto, A. M., Hannon, N., Jonsson, F., Kapral, M. K., Mart\u00ed-F\u00e0bregas, J., Mart\u00ednez-S\u00e1nchez, P., Milionis, H. J., Montaner, J., Muscari, A., \u2026 Kelly, P. J. (2013). Statin Therapy and Outcome After Ischemic Stroke: Systematic Review and Meta-Analysis of Observational Studies and Randomized Trials. <em>Stroke</em>, <em>44</em>(2), 448\u2013456. <a href=\"https://doi.org/10.1161/STROKEAHA.112.668277\">https://doi.org/10.1161/STROKEAHA.112.668277</a>{/ref}</p>\n\n\n\n<p>If researchers simply compared the rates of stroke in those who used statins with those who did not, they would miss the fact that there were other differences between the two groups, which could have caused differences in their rates of stroke. </p>\n\n\n\n<p>Important differences such as these \u2013 which affect people’s likelihood of receiving the treatment (using statins) <em>and also</em> affect the outcome (the risk of a stroke) \u2013 are called \u2018confounders\u2019.</p>\n\n\n\n<p>But it can be difficult to know what all of these confounders might be. It can also be difficult to measure and account for them properly. In fact, scientists can actually <em>worsen</em> a study by accounting for the wrong factors.{ref}Cinelli, C., Forney, A., & Pearl, J. (2020). A crash course in good and bad controls. <em>Available at SSRN, 3689437:</em> <a href=\"https://dx.doi.org/10.2139/ssrn.3689437\">http://dx.doi.org/10.2139/ssrn.3689437</a>{/ref}</p>\n\n\n\n<p>What happens when participants are randomized in a trial? </p>\n\n\n\n<p>Randomization is the procedure of allocating them into one of two groups at random.</p>\n\n\n\n<p>With randomization, the problems above are minimized: everyone has the possibility of receiving the treatment. Whether people receive the treatment is not determined by the risks they have, but whether they are randomly selected to receive the treatment.</p>\n\n\n\n<p>So, the overall risks of developing the outcome in one group become comparable to the risks in the other group.</p>\n\n\n\n<p>Randomization means that it is not a problem when there are confounders that are not known or not measured. Researchers don’t have to know about why or how the outcome usually occurs.{ref}Aronow, P., Robins, J. M., Saarinen, T., S\u00e4vje, F., & Sekhon, J. (2021). Nonparametric identification is not enough, but randomized controlled trials are. <em>ArXiv Preprint ArXiv:2108.11342</em>.{/ref}</p>\n\n\n\n<h4>Concealment and blinding limit the biases of researchers and expectations of participants</h4>\n\n\n\n<p>In a clinical trial, participants or scientists might realize which groups they are assigned to. For example, the drug might smell, taste or look different from the placebo. Or it might have obvious benefits or different side effects compared to the placebo.{ref}Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., Cochrane Bias Methods Group, & Cochrane Statistical Methods Group. (2011). The Cochrane Collaboration\u2019s tool for assessing risk of bias in randomised trials. <em>BMJ</em>, <em>343</em>(oct18 2), d5928\u2013d5928.<a href=\"https://doi.org/10.1136/bmj.d5928\"> https://doi.org/10.1136/bmj.d5928</a>{/ref}</p>\n\n\n\n<p>Concealment is a first step in preventing this: this is the procedure of preventing scientists from knowing which treatment people <em>will be</em> assigned to. </p>\n\n\n\n<p>Blinding is a second step: this is the procedure of preventing participants and scientists from finding out which treatment group people <em>have been</em> assigned to.{ref}Schulz, K. F., Chalmers, I., & Altman, D. G. (2002). The Landscape and Lexicon of Blinding in Randomized Trials. <em>Annals of Internal Medicine</em>, <em>136</em>(3), 254. <a href=\"https://doi.org/10.7326/0003-4819-136-3-200202050-00022\">https://doi.org/10.7326/0003-4819-136-3-200202050-00022</a>{/ref} </p>\n\n\n\n<p>When blinding is incomplete, it can partly reverse the benefits of randomization. For example, in clinical trials for oral health, the benefits of treatments appear larger when patients and the scientists who assess their health are not blinded sufficiently.{ref}Saltaji, H. <em>et al.</em> Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions. <em>BMC Med Res Methodol</em> <strong>18</strong>, 42 (2018).{/ref} </p>\n\n\n\n<p>If randomization was maintained, the only reason that groups would differ on their outcomes was the treatment they received. However, if the treatments were not hidden from scientists and participants, other factors could cause differences between them.</p>\n\n\n\n<p>Sometimes, blinding is not possible \u2013 there might not be something that is safe and closely resembles the treatment, which could be used as a placebo. </p>\n\n\n\n<p>Fortunately, this does not necessarily mean that these trials cannot be useful. Researchers can measure verifiable outcomes (such as changes in blood levels or even deaths) to avoid some placebo effects.</p>\n\n\n\n<p>But even when blinding occurs, participants and researchers might still make different decisions, because of the effects of the treatment or placebo. </p>\n\n\n\n<p>For example, some participants might decide to withdraw from the trial or not follow the protocols of the trial closely. Similarly, scientists might guess which groups people are in and therefore treat them differently or measure their outcomes in a biased way.</p>\n\n\n\n<p>It’s difficult to predict how this might affect the results of a trial. It could cause us to overestimate or underestimate the benefit of a treatment. For example, in the clinical trials for <a href=\"https://ourworldindata.org/covid-vaccinations\">Covid-19 vaccines</a>, some participants may have guessed that they received the vaccine because they experienced side effects. </p>\n\n\n\n<p>So, they may have believed that they were more protected and took fewer precautions. This means they may have increased their risk of catching Covid-19. This would make it appear as if the vaccines gave less protection than they actually did: it would result in an underestimate of their efficacy.</p>\n\n\n\n<h4>Preregistration allows us to hold researchers accountable to their original study plans</h4>\n\n\n\n<div class=\"wp-block-columns\">\n<div class=\"wp-block-column\">\n<p>Some of the procedures we’ve explored so far are used to safeguard research against errors and biases that scientists can have. Pre-registration is another procedure that contributes to the same goal. </p>\n\n\n\n<p>After the data in a study is analyzed, scientists have some choice in which results they present to their colleagues and the wider community. This opens up research to the possibility of cherry-picking.</p>\n\n\n\n<p>This problem unfortunately often arises in studies that are sponsored by industry. If a pharmaceutical company is testing their new drugs in trials, disappointing results can lead to financial losses. So, they may decide not to publish them. </p>\n\n\n\n<p>But this problem is not limited to trials conducted by pharmaceutical companies.</p>\n\n\n\n<p>For many reasons, scientists may decide not to publish some of their studies. Or they might re-analyze their data in a different way to find more positive results. Even if scientists want to publish their findings, journals may decide not to publish them because they may be seen as disappointing, controversial or uninteresting. </p>\n\n\n\n<p>To counter these incentives, scientists can follow the practice of “pre-registration.” This is when they publicly declare which analyses they plan to do in advance of collecting the data.</p>\n\n\n\n<p>In 2000, the United States Food and Drug Administration (FDA) established an online registry for the details of clinical trials. The FDA required that scientists who were studying therapies for serious diseases needed to provide some details of their study designs and the outcomes that they planned to investigate.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. <em>Jama</em>, <em>307</em>(17), 1861\u20131864.{/ref}</p>\n\n\n\n<p>This requirement reduced the bias towards positive findings in published research. We see evidence of this in the chart, showing clinical trials funded by the National Heart, Lung and Blood Institute (NHLBI), which adopted this requirement. </p>\n\n\n\n<p>Before 2000 \u2013 when pre-registration of studies was not required \u2013 most candidate drugs to treat or prevent cardiovascular disease showed large benefits. But most trials published after 2000 showed no benefit.{ref}Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. <em>PLOS ONE</em>, <em>10</em>(8), e0132382. <a href=\"https://doi.org/10.1371/journal.pone.0132382\">https://doi.org/10.1371/journal.pone.0132382</a>{/ref}</p>\n</div>\n\n\n\n<div class=\"wp-block-column\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" width=\"1414\" height=\"1169\" src=\"https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2.jpg\" alt=\"\" class=\"wp-image-48462\" srcset=\"https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2.jpg 1414w, https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2-400x331.jpg 400w, https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2-665x550.jpg 665w, https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2-150x124.jpg 150w, https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2-768x635.jpg 768w\" sizes=\"(max-width: 1414px) 100vw, 1414px\" /></figure>\n</div>\n</div>\n\n\n\n<div class=\"wp-block-columns is-style-sticky-right\">\n<div class=\"wp-block-column\">\n<p>Over the last two decades, this practice was strengthened and expanded. In 2007, the FDA required that most approved trials must be registered when people are enrolled into the study. They introduced notices and civil penalties for not doing so. Now, similar requirements are in place in Europe, Canada and some other parts of the world.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. <em>Jama</em>, <em>307</em>(17), 1861\u20131864. <a href=\"https://doi.org/10.1001/jama.2012.4230\">https://doi.org/10.1001/jama.2012.4230</a>{/ref}</p>\n\n\n\n<p>Importantly, sponsors of clinical trials are required to share the results online after the trial is completed. </p>\n\n\n\n<p>But unfortunately many still violate these requirements. For example, less than half (40%) of trials that were conducted in the US since 2017 actually reported their results on time.{ref}DeVito, N. J., & Goldacre, B. (2021). Evaluation of Compliance With Legal Requirements Under the FDA Amendments Act of 2007 for Timely Registration of Clinical Trials, Data Verification, Delayed Reporting, and Trial Document Submission. JAMA Internal Medicine, 181(8), 1128. <a href=\"https://doi.org/10.1001/jamainternmed.2021.2036\">https://doi.org/10.1001/jamainternmed.2021.2036</a>{/ref} </p>\n\n\n\n<p>Here, you can see patterns in reporting results in Europe. According to EU regulations, all clinical trials in the European Economic Area are required to report their results to the European Clinical Trials Register (EU-CTR) within a year of completing the trial. But reporting rates vary a lot by country. By January 2021, most clinical trials by non-commercial sponsors in the UK (94%) reported their results in time, while nearly none (4%) in France had.{ref}Note: Data is shown until January 2021 for trials. After the UK left the European Union in January 2021, clinical trials in the UK were then no longer required to report their results to the EU-CTR. Only data from trials by non-commercial sponsors is shown. This includes trials sponsored by institutions such as universities, hospitals, research foundations and so on.<br>Dal-R\u00e9, R., Goldacre, B., Mahillo-Fern\u00e1ndez, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. <em>Journal of Clinical Epidemiology</em>, S0895435621003577. <a href=\"https://doi.org/10.1016/j.jclinepi.2021.11.005\">https://doi.org/10.1016/j.jclinepi.2021.11.005</a>{/ref}</p>\n\n\n\n<p>Although a large share of clinical trials fail to report their results when they are due, this has improved recently: only half (50%) of clinical trials by non-commercial sponsors reported their results in time in 2018, while more than two-thirds (69%) did in 2021.{ref}Dal-R\u00e9, R., Goldacre, B., Mahillo-Fern\u00e1ndez, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. <em>Journal of Clinical Epidemiology</em>, S0895435621003577. <a href=\"https://doi.org/10.1016/j.jclinepi.2021.11.005\">https://doi.org/10.1016/j.jclinepi.2021.11.005</a>{/ref}</p>\n</div>\n\n\n\n<div class=\"wp-block-column\"></div>\n</div>\n\n\n\n<div class=\"wp-block-columns is-style-side-by-side\">\n<div class=\"wp-block-column\" style=\"flex-basis:50%\">\n<iframe src=\"https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-in-time-by-country?country=GBR~FRA~BEL~NLD~DEU~ITA~ESP~SWE~AUT~DNK\" loading=\"lazy\" style=\"width: 100%; height: 600px; border: 0px none;\"></iframe>\n</div>\n\n\n\n<div class=\"wp-block-column\" style=\"flex-basis:50%\">\n<iframe src=\"https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-within-a-year-over-time?country=~European+Economic+Area\" loading=\"lazy\" style=\"width: 100%; height: 600px; border: 0px none;\"></iframe>\n</div>\n</div>\n\n\n\n<h4>Pulling the layers together</h4>\n\n\n\n<p>Together, these procedures give us a far more reliable way to distinguish between what works and what doesn’t \u2013 even when we don’t know enough, when we’re wrong and when we see what we want to see.</p>\n\n\n\n<p>Unfortunately, we have seen that many clinical trials do not follow them. They tend not to report the details of their methods or test whether their participants were blinded to what they received. These standards cannot just be expected from scientists \u2013 they require cooperation from funders and journals, and they need to be actively enforced with penalties and incentives.</p>\n\n\n\n<p>We’ve also seen that trials can still suffer from remaining problems: that participants may drop out of the study and not adhere to the treatment. And clinical trials tend to not share their data or the code that they used to analyze it.</p>\n\n\n\n<p>So, despite all the benefits they provide, we shouldn’t see these layers as a fixed checklist. Just as some of them were introduced recently, they may still evolve in the future \u2013 open data sharing, registered reports and procedures to reduce dropouts may be next on the list.{ref}Chambers, C., & Tzavella, L. (2020). The past, present, and future of Registered Reports.{/ref}</p>\n\n\n\n<p>The procedures used in these trials are not a silver bullet. But when they are upheld, they can make these trials a more powerful source of evidence for understanding cause and effect. </p>\n\n\n\n<p>To make progress, we need to be able to understand the problems that affect us, their causes and solutions. Randomized controlled trials can give scientists a reliable way to collect evidence on these important questions, and give us the ability to make better decisions. At Our World in Data, this is why they matter most of all.</p>\n\n\n\n<hr class=\"wp-block-separator\"/>\n\n\n\n<h5><em>Keep reading on Our World in Data</em></h5>\n\n\n <block type=\"prominent-link\" style=\"is-style-thin\">\n <link-url>https://ourworldindata.org/what-is-depression</link-url>\n <title></title>\n <content></content>\n <figure></figure>\n </block>\n\n <block type=\"prominent-link\" style=\"is-style-thin\">\n <link-url>https://ourworldindata.org/mental-health</link-url>\n <title></title>\n <content></content>\n <figure></figure>\n </block>\n\n\n<p><strong>Acknowledgements:</strong> I would like to thank Darren Dahly, Nathaniel Bechhofer, Hannah Ritchie and Max Roser for reading drafts of this post and their very helpful guidance and suggestions to improve it.</p>\n\n\n\n<hr class=\"wp-block-separator\"/>\n\n\n\t<block type=\"additional-information\" default-open=\"false\">\n\t\t<content>\n\n<h3>Appendix: When are randomized controls ethical?</h3>\n\n\n\n<div class=\"wp-block-columns is-style-sticky-right\">\n<div class=\"wp-block-column\">\n<p>Trials are useful to test new ideas or treatments, when people do not have enough knowledge to recommend them on a larger scale. </p>\n\n\n\n<p>But people also have ethical concerns surrounding trials of treatments that are already used by doctors. For one, they might believe that it isn’t justified to give patients a placebo when a drug is available, even if it is unclear how well the drug works. </p>\n\n\n\n<p>How can we balance these concerns with the benefits of trials? When are trials important?</p>\n\n\n\n<p>A leading view is that they are justified when there is disagreement or uncertainty about the benefits of an intervention.{ref}Wendler, D. The Ethics of Clinical Research. in The Stanford Encyclopedia of Philosophy (ed. Zalta, E. N.) (Metaphysics Research Lab, Stanford University, 2021).<br>London, A. J. Equipoise in Research: Integrating Ethics and Science in Human Research. JAMA 317, 525 (2017).{/ref}</p>\n\n\n\n<p>Outside of a trial, some doctors might prescribe a particular treatment, while other doctors would not, because they believe that same treatment is ineffective. In this case, the options in a trial \u2013 an intervention or control \u2013 would not be so different from what people could already encounter. </p>\n\n\n\n<p>When experts disagree on a topic, an RCT is useful because it can resolve these conflicts and inform us about which option is better. </p>\n\n\n\n<p>So, to make the most of trials, they should be planned in line with the questions that people have. For example, if we wanted to understand whether vaccines gave less protection to people who were immunocompromised, then we should plan a trial with enough participants with these conditions.</p>\n\n\n\n<p></p>\n</div>\n\n\n\n<div class=\"wp-block-column\"></div>\n</div>\n\n</content>\n\t</block>\n\n\n<p></p>\n", "protected": false }, "excerpt": { "rendered": "Randomized controlled trials are a key tool to study cause and effect. Why do they matter and how do they work?", "protected": false }, "date_gmt": "2022-02-10T11:00:00", "modified": "2023-04-12T13:57:29", "template": "", "categories": [ 46, 1 ], "ping_status": "closed", "authors_name": [ "Saloni Dattani" ], "modified_gmt": "2023-04-12T12:57:29", "comment_status": "closed", "featured_media": 48379, "featured_media_paths": { "thumbnail": "/app/uploads/2022/02/RCT-thumbnail-1-150x53.jpg", "medium_large": "/app/uploads/2022/02/RCT-thumbnail-1-768x269.jpg" } }

title

slug

type

status

content

archieml

archieml_update_statistics

published_at

updated_at

gdocSuccessorId

authors

excerpt

created_at_in_wordpress

updated_at_in_wordpress

featured_image

formattingOptions

markdown

wpApiSnapshot

48378

Why randomized controlled trials matter and the procedures that strengthen them

randomized-controlled-trials

post

publish

In order to tackle the world's largest problems, we need to be able to identify their causes and effective ways to solve them. How would we know about the effects of a new idea, treatment or policy? Answering this question can be difficult. Some research is poorly done and there are incentives for scientists and funders to exaggerate their findings.   In this post, I look at a key tool to study causal effects reliably: randomized controlled trials (RCTs). What makes RCTs useful is the set of procedures they are expected to follow, such as placebo controls, concealment, randomization and pre-registration. These procedures are layers of protection against challenges that scientists usually face in research.   To be able to read published research, we need to understand why these procedures matter and why they are not always followed in practice. With this knowledge, we have a greater ability to identify the causes and solutions to the largest problems that affect us.    At Our World in Data, we bring attention to the world's largest problems. We explore what these problems are, why they matter and how large they are. Whenever possible, we try to explain why these problems exist, and how we might solve them.   To make progress, we need to be able to identify real solutions and evaluate them carefully. But doing this well is not simple. It's difficult for scientists to collect good evidence on the costs and benefits of a new idea, policy, or treatment. And it's challenging for decision-makers to scrutinize the evidence that exists, to make better decisions.   What we need are reliable ways to distinguish between ideas that work and those that don't.   In this post, I will explain a crucial tool that helps us do this – randomized controlled trials (RCTs). We will see that RCTs matter for three reasons: when we don't know about the effects of interventions, when we don't know how to study them, and when scientific research is affected by biases.   <h4>What are randomized controlled trials?</h4>   To begin with, what are RCTs? These are experiments where people are given, at random, either an intervention (such as a drug) or a control and then followed up to see how they fare on various outcomes.   RCTs are conducted by researchers around the world. In the map, you can see how many RCTs have ever been published in high-ranked medical journals, by the country where the first author was based. Over 18,000 of these were from the United States, but most countries have had fewer than a hundred.{ref}Catalá-López, F., Aleixandre-Benavent, R., Caulley, L., Hutton, B., Tabarés-Seisdedos, R., Moher, D., & Alonso-Arroyo, A. (2020). Global mapping of randomised trials related articles published in high-impact-factor medical journals: a cross-sectional analysis. Trials, 21(1), 1-24.{/ref} RCTs have also become more common over time.{ref}Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., & Otte, W. M. (2021). The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. PLOS Biology, 19(4), e3001162. <a href="https://doi.org/10.1371/journal.pbio.3001162">https://doi.org/10.1371/journal.pbio.3001162</a>{/ref}   It's easy to take RCTs for granted, but these trials have transformed our understanding of cause and effect. They are a powerful tool to illuminate what is unknown or uncertain; to discern whether something works and how well it works.   But it's also important to recognize that these trials are not always perfect, and understand why.   The strengths of RCTs are subtle: they are powerful because of the set of procedures that they are expected to follow. This includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.   In this post, we will explore why these procedures matter – how each one adds a layer of protection against complications that scientists face when they do research.    <iframe src="https://ourworldindata.org/grapher/total-number-of-rcts-published-in-high-ranked-medical-journals?tab=map&country=GBR~USA~IND~DNK~NGA~BOL~MEX~FRA~JPN~CHN" loading="lazy" style="width: 100%; height: 600px; border: 0px none;"></iframe>   <h4>The fundamental problem of causal inference</h4>   We make decisions based on our understanding of how things work – we try to predict the consequences of our actions.    But understanding cause and effect is not just crucial for personal decisions: our knowledge of what works can have large consequences for people around us.    An example is antiretroviral therapy (ART), which is used to treat <a href="http://ourworldindata.org/hiv-aids">HIV/AIDS</a>. The benefits of these drugs were surprising. One of the first ART drugs discovered was azidothymidine, which had previously been abandoned as an ineffective treatment for cancer.{ref}National Research Council (U.S.) (1993). The social impact of AIDS in the United States. 4. Clinical Research and Drug Regulation. National Academy Press.{/ref} The discovery that azidothymidine and other antiretroviral therapies worked, and their use worldwide, has prevented millions of deaths from HIV/AIDS, as the chart shows.   Discovering what works can save lives. But discovering what doesn’t work can do the same. It means we can redirect our time and resources away from things that don't work, towards things that do.   Even when we already know that something works, understanding how well it works can help us make better decisions.    An example for this is <a href="https://ourworldindata.org/grapher/bcg-immunization-coverage-for-tb-among-1-year-olds?country=~OWID_WRL">the BCG vaccine</a>, which reduces the risk of tuberculosis. The efficacy of this vaccine is different for different people around the world, and the reasons for this are unclear.{ref}Dockrell, H. M., & Smith, S. G. (2017). What Have We Learnt about BCG Vaccination in the Last 20 Years? Frontiers in Immunology, 8, 1134. <a href="https://doi.org/10.3389/fimmu.2017.01134">https://doi.org/10.3389/fimmu.2017.01134</a> Mangtani, P., Abubakar, I., Ariti, C., Beynon, R., Pimpin, L., Fine, P. E. M., Rodrigues, L. C., Smith, P. G., Lipman, M., Whiting, P. F., & Sterne, J. A. (2014). Protection by BCG Vaccine Against Tuberculosis: A Systematic Review of Randomized Controlled Trials. Clinical Infectious Diseases, 58(4), 470–480. <a href="https://doi.org/10.1093/cid/cit790%7B/ref">https://doi.org/10.1093/cid/cit790</a>{/ref} Still, knowing this is informative, because it tells us that more effort is needed to protect people against the disease in places where the benefit of the vaccine is low.   If there was a reliable way to know about the effects of the measures we took, we could prioritize solutions that are most effective.   So, how would we understand cause and effect without trials?   One way is through observation: we can observe what different people did and track their outcomes afterwards. But it's possible that the events that followed their actions were simply a coincidence, or that they would have happened anyway.   The biggest challenge in trying to understand causes and effects is that we are only able to see one version of history.    When someone makes a particular decision, we are able to see what follows, but we cannot see what would have happened if they had made a different decision. This is known as “the fundamental problem of causal inference."{ref}Holland, P. W. (1986). Statistics and Causal Inference. Journal of the American Statistical Association, 81(396), 945–960. <a href="https://doi.org/10.1080/01621459.1986.10478354">https://doi.org/10.1080/01621459.1986.10478354</a> Imbens, G. W., & Rubin, D. B. (2010). Rubin causal model. In Microeconometrics (pp. 229–241). Springer. Rubin, D. B. (1977). Assignment to Treatment Group on the Basis of a Covariate. Journal of Educational Statistics, 2(1), 1–26. <a href="https://doi.org/10.3102/10769986002001001%7B/ref">https://doi.org/10.3102/10769986002001001</a>{/ref}   What this means is that it is impossible to predict the effects that an action will have for an individual person, but we can try to predict the effects it would have on average.   <iframe src="https://ourworldindata.org/grapher/hivaids-deaths-and-averted-due-to-art?country=~OWID_WRL" loading="lazy" style="width: 100%; height: 600px; border: 0px none;"></iframe>   <h4>Why randomized controlled trials matter</h4>   <div class="wp-block-columns"> <div class="wp-block-column"> Sometimes we do not need an RCT to identify causes and effects. The effect of smoking on lung cancer is one example, where scientists could be confident as early as the 1960s that the large increase in lung cancer rates could not have been caused by other factors.    This was because there was already knowledge from many different lines of evidence.{ref}Hill, G., Millar, W., & Connelly, J. (2003). “The Great Debate”: Smoking, Lung Cancer, and Cancer Epidemiology. Canadian Bulletin of Medical History, 20(2), 367-386.{/ref} Experiments, biopsies and population studies all showed that cigarette smoke was associated with specific changes in lung tissue and with the incidence of lung cancer. The association was so large and consistent that it could not be explained by other factors.{ref}Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B., & Wynder, E. L. (1959). Smoking and Lung Cancer: Recent Evidence and a Discussion of Some Questions. JNCI: Journal of the National Cancer Institute. <a href="https://doi.org/10.1093/jnci/22.1.173">https://doi.org/10.1093/jnci/22.1.173</a>{/ref}   Even now, smoking is estimated to cause a large proportion of cancer deaths in many countries, as you can see in the chart.   But in other situations, RCTs have made a huge difference to our understanding of the world. Prioritizing the drugs that were shown to be effective in trials saved lives, as we saw with the example of antiretroviral therapy to treat HIV/AIDS.   People sometimes refer to RCTs as the "gold standard of evidence", but it's more useful to recognize that their strengths emerge from a set of procedures that they aim to follow. </div>   <div class="wp-block-column"> <iframe src="https://ourworldindata.org/grapher/share-of-cancer-deaths-attributed-to-tobacco" loading="lazy" style="width: 100%; height: 600px; border: 0px none;"></iframe> </div> </div>   <div class="wp-block-columns"> <div class="wp-block-column"> <h4>Why do RCTs matter? </h4>   In my opinion, even though we do have lots of knowledge about various topics, these trials matter for three reasons. They matter because we may not know enough, because we can be wrong, and because we might see what we want to see.   First, they matter when we don't know.    Randomized controlled trials can illuminate our understanding of effects, especially when there is uncertainty around them. They can help resolve disagreements between experts.   In some cases, scientists can use other methods to investigate these topics. But when there is insufficient knowledge, other methods may not be enough.    This is because, in a typical study, scientists need to be able to account for the possibility that other factors are causing the outcome.    They need to have knowledge about why an effect usually occurs and whether it will happen anyway. They need to know about which other risk factors can cause the outcome and what makes it likely that someone will receive the treatment. They also need to know how to measure these factors properly, and how to account for them.   The second reason they matter is when we are wrong.    Even when scientists think they know which risk factors affect the outcome, they might be incorrect. They might not account for the right risk factors or they might account for the wrong ones.   In an RCT where scientists use randomization, blinding and concealment, they can minimize both of these problems. Despite people's risks before the trial, the reason that someone will receive the treatment is the random number they are given. The reason that participants in each group differ at the end of the study is the group they were randomized to.    Even if we don't know about other risk factors, or even if we are wrong about them, these procedures mean that we can still find out whether a treatment has an effect and how large the effect is.   The third reason is when we see what we want to see.    When participants expect to feel different after a treatment, they might report that they feel different, even if they didn't receive the treatment at all.    When scientists want to see people improve after a treatment, they might decide to allocate the treatment to healthier participants. They might measure their data differently or re-analyze their data until they can find an improvement. They might even decide against publishing their findings when they don't find a benefit.   If scientists use concealment, blinding, and pre-registration, they can reduce these biases. They can protect research against their own prejudices and the expectations of participants. They can also defend scientific knowledge from poor incentives, such as the desires of scientists to hype up their findings and the incentives for pharmaceutical companies to claim that their drugs work. </div>   <div class="wp-block-column"></div> </div>   <h3>The layers of protection against bias</h3>   Previously, I described how the strengths of RCTs emerge from the set of procedures that they aim to follow.   In medicine, this commonly includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.{ref}Although these procedures are also used outside of medicine, it can be difficult to apply them elsewhere. For example, in a trial that tests the effectiveness of talking therapy, it would be known to the participants that they are receiving it; it may not be possible to find a placebo control version to disguise the procedure. Due to constraints in length and focus, I will not detail the advantages of intention-to-treat analysis or experimentation.{/ref} Each of these helps to protect the evidence against distortions, which can come from many sources. But they are not always enforced. Below, we will explore them in more detail.   <h4>The control group gives us a comparison to see what would have happened otherwise</h4>   The most crucial element to understand causes and effects is a control group.    Let's use antidepressants as an example to see why we need them.   If no one in the world received antidepressants, we wouldn’t know what their effects were. But the opposite is also true. If everyone received antidepressants, we wouldn't know how things would be without them.    In order to understand the effect of an antidepressant, we need to see which other outcomes are possible. The most important thing we need to understand effects is a comparison – we need some people who receive the treatment and some who don’t. The people who don’t could be our control group.   An ideal control group should allow us to control the biases and measurement errors in a study completely.    For example, different doctors may diagnose depression differently. If someone had symptoms of depression, then the chances they get diagnosed should be equal in the antidepressant group and the control group. If they weren't equal, we could mistake differences between the groups for an effect of antidepressants, even if they didn't have any effect.   In fact, an ideal control group should allow us to control the total effects of all of the other factors that could affect people's mood, not just control for the biases and errors in studies.    As an example, we know that the symptoms of depression tend to change over time, as I explained in <a href="https://ourworldindata.org/what-is-depression">this earlier post</a>.    You can see this in the chart. This shows that the symptoms of depression tend to decline over time, among people who are diagnosed with depression but not treated for it. This is measured by seeing how many patients would still meet the threshold for a diagnosis of depression later on.    Almost a quarter of patients with depression (23%) would no longer meet the threshold for depression after three months, despite receiving no treatment. Just over half (53%) would no longer meet the threshold after one year.{ref}Whiteford, H. A., Harris, M. G., McKeon, G., Baxter, A., Pennell, C., Barendregt, J. J., & Wang, J. (2013). Estimating remission from untreated major depression: A systematic review and meta-analysis. Psychological Medicine, 43(8), 1569–1585. <a href="https://doi.org/10.1017/S0033291712001717">https://doi.org/10.1017/S0033291712001717</a>{/ref} This change is known as "regression to the mean."   If we didn’t have a control group in our study, we might misattribute such an improvement to the antidepressant. We need a control group to know how much their symptoms would improve anyway.   <figure class="wp-block-image"><img src="https://lh5.googleusercontent.com/khK-H11avMDGkTpCLAqCJIsIyAJSX63_Blm-15GL-0fkGYoDK-47Q1Ancs-i06algU5S3GRWTTeL4hNMQgfezt2_DlxwUm1kTykSx6oc3Unzbnz0UNaKUbGZdQF4fagJFIwrXvCu" alt=""/></figure>   <h4>Placebos allow us to account for placebo effects</h4>   Some types of controls are special because they resemble the treatment without actually being it – these controls are called placebos. An ideal placebo has all the qualities above and also allows us to account for "placebo effects." In this case, the placebo effect refers to when people's moods improve from the mere procedure of receiving the antidepressant.    For example, taking a pill might improve people's mood because they believe that taking a pill will give them some benefit, even if the pill does not actually contain any active ingredient.    How large is the placebo effect?    Some clinical trials have tried to estimate this by comparing patients who received a placebo to patients who received no treatment at all. Overall, these studies have found that placebo effects are small in clinical trials for many conditions, but are larger for physical treatments, such as acupuncture to treat pain.{ref}Hróbjartsson, A., & Gøtzsche, P. C. (2010). Placebo interventions for all clinical conditions. Cochrane Database of Systematic Reviews. https://doi.org/10.1002/14651858.CD003974.pub3   Hróbjartsson, A., & Gøtzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. New England Journal of Medicine, 344(21), 1594–1602. <a href="https://doi.org/10.1056/NEJM200105243442106">https://doi.org/10.1056/NEJM200105243442106</a>{/ref}    Placebo effects tend to be larger when scientists are recording symptoms reported by patients, rather than when they are measuring something that can be verified by others, such as results from a blood test or death.    For depression, the placebo effect is non-significant. Studies do not detect a difference in the moods of patients who receive a placebo and those who do not receive any treatment at all.{ref}Hróbjartsson, A., & Gøtzsche, P. C. (2010). Placebo interventions for all clinical conditions. Cochrane Database of Systematic Reviews. <a href="https://doi.org/10.1002/14651858.CD003974.pub3">https://doi.org/10.1002/14651858.CD003974.pub3</a>   Hróbjartsson, A., & Gøtzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. New England Journal of Medicine, 344(21), 1594–1602. <a href="https://doi.org/10.1056/NEJM200105243442106%7B/ref">https://doi.org/10.1056/NEJM200105243442106</a>{/ref} Instead, the placebo group serves as a typical control group, to show what would happen to these patients even if they did not receive treatment.   <h4>Randomization ensures that there are no differences between control and treatment group apart from whether they received the treatment</h4>   Before participants are enrolled in a trial, they might already have different levels of risk of developing the outcomes.    Let's look at an example to illustrate this point.    Statins are a type of drug commonly used to prevent stroke. But strokes are more common among people who use statins. Does that mean that statins caused an increase in the rates of stroke?    No – people who are prescribed statins are more likely to have cardiovascular problems to begin with, which increases the chances of having a stroke later on. When this is accounted for, researchers find that people who take statins are actually less likely to develop a stroke.{ref}Orkaby, A. R., Gaziano, J. M., Djousse, L., & Driver, J. A. (2017). Statins for Primary Prevention of Cardiovascular Events and Mortality in Older Men. Journal of the American Geriatrics Society, 65(11), 2362–2368. <a href="https://doi.org/10.1111/jgs.14993">https://doi.org/10.1111/jgs.14993</a> Makihara, N., Kamouchi, M., Hata, J., Matsuo, R., Ago, T., Kuroda, J., Kuwashiro, T., Sugimori, H., & Kitazono, T. (2013). Statins and the risks of stroke recurrence and death after ischemic stroke: The Fukuoka Stroke Registry. Atherosclerosis, 231(2), 211–215. <a href="https://doi.org/10.1016/j.atherosclerosis.2013.09.017">https://doi.org/10.1016/j.atherosclerosis.2013.09.017</a> Ní Chróinín, D., Asplund, K., Åsberg, S., Callaly, E., Cuadrado-Godia, E., Díez-Tejedor, E., Di Napoli, M., Engelter, S. T., Furie, K. L., Giannopoulos, S., Gotto, A. M., Hannon, N., Jonsson, F., Kapral, M. K., Martí-Fàbregas, J., Martínez-Sánchez, P., Milionis, H. J., Montaner, J., Muscari, A., … Kelly, P. J. (2013). Statin Therapy and Outcome After Ischemic Stroke: Systematic Review and Meta-Analysis of Observational Studies and Randomized Trials. Stroke, 44(2), 448–456. <a href="https://doi.org/10.1161/STROKEAHA.112.668277">https://doi.org/10.1161/STROKEAHA.112.668277</a>{/ref}   If researchers simply compared the rates of stroke in those who used statins with those who did not, they would miss the fact that there were other differences between the two groups, which could have caused differences in their rates of stroke.    Important differences such as these – which affect people's likelihood of receiving the treatment (using statins) and also affect the outcome (the risk of a stroke) – are called ‘confounders’.   But it can be difficult to know what all of these confounders might be. It can also be difficult to measure and account for them properly. In fact, scientists can actually worsen a study by accounting for the wrong factors.{ref}Cinelli, C., Forney, A., & Pearl, J. (2020). A crash course in good and bad controls. Available at SSRN, 3689437: <a href="https://dx.doi.org/10.2139/ssrn.3689437">http://dx.doi.org/10.2139/ssrn.3689437</a>{/ref}   What happens when participants are randomized in a trial?    Randomization is the procedure of allocating them into one of two groups at random.   With randomization, the problems above are minimized: everyone has the possibility of receiving the treatment. Whether people receive the treatment is not determined by the risks they have, but whether they are randomly selected to receive the treatment.   So, the overall risks of developing the outcome in one group become comparable to the risks in the other group.   Randomization means that it is not a problem when there are confounders that are not known or not measured. Researchers don't have to know about why or how the outcome usually occurs.{ref}Aronow, P., Robins, J. M., Saarinen, T., Sävje, F., & Sekhon, J. (2021). Nonparametric identification is not enough, but randomized controlled trials are. ArXiv Preprint ArXiv:2108.11342.{/ref}   <h4>Concealment and blinding limit the biases of researchers and expectations of participants</h4>   In a clinical trial, participants or scientists might realize which groups they are assigned to. For example, the drug might smell, taste or look different from the placebo. Or it might have obvious benefits or different side effects compared to the placebo.{ref}Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., Cochrane Bias Methods Group, & Cochrane Statistical Methods Group. (2011). The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ, 343(oct18 2), d5928–d5928.<a href="https://doi.org/10.1136/bmj.d5928"> https://doi.org/10.1136/bmj.d5928</a>{/ref}   Concealment is a first step in preventing this: this is the procedure of preventing scientists from knowing which treatment people will be assigned to.    Blinding is a second step: this is the procedure of preventing participants and scientists from finding out which treatment group people have been assigned to.{ref}Schulz, K. F., Chalmers, I., & Altman, D. G. (2002). The Landscape and Lexicon of Blinding in Randomized Trials. Annals of Internal Medicine, 136(3), 254. <a href="https://doi.org/10.7326/0003-4819-136-3-200202050-00022">https://doi.org/10.7326/0003-4819-136-3-200202050-00022</a>{/ref}    When blinding is incomplete, it can partly reverse the benefits of randomization. For example, in clinical trials for oral health, the benefits of treatments appear larger when patients and the scientists who assess their health are not blinded sufficiently.{ref}Saltaji, H. et al. Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions. BMC Med Res Methodol 18, 42 (2018).{/ref}    If randomization was maintained, the only reason that groups would differ on their outcomes was the treatment they received. However, if the treatments were not hidden from scientists and participants, other factors could cause differences between them.   Sometimes, blinding is not possible – there might not be something that is safe and closely resembles the treatment, which could be used as a placebo.    Fortunately, this does not necessarily mean that these trials cannot be useful. Researchers can measure verifiable outcomes (such as changes in blood levels or even deaths) to avoid some placebo effects.   But even when blinding occurs, participants and researchers might still make different decisions, because of the effects of the treatment or placebo.    For example, some participants might decide to withdraw from the trial or not follow the protocols of the trial closely. Similarly, scientists might guess which groups people are in and therefore treat them differently or measure their outcomes in a biased way.   It's difficult to predict how this might affect the results of a trial. It could cause us to overestimate or underestimate the benefit of a treatment. For example, in the clinical trials for <a href="https://ourworldindata.org/covid-vaccinations">Covid-19 vaccines</a>, some participants may have guessed that they received the vaccine because they experienced side effects.    So, they may have believed that they were more protected and took fewer precautions. This means they may have increased their risk of catching Covid-19. This would make it appear as if the vaccines gave less protection than they actually did: it would result in an underestimate of their efficacy.   <h4>Preregistration allows us to hold researchers accountable to their original study plans</h4>   <div class="wp-block-columns"> <div class="wp-block-column"> Some of the procedures we've explored so far are used to safeguard research against errors and biases that scientists can have. Pre-registration is another procedure that contributes to the same goal.    After the data in a study is analyzed, scientists have some choice in which results they present to their colleagues and the wider community. This opens up research to the possibility of cherry-picking.   This problem unfortunately often arises in studies that are sponsored by industry. If a pharmaceutical company is testing their new drugs in trials, disappointing results can lead to financial losses. So, they may decide not to publish them.    But this problem is not limited to trials conducted by pharmaceutical companies.   For many reasons, scientists may decide not to publish some of their studies. Or they might re-analyze their data in a different way to find more positive results. Even if scientists want to publish their findings, journals may decide not to publish them because they may be seen as disappointing, controversial or uninteresting.    To counter these incentives, scientists can follow the practice of "pre-registration." This is when they publicly declare which analyses they plan to do in advance of collecting the data.   In 2000, the United States Food and Drug Administration (FDA) established an online registry for the details of clinical trials. The FDA required that scientists who were studying therapies for serious diseases needed to provide some details of their study designs and the outcomes that they planned to investigate.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. Jama, 307(17), 1861–1864.{/ref}   This requirement reduced the bias towards positive findings in published research. We see evidence of this in the chart, showing clinical trials funded by the National Heart, Lung and Blood Institute (NHLBI), which adopted this requirement.   Before 2000 – when pre-registration of studies was not required – most candidate drugs to treat or prevent cardiovascular disease showed large benefits. But most trials published after 2000 showed no benefit.{ref}Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. PLOS ONE, 10(8), e0132382. <a href="https://doi.org/10.1371/journal.pone.0132382">https://doi.org/10.1371/journal.pone.0132382</a>{/ref} </div>   <div class="wp-block-column"> <figure class="wp-block-image size-full"><img src="https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2.jpg" alt="" class="wp-image-48462"/></figure> </div> </div>   <div class="wp-block-columns is-style-sticky-right"> <div class="wp-block-column"> Over the last two decades, this practice was strengthened and expanded. In 2007, the FDA required that most approved trials must be registered when people are enrolled into the study. They introduced notices and civil penalties for not doing so. Now, similar requirements are in place in Europe, Canada and some other parts of the world.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. Jama, 307(17), 1861–1864. <a href="https://doi.org/10.1001/jama.2012.4230">https://doi.org/10.1001/jama.2012.4230</a>{/ref}   Importantly, sponsors of clinical trials are required to share the results online after the trial is completed.    But unfortunately many still violate these requirements. For example, less than half (40%) of trials that were conducted in the US since 2017 actually reported their results on time.{ref}DeVito, N. J., & Goldacre, B. (2021). Evaluation of Compliance With Legal Requirements Under the FDA Amendments Act of 2007 for Timely Registration of Clinical Trials, Data Verification, Delayed Reporting, and Trial Document Submission. JAMA Internal Medicine, 181(8), 1128. <a href="https://doi.org/10.1001/jamainternmed.2021.2036">https://doi.org/10.1001/jamainternmed.2021.2036</a>{/ref}    Here, you can see patterns in reporting results in Europe. According to EU regulations, all clinical trials in the European Economic Area are required to report their results to the European Clinical Trials Register (EU-CTR) within a year of completing the trial. But reporting rates vary a lot by country. By January 2021, most clinical trials by non-commercial sponsors in the UK (94%) reported their results in time, while nearly none (4%) in France had.{ref}Note: Data is shown until January 2021 for trials. After the UK left the European Union in January 2021, clinical trials in the UK were then no longer required to report their results to the EU-CTR. Only data from trials by non-commercial sponsors is shown. This includes trials sponsored by institutions such as universities, hospitals, research foundations and so on. Dal-Ré, R., Goldacre, B., Mahillo-Fernández, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. Journal of Clinical Epidemiology, S0895435621003577. <a href="https://doi.org/10.1016/j.jclinepi.2021.11.005">https://doi.org/10.1016/j.jclinepi.2021.11.005</a>{/ref}   Although a large share of clinical trials fail to report their results when they are due, this has improved recently: only half (50%) of clinical trials by non-commercial sponsors reported their results in time in 2018, while more than two-thirds (69%) did in 2021.{ref}Dal-Ré, R., Goldacre, B., Mahillo-Fernández, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. Journal of Clinical Epidemiology, S0895435621003577. <a href="https://doi.org/10.1016/j.jclinepi.2021.11.005">https://doi.org/10.1016/j.jclinepi.2021.11.005</a>{/ref} </div>   <div class="wp-block-column"></div> </div>   <div class="wp-block-columns is-style-side-by-side"> <div class="wp-block-column" style="flex-basis:50%"> <iframe src="https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-in-time-by-country?country=GBR~FRA~BEL~NLD~DEU~ITA~ESP~SWE~AUT~DNK" loading="lazy" style="width: 100%; height: 600px; border: 0px none;"></iframe> </div>   <div class="wp-block-column" style="flex-basis:50%"> <iframe src="https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-within-a-year-over-time?country=~European+Economic+Area" loading="lazy" style="width: 100%; height: 600px; border: 0px none;"></iframe> </div> </div>   <h4>Pulling the layers together</h4>   Together, these procedures give us a far more reliable way to distinguish between what works and what doesn't – even when we don't know enough, when we're wrong and when we see what we want to see.   Unfortunately, we have seen that many clinical trials do not follow them. They tend not to report the details of their methods or test whether their participants were blinded to what they received. These standards cannot just be expected from scientists – they require cooperation from funders and journals, and they need to be actively enforced with penalties and incentives.   We've also seen that trials can still suffer from remaining problems: that participants may drop out of the study and not adhere to the treatment. And clinical trials tend to not share their data or the code that they used to analyze it.   So, despite all the benefits they provide, we shouldn't see these layers as a fixed checklist. Just as some of them were introduced recently, they may still evolve in the future – open data sharing, registered reports and procedures to reduce dropouts may be next on the list.{ref}Chambers, C., & Tzavella, L. (2020). The past, present, and future of Registered Reports.{/ref}   The procedures used in these trials are not a silver bullet. But when they are upheld, they can make these trials a more powerful source of evidence for understanding cause and effect.    To make progress, we need to be able to understand the problems that affect us, their causes and solutions. Randomized controlled trials can give scientists a reliable way to collect evidence on these important questions, and give us the ability to make better decisions. At Our World in Data, this is why they matter most of all.   <hr class="wp-block-separator"/>   <h5>Keep reading on Our World in Data</h5>     Acknowledgements: I would like to thank Darren Dahly, Nathaniel Bechhofer, Hannah Ritchie and Max Roser for reading drafts of this post and their very helpful guidance and suggestions to improve it.   <hr class="wp-block-separator"/>    <h3>Appendix: When are randomized controls ethical?</h3>   <div class="wp-block-columns is-style-sticky-right"> <div class="wp-block-column"> Trials are useful to test new ideas or treatments, when people do not have enough knowledge to recommend them on a larger scale.    But people also have ethical concerns surrounding trials of treatments that are already used by doctors. For one, they might believe that it isn't justified to give patients a placebo when a drug is available, even if it is unclear how well the drug works.    How can we balance these concerns with the benefits of trials? When are trials important?   A leading view is that they are justified when there is disagreement or uncertainty about the benefits of an intervention.{ref}Wendler, D. The Ethics of Clinical Research. in The Stanford Encyclopedia of Philosophy (ed. Zalta, E. N.) (Metaphysics Research Lab, Stanford University, 2021). London, A. J. Equipoise in Research: Integrating Ethics and Science in Human Research. JAMA 317, 525 (2017).{/ref}   Outside of a trial, some doctors might prescribe a particular treatment, while other doctors would not, because they believe that same treatment is ineffective. In this case, the options in a trial – an intervention or control – would not be so different from what people could already encounter.    When experts disagree on a topic, an RCT is useful because it can resolve these conflicts and inform us about which option is better.    So, to make the most of trials, they should be planned in line with the questions that people have. For example, if we wanted to understand whether vaccines gave less protection to people who were immunocompromised, then we should plan a trial with enough participants with these conditions.   </div>   <div class="wp-block-column"></div> </div>

{
    "id": "wp-48378",
    "slug": "randomized-controlled-trials",
    "content": {
        "toc": [],
        "body": [
            {
                "text": [
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "In order to tackle the world's largest problems, we need to be able to identify their causes and effective ways to solve them. How would we know about the effects of a new idea, treatment or policy? Answering this question can be difficult. Some research is poorly done and there are incentives for scientists and funders to exaggerate their findings.",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "In this post, I look at a key tool to study causal effects reliably: randomized controlled trials (RCTs). What makes RCTs useful is the set of procedures they are expected to follow, such as placebo controls, concealment, randomization and pre-registration. These procedures are layers of protection against challenges that scientists usually face in research. ",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "To be able to read published research, we need to understand why these procedures matter and why they are not always followed in practice. With this knowledge, we have a greater ability to identify the causes and solutions to the largest problems that affect us.",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    }
                ],
                "type": "callout",
                "title": "Summary",
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "At Our World in Data, we bring attention to the world's largest problems. We explore what these problems are, why they matter and how large they are. Whenever possible, we try to explain why these problems exist, and how we might solve them.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "To make progress, we need to be able to identify real solutions and evaluate them carefully. But doing this well is not simple. It's difficult for scientists to collect good evidence on the costs and benefits of a new idea, policy, or treatment. And it's challenging for decision-makers to scrutinize the evidence that exists, to make better decisions.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "What we need are reliable ways to distinguish between ideas that work and those that don't.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "In this post, I will explain a crucial tool that helps us do this \u2013 randomized controlled trials (RCTs). We will see that RCTs matter for three reasons: when we don't know about the effects of interventions, when we don't know how to study them, and when scientific research is affected by biases.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "text": [
                    {
                        "text": "What are randomized controlled trials?",
                        "spanType": "span-simple-text"
                    }
                ],
                "type": "heading",
                "level": 3,
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "To begin with, what are RCTs? These are experiments where people are given, at random, either an intervention (such as a drug) or a control and then followed up to see how they fare on various outcomes.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "RCTs are conducted by researchers around the world. In the map, you can see how many RCTs have ever been published in high-ranked medical journals, by the country where the first author was based. Over 18,000 of these were from the United States, but most countries have had fewer than a hundred.{ref}Catal\u00e1-L\u00f3pez, F., Aleixandre-Benavent, R., Caulley, L., Hutton, B., Tabar\u00e9s-Seisdedos, R., Moher, D., & Alonso-Arroyo, A. (2020). Global mapping of randomised trials related articles published in high-impact-factor medical journals: a cross-sectional analysis. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Trials",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "21",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(1), 1-24.{/ref} RCTs have also become more common over time.{ref}Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., & Otte, W. M. (2021). The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "PLOS Biology",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "19",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(4), e3001162. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1371/journal.pbio.3001162",
                        "children": [
                            {
                                "text": "https://doi.org/10.1371/journal.pbio.3001162",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref}",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "It's easy to take RCTs for granted, but these trials have transformed our understanding of cause and effect. They are a powerful tool to illuminate what is unknown or uncertain; to discern whether something works and how well it works.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "But it's also important to recognize that these trials are not always perfect, and understand why.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "The strengths of RCTs are subtle: they are powerful because of the set of procedures that they are expected to follow. This includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "In this post, we will explore why these procedures matter \u2013 how each one adds a layer of protection against complications that scientists face when they do research.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "url": "https://ourworldindata.org/grapher/total-number-of-rcts-published-in-high-ranked-medical-journals?tab=map&country=GBR~USA~IND~DNK~NGA~BOL~MEX~FRA~JPN~CHN",
                "type": "chart",
                "parseErrors": []
            },
            {
                "text": [
                    {
                        "text": "The fundamental problem of causal inference",
                        "spanType": "span-simple-text"
                    }
                ],
                "type": "heading",
                "level": 3,
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "We make decisions based on our understanding of how things work \u2013 we try to predict the consequences of our actions.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "But understanding cause and effect is not just crucial for personal decisions: our knowledge of what works can have large consequences for people around us.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "An example is antiretroviral therapy (ART), which is used to treat ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "http://ourworldindata.org/hiv-aids",
                        "children": [
                            {
                                "text": "HIV/AIDS",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": ". The benefits of these drugs were surprising. One of the first ART drugs discovered was azidothymidine, which had previously been abandoned as an ineffective treatment for cancer.{ref}National Research Council (U.S.) (1993). The social impact of AIDS in the United States. 4. Clinical Research and Drug Regulation. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "National Academy Press.",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "{/ref} The discovery that azidothymidine and other antiretroviral therapies worked, and their use worldwide, has prevented millions of deaths from HIV/AIDS, as the chart shows.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Discovering what works can save lives. But discovering what ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "doesn\u2019t",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " work can do the same. It means we can redirect our time and resources away from things that don't work, towards things that do.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Even when we already know that something works, understanding how ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "well",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " it works can help us make better decisions.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "An example for this is ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://ourworldindata.org/grapher/bcg-immunization-coverage-for-tb-among-1-year-olds?country=~OWID_WRL",
                        "children": [
                            {
                                "text": "the BCG vaccine",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": ", which reduces the risk of tuberculosis. The efficacy of this vaccine is different for different people around the world, and the reasons for this are unclear.{ref}Dockrell, H. M., & Smith, S. G. (2017). What Have We Learnt about BCG Vaccination in the Last 20 Years? ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Frontiers in Immunology",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "8",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", 1134. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.3389/fimmu.2017.01134",
                        "children": [
                            {
                                "text": "https://doi.org/10.3389/fimmu.2017.01134",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "spanType": "span-newline"
                    },
                    {
                        "text": "Mangtani, P., Abubakar, I., Ariti, C., Beynon, R., Pimpin, L., Fine, P. E. M., Rodrigues, L. C., Smith, P. G., Lipman, M., Whiting, P. F., & Sterne, J. A. (2014). Protection by BCG Vaccine Against Tuberculosis: A Systematic Review of Randomized Controlled Trials. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Clinical Infectious Diseases",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "58",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(4), 470\u2013480. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1093/cid/cit790%7B/ref",
                        "children": [
                            {
                                "text": "https://doi.org/10.1093/cid/cit790",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref} Still, knowing this is informative, because it tells us that more effort is needed to protect people against the disease in places where the benefit of the vaccine is low.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "If there was a reliable way to know about the effects of the measures we took, we could prioritize solutions that are most effective.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "So, how would we understand cause and effect without trials?",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "One way is through observation: we can observe what different people did and track their outcomes afterwards. But it's possible that the events that followed their actions were simply a coincidence, or that they would have happened anyway.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "The biggest challenge in trying to understand causes and effects is that we are only able to see one version of history.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "When someone makes a particular decision, we are able to see what follows, but we cannot see what would have happened if they had made a different decision. This is known as \u201cthe fundamental problem of causal inference.\"{ref}Holland, P. W. (1986). Statistics and Causal Inference. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Journal of the American Statistical Association",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "81",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(396), 945\u2013960. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1080/01621459.1986.10478354",
                        "children": [
                            {
                                "text": "https://doi.org/10.1080/01621459.1986.10478354",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "spanType": "span-newline"
                    },
                    {
                        "text": "Imbens, G. W., & Rubin, D. B. (2010). Rubin causal model. In ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Microeconometrics",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " (pp. 229\u2013241). Springer.",
                        "spanType": "span-simple-text"
                    },
                    {
                        "spanType": "span-newline"
                    },
                    {
                        "text": "Rubin, D. B. (1977). Assignment to Treatment Group on the Basis of a Covariate. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Journal of Educational Statistics",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "2",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(1), 1\u201326. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.3102/10769986002001001%7B/ref",
                        "children": [
                            {
                                "text": "https://doi.org/10.3102/10769986002001001",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref}",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "What this means is that it is impossible to predict the effects that an action will have for an individual person, but we can try to predict the effects it would have on average.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "url": "https://ourworldindata.org/grapher/hivaids-deaths-and-averted-due-to-art?country=~OWID_WRL",
                "type": "chart",
                "parseErrors": []
            },
            {
                "text": [
                    {
                        "text": "Why randomized controlled trials matter",
                        "spanType": "span-simple-text"
                    }
                ],
                "type": "heading",
                "level": 3,
                "parseErrors": []
            },
            {
                "left": [
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "Sometimes we do not need an RCT to identify causes and effects. The effect of smoking on lung cancer is one example, where scientists could be confident as early as the 1960s that the large increase in lung cancer rates could not have been caused by other factors.\u00a0",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "This was because there was already knowledge from many different lines of evidence.{ref}Hill, G., Millar, W., & Connelly, J. (2003). \u201cThe Great Debate\u201d: Smoking, Lung Cancer, and Cancer Epidemiology. ",
                                "spanType": "span-simple-text"
                            },
                            {
                                "children": [
                                    {
                                        "text": "Canadian Bulletin of Medical History",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "spanType": "span-italic"
                            },
                            {
                                "text": ", ",
                                "spanType": "span-simple-text"
                            },
                            {
                                "children": [
                                    {
                                        "text": "20",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "spanType": "span-italic"
                            },
                            {
                                "text": "(2), 367-386.{/ref} Experiments, biopsies and population studies all showed that cigarette smoke was associated with specific changes in lung tissue and with the incidence of lung cancer. The association was so large and consistent that it could not be explained by other factors.{ref}Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B., & Wynder, E. L. (1959). Smoking and Lung Cancer: Recent Evidence and a Discussion of Some Questions. ",
                                "spanType": "span-simple-text"
                            },
                            {
                                "children": [
                                    {
                                        "text": "JNCI: Journal of the National Cancer Institute",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "spanType": "span-italic"
                            },
                            {
                                "text": ". ",
                                "spanType": "span-simple-text"
                            },
                            {
                                "url": "https://doi.org/10.1093/jnci/22.1.173",
                                "children": [
                                    {
                                        "text": "https://doi.org/10.1093/jnci/22.1.173",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "spanType": "span-link"
                            },
                            {
                                "text": "{/ref}",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "Even now, smoking is estimated to cause a large proportion of cancer deaths in many countries, as you can see in the chart.",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "But in other situations, RCTs have made a huge difference to our understanding of the world. Prioritizing the drugs that were shown to be effective in trials saved lives, as we saw with the example of antiretroviral therapy to treat HIV/AIDS.",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "People sometimes refer to RCTs as the \"gold standard of evidence\", but it's more useful to recognize that their strengths emerge from a ",
                                "spanType": "span-simple-text"
                            },
                            {
                                "children": [
                                    {
                                        "text": "set of procedures",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "spanType": "span-italic"
                            },
                            {
                                "text": " that they aim to follow.",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    }
                ],
                "type": "sticky-right",
                "right": [
                    {
                        "url": "https://ourworldindata.org/grapher/share-of-cancer-deaths-attributed-to-tobacco",
                        "type": "chart",
                        "parseErrors": []
                    }
                ],
                "parseErrors": []
            },
            {
                "text": [
                    {
                        "text": "Why do RCTs matter?\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "type": "heading",
                "level": 3,
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "In my opinion, even though we do have lots of knowledge about various topics, these trials matter for three reasons. They matter because we may not know enough, because we can be wrong, and because we might see what we want to see.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "First, they matter when ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "we don't know",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ".\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Randomized controlled trials can illuminate our understanding of effects, especially when there is uncertainty around them. They can help resolve disagreements between experts.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "In some cases, scientists can use other methods to investigate these topics. But when there is insufficient knowledge, other methods may not be enough.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "This is because, in a typical study, scientists need to be able to account for the possibility that other factors are causing the outcome.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "They need to have knowledge about why an effect usually occurs and whether it will happen anyway. They need to know about which other risk factors can cause the outcome and what makes it likely that someone will receive the treatment. They also need to know how to measure these factors properly, and how to account for them.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "The second reason they matter is when ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "we are wrong",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ".\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Even when scientists think they know which risk factors affect the outcome, they might be incorrect. They might not account for the right risk factors or they might account for the wrong ones.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "In an RCT where scientists use randomization, blinding and concealment, they can minimize both of these problems. Despite people's risks before the trial, the reason that someone will receive the treatment is the random number they are given. The reason that participants in each group differ at the end of the study is the group they were randomized to.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Even if we don't know about other risk factors, or even if we are wrong about them, these procedures mean that we can still find out whether a treatment has an effect and how large the effect is.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "The third reason is when ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "we see what we want to see",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ".\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "When participants expect to feel different after a treatment, they might report that they feel different, even if they didn't receive the treatment at all.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "When scientists want to see people improve after a treatment, they might decide to allocate the treatment to healthier participants. They might measure their data differently or re-analyze their data until they can find an improvement. They might even decide against publishing their findings when they don't find a benefit.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "If scientists use concealment, blinding, and pre-registration, they can reduce these biases. They can protect research against their own prejudices and the expectations of participants. They can also defend scientific knowledge from poor incentives, such as the desires of scientists to hype up their findings and the incentives for pharmaceutical companies to claim that their drugs work.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "text": [
                    {
                        "text": "The layers of protection against bias",
                        "spanType": "span-simple-text"
                    }
                ],
                "type": "heading",
                "level": 2,
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Previously, I described how the strengths of RCTs emerge from the ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "set of procedures",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " that they aim to follow.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "In medicine, this commonly includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.{ref}Although these procedures are also used outside of medicine, it can be difficult to apply them elsewhere. For example, in a trial that tests the effectiveness of talking therapy, it would be known to the participants that they are receiving it; it may not be possible to find a placebo control version to disguise the procedure. Due to constraints in length and focus, I will not detail the advantages of intention-to-treat analysis or experimentation.{/ref} Each of these helps to protect the evidence against distortions, which can come from many sources. But they are not always enforced. Below, we will explore them in more detail.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "text": [
                    {
                        "text": "The control group gives us a comparison to see what would have happened otherwise",
                        "spanType": "span-simple-text"
                    }
                ],
                "type": "heading",
                "level": 3,
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "The most crucial element to understand causes and effects is a control group.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Let's use antidepressants as an example to see why we need them.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "If no one in the world received antidepressants, we wouldn\u2019t know what their effects were. But the opposite is also true. If ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "everyone",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " received antidepressants, we wouldn't know how things would be without them.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "In order to understand the effect of an antidepressant, we need to see which other outcomes are possible. The most important thing we need to understand effects is a ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "comparison",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " \u2013 we need some people who receive the treatment and some who don\u2019t. The people who don\u2019t could be our control group.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "An ideal control group should allow us to ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "control",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " the biases and measurement errors in a study completely.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "For example, different doctors may diagnose depression differently. If someone had symptoms of depression, then the chances they get diagnosed should be equal in the antidepressant group and the control group. If they weren't equal, we could mistake differences between the groups for an effect of antidepressants, even if they didn't have any effect.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "In fact, an ideal control group should allow us to control the total effects of ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "all of the other factors",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " that could affect people's mood, not just control for the biases and errors in studies.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "As an example, we know that the symptoms of depression tend to change over time, as I explained in ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://ourworldindata.org/what-is-depression",
                        "children": [
                            {
                                "text": "this earlier post",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": ".\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "You can see this in the chart. This shows that the symptoms of depression tend to decline over time, among people who are diagnosed with depression but not treated for it. This is measured by seeing how many patients would still meet the threshold for a diagnosis of depression later on.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Almost a quarter of patients with depression (23%) would no longer meet the threshold for depression after three months, despite receiving no treatment. Just over half (53%) would no longer meet the threshold after one year.{ref}Whiteford, H. A., Harris, M. G., McKeon, G., Baxter, A., Pennell, C., Barendregt, J. J., & Wang, J. (2013). Estimating remission from untreated major depression: A systematic review and meta-analysis. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Psychological Medicine",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "43",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(8), 1569\u20131585. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1017/S0033291712001717",
                        "children": [
                            {
                                "text": "https://doi.org/10.1017/S0033291712001717",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref} This change is known as \"regression to the mean.\"",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "If we didn\u2019t have a control group in our study, we might misattribute such an improvement to the antidepressant. We need a control group to know how much their symptoms would improve anyway.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "alt": "",
                "size": "wide",
                "type": "image",
                "filename": "khK-H11avMDGkTpCLAqCJIsIyAJSX63_Blm-15GL-0fkGYoDK-47Q1Ancs-i06algU5S3GRWTTeL4hNMQgfezt2_DlxwUm1kTykSx6oc3Unzbnz0UNaKUbGZdQF4fagJFIwrXvCu",
                "parseErrors": []
            },
            {
                "text": [
                    {
                        "text": "Placebos allow us to account for placebo effects",
                        "spanType": "span-simple-text"
                    }
                ],
                "type": "heading",
                "level": 3,
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Some types of controls are special because they resemble the treatment without actually being it \u2013 these controls are called placebos. An ideal placebo has all the qualities above and also allows us to account for \"placebo effects.\" In this case, the placebo effect refers to when people's moods improve from the mere ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "procedure",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " of receiving the antidepressant.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "For example, taking a pill might improve people's mood because they believe that taking a pill will give them some benefit, even if the pill does not actually contain any active ingredient.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "How large is the placebo effect?\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Some clinical trials have tried to estimate this by comparing patients who received a placebo to patients who received no treatment at all. Overall, these studies have found that placebo effects are small in clinical trials for many conditions, but are larger for physical treatments, such as acupuncture to treat pain.{ref}Hr\u00f3bjartsson, A., & G\u00f8tzsche, P. C. (2010). Placebo interventions for all clinical conditions. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Cochrane Database of Systematic Reviews",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ". https://doi.org/10.1002/14651858.CD003974.pub3 \u00a0",
                        "spanType": "span-simple-text"
                    },
                    {
                        "spanType": "span-newline"
                    },
                    {
                        "text": "Hr\u00f3bjartsson, A., & G\u00f8tzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "New England Journal of Medicine",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "344",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(21), 1594\u20131602. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1056/NEJM200105243442106",
                        "children": [
                            {
                                "text": "https://doi.org/10.1056/NEJM200105243442106",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref}\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Placebo effects tend to be larger when scientists are recording symptoms reported by patients, rather than when they are measuring something that can be verified by others, such as results from a blood test or death.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "For depression, the placebo effect is non-significant. Studies do not detect a difference in the moods of patients who receive a placebo and those who do not receive any treatment at all.{ref}Hr\u00f3bjartsson, A., & G\u00f8tzsche, P. C. (2010). Placebo interventions for all clinical conditions. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Cochrane Database of Systematic Reviews",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ". ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1002/14651858.CD003974.pub3",
                        "children": [
                            {
                                "text": "https://doi.org/10.1002/14651858.CD003974.pub3",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "spanType": "span-newline"
                    },
                    {
                        "text": "Hr\u00f3bjartsson, A., & G\u00f8tzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "New England Journal of Medicine",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "344",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(21), 1594\u20131602. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1056/NEJM200105243442106%7B/ref",
                        "children": [
                            {
                                "text": "https://doi.org/10.1056/NEJM200105243442106",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref} Instead, the placebo group serves as a typical control group, to show what would happen to these patients even if they did not receive treatment.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "text": [
                    {
                        "text": "Randomization ensures that there are no differences between control and treatment group apart from whether they received the treatment",
                        "spanType": "span-simple-text"
                    }
                ],
                "type": "heading",
                "level": 3,
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Before participants are enrolled in a trial, they might already have different levels of risk of developing the outcomes.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Let's look at an example to illustrate this point.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Statins are a type of drug commonly used to prevent stroke. But strokes are more common among people who use statins. Does that mean that statins ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "caused",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " an increase in the rates of stroke?\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "No \u2013 people who are prescribed statins are more likely to have cardiovascular problems to begin with, which increases the chances of having a stroke later on. When this is accounted for, researchers find that people who take statins are actually ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "less likely",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " to develop a stroke.{ref}Orkaby, A. R., Gaziano, J. M., Djousse, L., & Driver, J. A. (2017). Statins for Primary Prevention of Cardiovascular Events and Mortality in Older Men. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Journal of the American Geriatrics Society",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "65",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(11), 2362\u20132368. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1111/jgs.14993",
                        "children": [
                            {
                                "text": "https://doi.org/10.1111/jgs.14993",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "spanType": "span-newline"
                    },
                    {
                        "text": "Makihara, N., Kamouchi, M., Hata, J., Matsuo, R., Ago, T., Kuroda, J., Kuwashiro, T., Sugimori, H., & Kitazono, T. (2013). Statins and the risks of stroke recurrence and death after ischemic stroke: The Fukuoka Stroke Registry. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Atherosclerosis",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "231",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(2), 211\u2013215. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1016/j.atherosclerosis.2013.09.017",
                        "children": [
                            {
                                "text": "https://doi.org/10.1016/j.atherosclerosis.2013.09.017",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "spanType": "span-newline"
                    },
                    {
                        "text": "N\u00ed Chr\u00f3in\u00edn, D., Asplund, K., \u00c5sberg, S., Callaly, E., Cuadrado-Godia, E., D\u00edez-Tejedor, E., Di Napoli, M., Engelter, S. T., Furie, K. L., Giannopoulos, S., Gotto, A. M., Hannon, N., Jonsson, F., Kapral, M. K., Mart\u00ed-F\u00e0bregas, J., Mart\u00ednez-S\u00e1nchez, P., Milionis, H. J., Montaner, J., Muscari, A., \u2026 Kelly, P. J. (2013). Statin Therapy and Outcome After Ischemic Stroke: Systematic Review and Meta-Analysis of Observational Studies and Randomized Trials. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Stroke",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "44",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(2), 448\u2013456. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1161/STROKEAHA.112.668277",
                        "children": [
                            {
                                "text": "https://doi.org/10.1161/STROKEAHA.112.668277",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref}",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "If researchers simply compared the rates of stroke in those who used statins with those who did not, they would miss the fact that there were other differences between the two groups, which could have caused differences in their rates of stroke.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Important differences such as these \u2013 which affect people's likelihood of receiving the treatment (using statins) ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "and also",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " affect the outcome (the risk of a stroke) \u2013 are called \u2018confounders\u2019.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "But it can be difficult to know what all of these confounders might be. It can also be difficult to measure and account for them properly. In fact, scientists can actually ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "worsen",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " a study by accounting for the wrong factors.{ref}Cinelli, C., Forney, A., & Pearl, J. (2020). A crash course in good and bad controls. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Available at SSRN, 3689437:",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "url": "https://dx.doi.org/10.2139/ssrn.3689437",
                        "children": [
                            {
                                "text": "http://dx.doi.org/10.2139/ssrn.3689437",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref}",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "What happens when participants are randomized in a trial?\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Randomization is the procedure of allocating them into one of two groups at random.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "With randomization, the problems above are minimized: everyone has the possibility of receiving the treatment. Whether people receive the treatment is not determined by the risks they have, but whether they are randomly selected to receive the treatment.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "So, the overall risks of developing the outcome in one group become comparable to the risks in the other group.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Randomization means that it is not a problem when there are confounders that are not known or not measured. Researchers don't have to know about why or how the outcome usually occurs.{ref}Aronow, P., Robins, J. M., Saarinen, T., S\u00e4vje, F., & Sekhon, J. (2021). Nonparametric identification is not enough, but randomized controlled trials are. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "ArXiv Preprint ArXiv:2108.11342",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ".{/ref}",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "text": [
                    {
                        "text": "Concealment and blinding limit the biases of researchers and expectations of participants",
                        "spanType": "span-simple-text"
                    }
                ],
                "type": "heading",
                "level": 3,
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "In a clinical trial, participants or scientists might realize which groups they are assigned to. For example, the drug might smell, taste or look different from the placebo. Or it might have obvious benefits or different side effects compared to the placebo.{ref}Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., Cochrane Bias Methods Group, & Cochrane Statistical Methods Group. (2011). The Cochrane Collaboration\u2019s tool for assessing risk of bias in randomised trials. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "BMJ",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "343",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(oct18 2), d5928\u2013d5928.",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1136/bmj.d5928",
                        "children": [
                            {
                                "text": " https://doi.org/10.1136/bmj.d5928",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref}",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Concealment is a first step in preventing this: this is the procedure of preventing scientists from knowing which treatment people ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "will be",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " assigned to.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Blinding is a second step: this is the procedure of preventing participants and scientists from finding out which treatment group people ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "have been",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " assigned to.{ref}Schulz, K. F., Chalmers, I., & Altman, D. G. (2002). The Landscape and Lexicon of Blinding in Randomized Trials. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Annals of Internal Medicine",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "136",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(3), 254. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.7326/0003-4819-136-3-200202050-00022",
                        "children": [
                            {
                                "text": "https://doi.org/10.7326/0003-4819-136-3-200202050-00022",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref}\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "When blinding is incomplete, it can partly reverse the benefits of randomization. For example, in clinical trials for oral health, the benefits of treatments appear larger when patients and the scientists who assess their health are not blinded sufficiently.{ref}Saltaji, H. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "et al.",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": " Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "BMC Med Res Methodol",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "children": [
                            {
                                "text": "18",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-bold"
                    },
                    {
                        "text": ", 42 (2018).{/ref}\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "If randomization was maintained, the only reason that groups would differ on their outcomes was the treatment they received. However, if the treatments were not hidden from scientists and participants, other factors could cause differences between them.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Sometimes, blinding is not possible \u2013 there might not be something that is safe and closely resembles the treatment, which could be used as a placebo.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Fortunately, this does not necessarily mean that these trials cannot be useful. Researchers can measure verifiable outcomes (such as changes in blood levels or even deaths) to avoid some placebo effects.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "But even when blinding occurs, participants and researchers might still make different decisions, because of the effects of the treatment or placebo.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "For example, some participants might decide to withdraw from the trial or not follow the protocols of the trial closely. Similarly, scientists might guess which groups people are in and therefore treat them differently or measure their outcomes in a biased way.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "It's difficult to predict how this might affect the results of a trial. It could cause us to overestimate or underestimate the benefit of a treatment. For example, in the clinical trials for ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://ourworldindata.org/covid-vaccinations",
                        "children": [
                            {
                                "text": "Covid-19 vaccines",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": ", some participants may have guessed that they received the vaccine because they experienced side effects.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "So, they may have believed that they were more protected and took fewer precautions. This means they may have increased their risk of catching Covid-19. This would make it appear as if the vaccines gave less protection than they actually did: it would result in an underestimate of their efficacy.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "text": [
                    {
                        "text": "Preregistration allows us to hold researchers accountable to their original study plans",
                        "spanType": "span-simple-text"
                    }
                ],
                "type": "heading",
                "level": 3,
                "parseErrors": []
            },
            {
                "left": [
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "Some of the procedures we've explored so far are used to safeguard research against errors and biases that scientists can have. Pre-registration is another procedure that contributes to the same goal.\u00a0",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "After the data in a study is analyzed, scientists have some choice in which results they present to their colleagues and the wider community. This opens up research to the possibility of cherry-picking.",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "This problem unfortunately often arises in studies that are sponsored by industry. If a pharmaceutical company is testing their new drugs in trials, disappointing results can lead to financial losses. So, they may decide not to publish them.\u00a0",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "But this problem is not limited to trials conducted by pharmaceutical companies.",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "For many reasons, scientists may decide not to publish some of their studies. Or they might re-analyze their data in a different way to find more positive results. Even if scientists want to publish their findings, journals may decide not to publish them because they may be seen as disappointing, controversial or uninteresting.\u00a0",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "To counter these incentives, scientists can follow the practice of \"pre-registration.\" This is when they publicly declare which analyses they plan to do in advance of collecting the data.",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "In 2000, the United States Food and Drug Administration (FDA) established an online registry for the details of clinical trials. The FDA required that scientists who were studying therapies for serious diseases needed to provide some details of their study designs and the outcomes that they planned to investigate.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. ",
                                "spanType": "span-simple-text"
                            },
                            {
                                "children": [
                                    {
                                        "text": "Jama",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "spanType": "span-italic"
                            },
                            {
                                "text": ", ",
                                "spanType": "span-simple-text"
                            },
                            {
                                "children": [
                                    {
                                        "text": "307",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "spanType": "span-italic"
                            },
                            {
                                "text": "(17), 1861\u20131864.{/ref}",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "This requirement reduced the bias towards positive findings in published research. We see evidence of this in the chart, showing clinical trials funded by the National Heart, Lung and Blood Institute (NHLBI), which adopted this requirement. ",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    },
                    {
                        "type": "text",
                        "value": [
                            {
                                "text": "Before 2000 \u2013 when pre-registration of studies was not required \u2013 most candidate drugs to treat or prevent cardiovascular disease showed large benefits. But most trials published after 2000 showed no benefit.{ref}Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. ",
                                "spanType": "span-simple-text"
                            },
                            {
                                "children": [
                                    {
                                        "text": "PLOS ONE",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "spanType": "span-italic"
                            },
                            {
                                "text": ", ",
                                "spanType": "span-simple-text"
                            },
                            {
                                "children": [
                                    {
                                        "text": "10",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "spanType": "span-italic"
                            },
                            {
                                "text": "(8), e0132382. ",
                                "spanType": "span-simple-text"
                            },
                            {
                                "url": "https://doi.org/10.1371/journal.pone.0132382",
                                "children": [
                                    {
                                        "text": "https://doi.org/10.1371/journal.pone.0132382",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "spanType": "span-link"
                            },
                            {
                                "text": "{/ref}",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "parseErrors": []
                    }
                ],
                "type": "sticky-right",
                "right": [
                    {
                        "alt": "",
                        "size": "wide",
                        "type": "image",
                        "filename": "Efficacy-in-trials-before-and-after-registration-requirement2.jpg",
                        "parseErrors": []
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Over the last two decades, this practice was strengthened and expanded. In 2007, the FDA required that most approved trials must be registered when people are enrolled into the study. They introduced notices and civil penalties for not doing so. Now, similar requirements are in place in Europe, Canada and some other parts of the world.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Jama",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "307",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": "(17), 1861\u20131864. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1001/jama.2012.4230",
                        "children": [
                            {
                                "text": "https://doi.org/10.1001/jama.2012.4230",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref}",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Importantly, sponsors of clinical trials are required to share the results online after the trial is completed.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "But unfortunately many still violate these requirements. For example, less than half (40%) of trials that were conducted in the US since 2017 actually reported their results on time.{ref}DeVito, N. J., & Goldacre, B. (2021). Evaluation of Compliance With Legal Requirements Under the FDA Amendments Act of 2007 for Timely Registration of Clinical Trials, Data Verification, Delayed Reporting, and Trial Document Submission. JAMA Internal Medicine, 181(8), 1128. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1001/jamainternmed.2021.2036",
                        "children": [
                            {
                                "text": "https://doi.org/10.1001/jamainternmed.2021.2036",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref}\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Here, you can see patterns in reporting results in Europe. According to EU regulations, all clinical trials in the European Economic Area are required to report their results to the European Clinical Trials Register (EU-CTR) within a year of completing the trial. But reporting rates vary a lot by country. By January 2021, most clinical trials by non-commercial sponsors in the UK (94%) reported their results in time, while nearly none (4%) in France had.{ref}Note: Data is shown until January 2021 for trials. After the UK left the European Union in January 2021, clinical trials in the UK were then no longer required to report their results to the EU-CTR. Only data from trials by non-commercial sponsors is shown. This includes trials sponsored by institutions such as universities, hospitals, research foundations and so on.",
                        "spanType": "span-simple-text"
                    },
                    {
                        "spanType": "span-newline"
                    },
                    {
                        "text": "Dal-R\u00e9, R., Goldacre, B., Mahillo-Fern\u00e1ndez, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Journal of Clinical Epidemiology",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", S0895435621003577. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1016/j.jclinepi.2021.11.005",
                        "children": [
                            {
                                "text": "https://doi.org/10.1016/j.jclinepi.2021.11.005",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref}",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Although a large share of clinical trials fail to report their results when they are due, this has improved recently: only half (50%) of clinical trials by non-commercial sponsors reported their results in time in 2018, while more than two-thirds (69%) did in 2021.{ref}Dal-R\u00e9, R., Goldacre, B., Mahillo-Fern\u00e1ndez, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "children": [
                            {
                                "text": "Journal of Clinical Epidemiology",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    },
                    {
                        "text": ", S0895435621003577. ",
                        "spanType": "span-simple-text"
                    },
                    {
                        "url": "https://doi.org/10.1016/j.jclinepi.2021.11.005",
                        "children": [
                            {
                                "text": "https://doi.org/10.1016/j.jclinepi.2021.11.005",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-link"
                    },
                    {
                        "text": "{/ref}",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "left": [
                    {
                        "url": "https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-in-time-by-country?country=GBR~FRA~BEL~NLD~DEU~ITA~ESP~SWE~AUT~DNK",
                        "type": "chart",
                        "parseErrors": []
                    }
                ],
                "type": "sticky-right",
                "right": [
                    {
                        "url": "https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-within-a-year-over-time?country=~European+Economic+Area",
                        "type": "chart",
                        "parseErrors": []
                    }
                ],
                "parseErrors": []
            },
            {
                "text": [
                    {
                        "text": "Pulling the layers together",
                        "spanType": "span-simple-text"
                    }
                ],
                "type": "heading",
                "level": 3,
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Together, these procedures give us a far more reliable way to distinguish between what works and what doesn't \u2013 even when we don't know enough, when we're wrong and when we see what we want to see.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "Unfortunately, we have seen that many clinical trials do not follow them. They tend not to report the details of their methods or test whether their participants were blinded to what they received. These standards cannot just be expected from scientists \u2013 they require cooperation from funders and journals, and they need to be actively enforced with penalties and incentives.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "We've also seen that trials can still suffer from remaining problems: that participants may drop out of the study and not adhere to the treatment. And clinical trials tend to not share their data or the code that they used to analyze it.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "So, despite all the benefits they provide, we shouldn't see these layers as a fixed checklist. Just as some of them were introduced recently, they may still evolve in the future \u2013 open data sharing, registered reports and procedures to reduce dropouts may be next on the list.{ref}Chambers, C., & Tzavella, L. (2020). The past, present, and future of Registered Reports.{/ref}",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "The procedures used in these trials are not a silver bullet. But when they are upheld, they can make these trials a more powerful source of evidence for understanding cause and effect.\u00a0",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "text": "To make progress, we need to be able to understand the problems that affect us, their causes and solutions. Randomized controlled trials can give scientists a reliable way to collect evidence on these important questions, and give us the ability to make better decisions. At Our World in Data, this is why they matter most of all.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "text": [
                    {
                        "children": [
                            {
                                "text": "Keep reading on Our World in Data",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-italic"
                    }
                ],
                "type": "heading",
                "level": 4,
                "parseErrors": []
            },
            {
                "url": "https://ourworldindata.org/what-is-depression",
                "type": "prominent-link",
                "title": "",
                "description": "",
                "parseErrors": []
            },
            {
                "url": "https://ourworldindata.org/mental-health",
                "type": "prominent-link",
                "title": "",
                "description": "",
                "parseErrors": []
            },
            {
                "type": "text",
                "value": [
                    {
                        "children": [
                            {
                                "text": "Acknowledgements:",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "spanType": "span-bold"
                    },
                    {
                        "text": " I would like to thank Darren Dahly, Nathaniel Bechhofer, Hannah Ritchie and Max Roser for reading drafts of this post and their very helpful guidance and suggestions to improve it.",
                        "spanType": "span-simple-text"
                    }
                ],
                "parseErrors": []
            },
            {
                "type": "gray-section",
                "items": [
                    {
                        "text": [
                            {
                                "text": "Additional information",
                                "spanType": "span-simple-text"
                            }
                        ],
                        "type": "heading",
                        "level": 2,
                        "parseErrors": []
                    },
                    {
                        "type": "expandable-paragraph",
                        "items": [
                            {
                                "type": "text",
                                "value": [
                                    {
                                        "text": "Trials are useful to test new ideas or treatments, when people do not have enough knowledge to recommend them on a larger scale.\u00a0",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "parseErrors": []
                            },
                            {
                                "type": "text",
                                "value": [
                                    {
                                        "text": "But people also have ethical concerns surrounding trials of treatments that are already used by doctors. For one, they might believe that it isn't justified to give patients a placebo when a drug is available, even if it is unclear how well the drug works.\u00a0",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "parseErrors": []
                            },
                            {
                                "type": "text",
                                "value": [
                                    {
                                        "text": "How can we balance these concerns with the benefits of trials? When are trials important?",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "parseErrors": []
                            },
                            {
                                "type": "text",
                                "value": [
                                    {
                                        "text": "A leading view is that they are justified when there is disagreement or uncertainty about the benefits of an intervention.{ref}Wendler, D. The Ethics of Clinical Research. in The Stanford Encyclopedia of Philosophy (ed. Zalta, E. N.) (Metaphysics Research Lab, Stanford University, 2021).",
                                        "spanType": "span-simple-text"
                                    },
                                    {
                                        "spanType": "span-newline"
                                    },
                                    {
                                        "text": "London, A. J. Equipoise in Research: Integrating Ethics and Science in Human Research. JAMA 317, 525 (2017).{/ref}",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "parseErrors": []
                            },
                            {
                                "type": "text",
                                "value": [
                                    {
                                        "text": "Outside of a trial, some doctors might prescribe a particular treatment, while other doctors would not, because they believe that same treatment is ineffective. In this case, the options in a trial \u2013 an intervention or control\u00a0\u2013 would not be so different from what people could already encounter.\u00a0",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "parseErrors": []
                            },
                            {
                                "type": "text",
                                "value": [
                                    {
                                        "text": "When experts disagree on a topic, an RCT is useful because it can resolve these conflicts and inform us about which option is better.\u00a0",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "parseErrors": []
                            },
                            {
                                "type": "text",
                                "value": [
                                    {
                                        "text": "So, to make the most of trials, they should be planned in line with the questions that people have. For example, if we wanted to understand whether vaccines gave less protection to people who were immunocompromised, then we should plan a trial with enough participants with these conditions.",
                                        "spanType": "span-simple-text"
                                    }
                                ],
                                "parseErrors": []
                            }
                        ],
                        "parseErrors": []
                    }
                ],
                "parseErrors": []
            }
        ],
        "type": "article",
        "title": "Why randomized controlled trials matter and the procedures that strengthen them",
        "authors": [
            "Saloni Dattani"
        ],
        "excerpt": "Randomized controlled trials are a key tool to study cause and effect. Why do they matter and how do they work?",
        "dateline": "February 10, 2022",
        "subtitle": "Randomized controlled trials are a key tool to study cause and effect. Why do they matter and how do they work?",
        "sidebar-toc": false,
        "featured-image": "RCT-thumbnail-1.jpg"
    },
    "createdAt": "2022-02-07T11:56:11.000Z",
    "published": false,
    "updatedAt": "2023-04-12T12:57:29.000Z",
    "revisionId": null,
    "publishedAt": "2022-02-10T11:00:00.000Z",
    "relatedCharts": [],
    "publicationContext": "listed"
}

{
    "errors": [
        {
            "name": "unexpected wp component tag",
            "details": "Found unhandled wp:comment tag image"
        },
        {
            "name": "unexpected wp component tag",
            "details": "Found unhandled wp:comment tag image"
        },
        {
            "name": "unhandled html tag found",
            "details": "Encountered the unhandled tag hr"
        },
        {
            "name": "unexpected wp component tag",
            "details": "Found unhandled wp:comment tag separator"
        },
        {
            "name": "unhandled html tag found",
            "details": "Encountered the unhandled tag hr"
        },
        {
            "name": "unexpected wp component tag",
            "details": "Found unhandled wp:comment tag separator"
        }
    ],
    "numBlocks": 112,
    "numErrors": 6,
    "wpTagCounts": {
        "html": 5,
        "image": 2,
        "column": 12,
        "columns": 6,
        "heading": 13,
        "paragraph": 116,
        "separator": 2,
        "owid/summary": 1,
        "owid/prominent-link": 2,
        "owid/additional-information": 1
    },
    "htmlTagCounts": {
        "p": 116,
        "h3": 2,
        "h4": 10,
        "h5": 1,
        "hr": 2,
        "div": 18,
        "figure": 2,
        "iframe": 5
    }
}

2022-02-10 11:00:00

2024-02-16 14:22:53

1Ugd_lwpRxqPgf1-l4X0TWnJUBC7V-cYJEJTJWTj7KfU

[
    "Saloni Dattani"
]

Randomized controlled trials are a key tool to study cause and effect. Why do they matter and how do they work?

2022-02-07 11:56:11

2023-04-12 12:57:29

https://ourworldindata.org/wp-content/uploads/2022/02/RCT-thumbnail-1.jpg

{}

<Callout title="Summary"/> At Our World in Data, we bring attention to the world's largest problems. We explore what these problems are, why they matter and how large they are. Whenever possible, we try to explain why these problems exist, and how we might solve them. To make progress, we need to be able to identify real solutions and evaluate them carefully. But doing this well is not simple. It's difficult for scientists to collect good evidence on the costs and benefits of a new idea, policy, or treatment. And it's challenging for decision-makers to scrutinize the evidence that exists, to make better decisions. What we need are reliable ways to distinguish between ideas that work and those that don't. In this post, I will explain a crucial tool that helps us do this – randomized controlled trials (RCTs). We will see that RCTs matter for three reasons: when we don't know about the effects of interventions, when we don't know how to study them, and when scientific research is affected by biases. ### What are randomized controlled trials? To begin with, what are RCTs? These are experiments where people are given, at random, either an intervention (such as a drug) or a control and then followed up to see how they fare on various outcomes. RCTs are conducted by researchers around the world. In the map, you can see how many RCTs have ever been published in high-ranked medical journals, by the country where the first author was based. Over 18,000 of these were from the United States, but most countries have had fewer than a hundred.{ref}Catalá-López, F., Aleixandre-Benavent, R., Caulley, L., Hutton, B., Tabarés-Seisdedos, R., Moher, D., & Alonso-Arroyo, A. (2020). Global mapping of randomised trials related articles published in high-impact-factor medical journals: a cross-sectional analysis. _Trials_, _21_(1), 1-24.{/ref} RCTs have also become more common over time.{ref}Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., & Otte, W. M. (2021). The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. _PLOS Biology_, _19_(4), e3001162. [https://doi.org/10.1371/journal.pbio.3001162](https://doi.org/10.1371/journal.pbio.3001162){/ref} It's easy to take RCTs for granted, but these trials have transformed our understanding of cause and effect. They are a powerful tool to illuminate what is unknown or uncertain; to discern whether something works and how well it works. But it's also important to recognize that these trials are not always perfect, and understand why. The strengths of RCTs are subtle: they are powerful because of the set of procedures that they are expected to follow. This includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration. In this post, we will explore why these procedures matter – how each one adds a layer of protection against complications that scientists face when they do research. <Chart url="https://ourworldindata.org/grapher/total-number-of-rcts-published-in-high-ranked-medical-journals?tab=map&country=GBR~USA~IND~DNK~NGA~BOL~MEX~FRA~JPN~CHN"/> ### The fundamental problem of causal inference We make decisions based on our understanding of how things work – we try to predict the consequences of our actions. But understanding cause and effect is not just crucial for personal decisions: our knowledge of what works can have large consequences for people around us. An example is antiretroviral therapy (ART), which is used to treat [HIV/AIDS](http://ourworldindata.org/hiv-aids). The benefits of these drugs were surprising. One of the first ART drugs discovered was azidothymidine, which had previously been abandoned as an ineffective treatment for cancer.{ref}National Research Council (U.S.) (1993). The social impact of AIDS in the United States. 4. Clinical Research and Drug Regulation. _National Academy Press._{/ref} The discovery that azidothymidine and other antiretroviral therapies worked, and their use worldwide, has prevented millions of deaths from HIV/AIDS, as the chart shows. Discovering what works can save lives. But discovering what _doesn’t_ work can do the same. It means we can redirect our time and resources away from things that don't work, towards things that do. Even when we already know that something works, understanding how _well_ it works can help us make better decisions. An example for this is [the BCG vaccine](https://ourworldindata.org/grapher/bcg-immunization-coverage-for-tb-among-1-year-olds?country=~OWID_WRL), which reduces the risk of tuberculosis. The efficacy of this vaccine is different for different people around the world, and the reasons for this are unclear.{ref}Dockrell, H. M., & Smith, S. G. (2017). What Have We Learnt about BCG Vaccination in the Last 20 Years? _Frontiers in Immunology_, _8_, 1134. [https://doi.org/10.3389/fimmu.2017.01134](https://doi.org/10.3389/fimmu.2017.01134) Mangtani, P., Abubakar, I., Ariti, C., Beynon, R., Pimpin, L., Fine, P. E. M., Rodrigues, L. C., Smith, P. G., Lipman, M., Whiting, P. F., & Sterne, J. A. (2014). Protection by BCG Vaccine Against Tuberculosis: A Systematic Review of Randomized Controlled Trials. _Clinical Infectious Diseases_, _58_(4), 470–480. [https://doi.org/10.1093/cid/cit790](https://doi.org/10.1093/cid/cit790%7B/ref){/ref} Still, knowing this is informative, because it tells us that more effort is needed to protect people against the disease in places where the benefit of the vaccine is low. If there was a reliable way to know about the effects of the measures we took, we could prioritize solutions that are most effective. So, how would we understand cause and effect without trials? One way is through observation: we can observe what different people did and track their outcomes afterwards. But it's possible that the events that followed their actions were simply a coincidence, or that they would have happened anyway. The biggest challenge in trying to understand causes and effects is that we are only able to see one version of history. When someone makes a particular decision, we are able to see what follows, but we cannot see what would have happened if they had made a different decision. This is known as “the fundamental problem of causal inference."{ref}Holland, P. W. (1986). Statistics and Causal Inference. _Journal of the American Statistical Association_, _81_(396), 945–960. [https://doi.org/10.1080/01621459.1986.10478354](https://doi.org/10.1080/01621459.1986.10478354) Imbens, G. W., & Rubin, D. B. (2010). Rubin causal model. In _Microeconometrics_ (pp. 229–241). Springer. Rubin, D. B. (1977). Assignment to Treatment Group on the Basis of a Covariate. _Journal of Educational Statistics_, _2_(1), 1–26. [https://doi.org/10.3102/10769986002001001](https://doi.org/10.3102/10769986002001001%7B/ref){/ref} What this means is that it is impossible to predict the effects that an action will have for an individual person, but we can try to predict the effects it would have on average. <Chart url="https://ourworldindata.org/grapher/hivaids-deaths-and-averted-due-to-art?country=~OWID_WRL"/> ### Why randomized controlled trials matter Sometimes we do not need an RCT to identify causes and effects. The effect of smoking on lung cancer is one example, where scientists could be confident as early as the 1960s that the large increase in lung cancer rates could not have been caused by other factors. This was because there was already knowledge from many different lines of evidence.{ref}Hill, G., Millar, W., & Connelly, J. (2003). “The Great Debate”: Smoking, Lung Cancer, and Cancer Epidemiology. _Canadian Bulletin of Medical History_, _20_(2), 367-386.{/ref} Experiments, biopsies and population studies all showed that cigarette smoke was associated with specific changes in lung tissue and with the incidence of lung cancer. The association was so large and consistent that it could not be explained by other factors.{ref}Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B., & Wynder, E. L. (1959). Smoking and Lung Cancer: Recent Evidence and a Discussion of Some Questions. _JNCI: Journal of the National Cancer Institute_. [https://doi.org/10.1093/jnci/22.1.173](https://doi.org/10.1093/jnci/22.1.173){/ref} Even now, smoking is estimated to cause a large proportion of cancer deaths in many countries, as you can see in the chart. But in other situations, RCTs have made a huge difference to our understanding of the world. Prioritizing the drugs that were shown to be effective in trials saved lives, as we saw with the example of antiretroviral therapy to treat HIV/AIDS. People sometimes refer to RCTs as the "gold standard of evidence", but it's more useful to recognize that their strengths emerge from a _set of procedures_ that they aim to follow. <Chart url="https://ourworldindata.org/grapher/share-of-cancer-deaths-attributed-to-tobacco"/> ### Why do RCTs matter? In my opinion, even though we do have lots of knowledge about various topics, these trials matter for three reasons. They matter because we may not know enough, because we can be wrong, and because we might see what we want to see. First, they matter when _we don't know_. Randomized controlled trials can illuminate our understanding of effects, especially when there is uncertainty around them. They can help resolve disagreements between experts. In some cases, scientists can use other methods to investigate these topics. But when there is insufficient knowledge, other methods may not be enough. This is because, in a typical study, scientists need to be able to account for the possibility that other factors are causing the outcome. They need to have knowledge about why an effect usually occurs and whether it will happen anyway. They need to know about which other risk factors can cause the outcome and what makes it likely that someone will receive the treatment. They also need to know how to measure these factors properly, and how to account for them. The second reason they matter is when _we are wrong_. Even when scientists think they know which risk factors affect the outcome, they might be incorrect. They might not account for the right risk factors or they might account for the wrong ones. In an RCT where scientists use randomization, blinding and concealment, they can minimize both of these problems. Despite people's risks before the trial, the reason that someone will receive the treatment is the random number they are given. The reason that participants in each group differ at the end of the study is the group they were randomized to. Even if we don't know about other risk factors, or even if we are wrong about them, these procedures mean that we can still find out whether a treatment has an effect and how large the effect is. The third reason is when _we see what we want to see_. When participants expect to feel different after a treatment, they might report that they feel different, even if they didn't receive the treatment at all. When scientists want to see people improve after a treatment, they might decide to allocate the treatment to healthier participants. They might measure their data differently or re-analyze their data until they can find an improvement. They might even decide against publishing their findings when they don't find a benefit. If scientists use concealment, blinding, and pre-registration, they can reduce these biases. They can protect research against their own prejudices and the expectations of participants. They can also defend scientific knowledge from poor incentives, such as the desires of scientists to hype up their findings and the incentives for pharmaceutical companies to claim that their drugs work. ## The layers of protection against bias Previously, I described how the strengths of RCTs emerge from the _set of procedures_ that they aim to follow. In medicine, this commonly includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.{ref}Although these procedures are also used outside of medicine, it can be difficult to apply them elsewhere. For example, in a trial that tests the effectiveness of talking therapy, it would be known to the participants that they are receiving it; it may not be possible to find a placebo control version to disguise the procedure. Due to constraints in length and focus, I will not detail the advantages of intention-to-treat analysis or experimentation.{/ref} Each of these helps to protect the evidence against distortions, which can come from many sources. But they are not always enforced. Below, we will explore them in more detail. ### The control group gives us a comparison to see what would have happened otherwise The most crucial element to understand causes and effects is a control group. Let's use antidepressants as an example to see why we need them. If no one in the world received antidepressants, we wouldn’t know what their effects were. But the opposite is also true. If _everyone_ received antidepressants, we wouldn't know how things would be without them. In order to understand the effect of an antidepressant, we need to see which other outcomes are possible. The most important thing we need to understand effects is a _comparison_ – we need some people who receive the treatment and some who don’t. The people who don’t could be our control group. An ideal control group should allow us to _control_ the biases and measurement errors in a study completely. For example, different doctors may diagnose depression differently. If someone had symptoms of depression, then the chances they get diagnosed should be equal in the antidepressant group and the control group. If they weren't equal, we could mistake differences between the groups for an effect of antidepressants, even if they didn't have any effect. In fact, an ideal control group should allow us to control the total effects of _all of the other factors_ that could affect people's mood, not just control for the biases and errors in studies. As an example, we know that the symptoms of depression tend to change over time, as I explained in [this earlier post](https://ourworldindata.org/what-is-depression). You can see this in the chart. This shows that the symptoms of depression tend to decline over time, among people who are diagnosed with depression but not treated for it. This is measured by seeing how many patients would still meet the threshold for a diagnosis of depression later on. Almost a quarter of patients with depression (23%) would no longer meet the threshold for depression after three months, despite receiving no treatment. Just over half (53%) would no longer meet the threshold after one year.{ref}Whiteford, H. A., Harris, M. G., McKeon, G., Baxter, A., Pennell, C., Barendregt, J. J., & Wang, J. (2013). Estimating remission from untreated major depression: A systematic review and meta-analysis. _Psychological Medicine_, _43_(8), 1569–1585. [https://doi.org/10.1017/S0033291712001717](https://doi.org/10.1017/S0033291712001717){/ref} This change is known as "regression to the mean." If we didn’t have a control group in our study, we might misattribute such an improvement to the antidepressant. We need a control group to know how much their symptoms would improve anyway. <Image filename="khK-H11avMDGkTpCLAqCJIsIyAJSX63_Blm-15GL-0fkGYoDK-47Q1Ancs-i06algU5S3GRWTTeL4hNMQgfezt2_DlxwUm1kTykSx6oc3Unzbnz0UNaKUbGZdQF4fagJFIwrXvCu" alt=""/> ### Placebos allow us to account for placebo effects Some types of controls are special because they resemble the treatment without actually being it – these controls are called placebos. An ideal placebo has all the qualities above and also allows us to account for "placebo effects." In this case, the placebo effect refers to when people's moods improve from the mere _procedure_ of receiving the antidepressant. For example, taking a pill might improve people's mood because they believe that taking a pill will give them some benefit, even if the pill does not actually contain any active ingredient. How large is the placebo effect? Some clinical trials have tried to estimate this by comparing patients who received a placebo to patients who received no treatment at all. Overall, these studies have found that placebo effects are small in clinical trials for many conditions, but are larger for physical treatments, such as acupuncture to treat pain.{ref}Hróbjartsson, A., & Gøtzsche, P. C. (2010). Placebo interventions for all clinical conditions. _Cochrane Database of Systematic Reviews_. https://doi.org/10.1002/14651858.CD003974.pub3 Hróbjartsson, A., & Gøtzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. _New England Journal of Medicine_, _344_(21), 1594–1602. [https://doi.org/10.1056/NEJM200105243442106](https://doi.org/10.1056/NEJM200105243442106){/ref} Placebo effects tend to be larger when scientists are recording symptoms reported by patients, rather than when they are measuring something that can be verified by others, such as results from a blood test or death. For depression, the placebo effect is non-significant. Studies do not detect a difference in the moods of patients who receive a placebo and those who do not receive any treatment at all.{ref}Hróbjartsson, A., & Gøtzsche, P. C. (2010). Placebo interventions for all clinical conditions. _Cochrane Database of Systematic Reviews_. [https://doi.org/10.1002/14651858.CD003974.pub3](https://doi.org/10.1002/14651858.CD003974.pub3) Hróbjartsson, A., & Gøtzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. _New England Journal of Medicine_, _344_(21), 1594–1602. [https://doi.org/10.1056/NEJM200105243442106](https://doi.org/10.1056/NEJM200105243442106%7B/ref){/ref} Instead, the placebo group serves as a typical control group, to show what would happen to these patients even if they did not receive treatment. ### Randomization ensures that there are no differences between control and treatment group apart from whether they received the treatment Before participants are enrolled in a trial, they might already have different levels of risk of developing the outcomes. Let's look at an example to illustrate this point. Statins are a type of drug commonly used to prevent stroke. But strokes are more common among people who use statins. Does that mean that statins _caused_ an increase in the rates of stroke? No – people who are prescribed statins are more likely to have cardiovascular problems to begin with, which increases the chances of having a stroke later on. When this is accounted for, researchers find that people who take statins are actually _less likely_ to develop a stroke.{ref}Orkaby, A. R., Gaziano, J. M., Djousse, L., & Driver, J. A. (2017). Statins for Primary Prevention of Cardiovascular Events and Mortality in Older Men. _Journal of the American Geriatrics Society_, _65_(11), 2362–2368. [https://doi.org/10.1111/jgs.14993](https://doi.org/10.1111/jgs.14993) Makihara, N., Kamouchi, M., Hata, J., Matsuo, R., Ago, T., Kuroda, J., Kuwashiro, T., Sugimori, H., & Kitazono, T. (2013). Statins and the risks of stroke recurrence and death after ischemic stroke: The Fukuoka Stroke Registry. _Atherosclerosis_, _231_(2), 211–215. [https://doi.org/10.1016/j.atherosclerosis.2013.09.017](https://doi.org/10.1016/j.atherosclerosis.2013.09.017) Ní Chróinín, D., Asplund, K., Åsberg, S., Callaly, E., Cuadrado-Godia, E., Díez-Tejedor, E., Di Napoli, M., Engelter, S. T., Furie, K. L., Giannopoulos, S., Gotto, A. M., Hannon, N., Jonsson, F., Kapral, M. K., Martí-Fàbregas, J., Martínez-Sánchez, P., Milionis, H. J., Montaner, J., Muscari, A., … Kelly, P. J. (2013). Statin Therapy and Outcome After Ischemic Stroke: Systematic Review and Meta-Analysis of Observational Studies and Randomized Trials. _Stroke_, _44_(2), 448–456. [https://doi.org/10.1161/STROKEAHA.112.668277](https://doi.org/10.1161/STROKEAHA.112.668277){/ref} If researchers simply compared the rates of stroke in those who used statins with those who did not, they would miss the fact that there were other differences between the two groups, which could have caused differences in their rates of stroke. Important differences such as these – which affect people's likelihood of receiving the treatment (using statins) _and also_ affect the outcome (the risk of a stroke) – are called ‘confounders’. But it can be difficult to know what all of these confounders might be. It can also be difficult to measure and account for them properly. In fact, scientists can actually _worsen_ a study by accounting for the wrong factors.{ref}Cinelli, C., Forney, A., & Pearl, J. (2020). A crash course in good and bad controls. _Available at SSRN, 3689437:_[http://dx.doi.org/10.2139/ssrn.3689437](https://dx.doi.org/10.2139/ssrn.3689437){/ref} What happens when participants are randomized in a trial? Randomization is the procedure of allocating them into one of two groups at random. With randomization, the problems above are minimized: everyone has the possibility of receiving the treatment. Whether people receive the treatment is not determined by the risks they have, but whether they are randomly selected to receive the treatment. So, the overall risks of developing the outcome in one group become comparable to the risks in the other group. Randomization means that it is not a problem when there are confounders that are not known or not measured. Researchers don't have to know about why or how the outcome usually occurs.{ref}Aronow, P., Robins, J. M., Saarinen, T., Sävje, F., & Sekhon, J. (2021). Nonparametric identification is not enough, but randomized controlled trials are. _ArXiv Preprint ArXiv:2108.11342_.{/ref} ### Concealment and blinding limit the biases of researchers and expectations of participants In a clinical trial, participants or scientists might realize which groups they are assigned to. For example, the drug might smell, taste or look different from the placebo. Or it might have obvious benefits or different side effects compared to the placebo.{ref}Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., Cochrane Bias Methods Group, & Cochrane Statistical Methods Group. (2011). The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. _BMJ_, _343_(oct18 2), d5928–d5928.[ https://doi.org/10.1136/bmj.d5928](https://doi.org/10.1136/bmj.d5928){/ref} Concealment is a first step in preventing this: this is the procedure of preventing scientists from knowing which treatment people _will be_ assigned to. Blinding is a second step: this is the procedure of preventing participants and scientists from finding out which treatment group people _have been_ assigned to.{ref}Schulz, K. F., Chalmers, I., & Altman, D. G. (2002). The Landscape and Lexicon of Blinding in Randomized Trials. _Annals of Internal Medicine_, _136_(3), 254. [https://doi.org/10.7326/0003-4819-136-3-200202050-00022](https://doi.org/10.7326/0003-4819-136-3-200202050-00022){/ref} When blinding is incomplete, it can partly reverse the benefits of randomization. For example, in clinical trials for oral health, the benefits of treatments appear larger when patients and the scientists who assess their health are not blinded sufficiently.{ref}Saltaji, H. _et al._ Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions. _BMC Med Res Methodol_**18**, 42 (2018).{/ref} If randomization was maintained, the only reason that groups would differ on their outcomes was the treatment they received. However, if the treatments were not hidden from scientists and participants, other factors could cause differences between them. Sometimes, blinding is not possible – there might not be something that is safe and closely resembles the treatment, which could be used as a placebo. Fortunately, this does not necessarily mean that these trials cannot be useful. Researchers can measure verifiable outcomes (such as changes in blood levels or even deaths) to avoid some placebo effects. But even when blinding occurs, participants and researchers might still make different decisions, because of the effects of the treatment or placebo. For example, some participants might decide to withdraw from the trial or not follow the protocols of the trial closely. Similarly, scientists might guess which groups people are in and therefore treat them differently or measure their outcomes in a biased way. It's difficult to predict how this might affect the results of a trial. It could cause us to overestimate or underestimate the benefit of a treatment. For example, in the clinical trials for [Covid-19 vaccines](https://ourworldindata.org/covid-vaccinations), some participants may have guessed that they received the vaccine because they experienced side effects. So, they may have believed that they were more protected and took fewer precautions. This means they may have increased their risk of catching Covid-19. This would make it appear as if the vaccines gave less protection than they actually did: it would result in an underestimate of their efficacy. ### Preregistration allows us to hold researchers accountable to their original study plans Some of the procedures we've explored so far are used to safeguard research against errors and biases that scientists can have. Pre-registration is another procedure that contributes to the same goal. After the data in a study is analyzed, scientists have some choice in which results they present to their colleagues and the wider community. This opens up research to the possibility of cherry-picking. This problem unfortunately often arises in studies that are sponsored by industry. If a pharmaceutical company is testing their new drugs in trials, disappointing results can lead to financial losses. So, they may decide not to publish them. But this problem is not limited to trials conducted by pharmaceutical companies. For many reasons, scientists may decide not to publish some of their studies. Or they might re-analyze their data in a different way to find more positive results. Even if scientists want to publish their findings, journals may decide not to publish them because they may be seen as disappointing, controversial or uninteresting. To counter these incentives, scientists can follow the practice of "pre-registration." This is when they publicly declare which analyses they plan to do in advance of collecting the data. In 2000, the United States Food and Drug Administration (FDA) established an online registry for the details of clinical trials. The FDA required that scientists who were studying therapies for serious diseases needed to provide some details of their study designs and the outcomes that they planned to investigate.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. _Jama_, _307_(17), 1861–1864.{/ref} This requirement reduced the bias towards positive findings in published research. We see evidence of this in the chart, showing clinical trials funded by the National Heart, Lung and Blood Institute (NHLBI), which adopted this requirement. Before 2000 – when pre-registration of studies was not required – most candidate drugs to treat or prevent cardiovascular disease showed large benefits. But most trials published after 2000 showed no benefit.{ref}Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. _PLOS ONE_, _10_(8), e0132382. [https://doi.org/10.1371/journal.pone.0132382](https://doi.org/10.1371/journal.pone.0132382){/ref} <Image filename="Efficacy-in-trials-before-and-after-registration-requirement2.jpg" alt=""/> Over the last two decades, this practice was strengthened and expanded. In 2007, the FDA required that most approved trials must be registered when people are enrolled into the study. They introduced notices and civil penalties for not doing so. Now, similar requirements are in place in Europe, Canada and some other parts of the world.{ref}Dickersin, K., & Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. _Jama_, _307_(17), 1861–1864. [https://doi.org/10.1001/jama.2012.4230](https://doi.org/10.1001/jama.2012.4230){/ref} Importantly, sponsors of clinical trials are required to share the results online after the trial is completed. But unfortunately many still violate these requirements. For example, less than half (40%) of trials that were conducted in the US since 2017 actually reported their results on time.{ref}DeVito, N. J., & Goldacre, B. (2021). Evaluation of Compliance With Legal Requirements Under the FDA Amendments Act of 2007 for Timely Registration of Clinical Trials, Data Verification, Delayed Reporting, and Trial Document Submission. JAMA Internal Medicine, 181(8), 1128. [https://doi.org/10.1001/jamainternmed.2021.2036](https://doi.org/10.1001/jamainternmed.2021.2036){/ref} Here, you can see patterns in reporting results in Europe. According to EU regulations, all clinical trials in the European Economic Area are required to report their results to the European Clinical Trials Register (EU-CTR) within a year of completing the trial. But reporting rates vary a lot by country. By January 2021, most clinical trials by non-commercial sponsors in the UK (94%) reported their results in time, while nearly none (4%) in France had.{ref}Note: Data is shown until January 2021 for trials. After the UK left the European Union in January 2021, clinical trials in the UK were then no longer required to report their results to the EU-CTR. Only data from trials by non-commercial sponsors is shown. This includes trials sponsored by institutions such as universities, hospitals, research foundations and so on. Dal-Ré, R., Goldacre, B., Mahillo-Fernández, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. _Journal of Clinical Epidemiology_, S0895435621003577. [https://doi.org/10.1016/j.jclinepi.2021.11.005](https://doi.org/10.1016/j.jclinepi.2021.11.005){/ref} Although a large share of clinical trials fail to report their results when they are due, this has improved recently: only half (50%) of clinical trials by non-commercial sponsors reported their results in time in 2018, while more than two-thirds (69%) did in 2021.{ref}Dal-Ré, R., Goldacre, B., Mahillo-Fernández, I., & DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. _Journal of Clinical Epidemiology_, S0895435621003577. [https://doi.org/10.1016/j.jclinepi.2021.11.005](https://doi.org/10.1016/j.jclinepi.2021.11.005){/ref} <Chart url="https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-in-time-by-country?country=GBR~FRA~BEL~NLD~DEU~ITA~ESP~SWE~AUT~DNK"/> <Chart url="https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-within-a-year-over-time?country=~European+Economic+Area"/> ### Pulling the layers together Together, these procedures give us a far more reliable way to distinguish between what works and what doesn't – even when we don't know enough, when we're wrong and when we see what we want to see. Unfortunately, we have seen that many clinical trials do not follow them. They tend not to report the details of their methods or test whether their participants were blinded to what they received. These standards cannot just be expected from scientists – they require cooperation from funders and journals, and they need to be actively enforced with penalties and incentives. We've also seen that trials can still suffer from remaining problems: that participants may drop out of the study and not adhere to the treatment. And clinical trials tend to not share their data or the code that they used to analyze it. So, despite all the benefits they provide, we shouldn't see these layers as a fixed checklist. Just as some of them were introduced recently, they may still evolve in the future – open data sharing, registered reports and procedures to reduce dropouts may be next on the list.{ref}Chambers, C., & Tzavella, L. (2020). The past, present, and future of Registered Reports.{/ref} The procedures used in these trials are not a silver bullet. But when they are upheld, they can make these trials a more powerful source of evidence for understanding cause and effect. To make progress, we need to be able to understand the problems that affect us, their causes and solutions. Randomized controlled trials can give scientists a reliable way to collect evidence on these important questions, and give us the ability to make better decisions. At Our World in Data, this is why they matter most of all. #### _Keep reading on Our World in Data_ ### https://ourworldindata.org/what-is-depression ### https://ourworldindata.org/mental-health **Acknowledgements:** I would like to thank Darren Dahly, Nathaniel Bechhofer, Hannah Ritchie and Max Roser for reading drafts of this post and their very helpful guidance and suggestions to improve it. ## Additional information Trials are useful to test new ideas or treatments, when people do not have enough knowledge to recommend them on a larger scale. But people also have ethical concerns surrounding trials of treatments that are already used by doctors. For one, they might believe that it isn't justified to give patients a placebo when a drug is available, even if it is unclear how well the drug works. How can we balance these concerns with the benefits of trials? When are trials important? A leading view is that they are justified when there is disagreement or uncertainty about the benefits of an intervention.{ref}Wendler, D. The Ethics of Clinical Research. in The Stanford Encyclopedia of Philosophy (ed. Zalta, E. N.) (Metaphysics Research Lab, Stanford University, 2021). London, A. J. Equipoise in Research: Integrating Ethics and Science in Human Research. JAMA 317, 525 (2017).{/ref} Outside of a trial, some doctors might prescribe a particular treatment, while other doctors would not, because they believe that same treatment is ineffective. In this case, the options in a trial – an intervention or control – would not be so different from what people could already encounter. When experts disagree on a topic, an RCT is useful because it can resolve these conflicts and inform us about which option is better. So, to make the most of trials, they should be planned in line with the questions that people have. For example, if we wanted to understand whether vaccines gave less protection to people who were immunocompromised, then we should plan a trial with enough participants with these conditions.

{
    "id": 48378,
    "date": "2022-02-10T11:00:00",
    "guid": {
        "rendered": "https://owid.cloud/?p=48378"
    },
    "link": "https://owid.cloud/randomized-controlled-trials",
    "meta": {
        "owid_publication_context_meta_field": {
            "latest": true,
            "homepage": true,
            "immediate_newsletter": true
        }
    },
    "slug": "randomized-controlled-trials",
    "tags": [],
    "type": "post",
    "title": {
        "rendered": "Why randomized controlled trials matter and the procedures that strengthen them"
    },
    "_links": {
        "self": [
            {
                "href": "https://owid.cloud/wp-json/wp/v2/posts/48378"
            }
        ],
        "about": [
            {
                "href": "https://owid.cloud/wp-json/wp/v2/types/post"
            }
        ],
        "author": [
            {
                "href": "https://owid.cloud/wp-json/wp/v2/users/47",
                "embeddable": true
            }
        ],
        "curies": [
            {
                "href": "https://api.w.org/{rel}",
                "name": "wp",
                "templated": true
            }
        ],
        "replies": [
            {
                "href": "https://owid.cloud/wp-json/wp/v2/comments?post=48378",
                "embeddable": true
            }
        ],
        "wp:term": [
            {
                "href": "https://owid.cloud/wp-json/wp/v2/categories?post=48378",
                "taxonomy": "category",
                "embeddable": true
            },
            {
                "href": "https://owid.cloud/wp-json/wp/v2/tags?post=48378",
                "taxonomy": "post_tag",
                "embeddable": true
            }
        ],
        "collection": [
            {
                "href": "https://owid.cloud/wp-json/wp/v2/posts"
            }
        ],
        "wp:attachment": [
            {
                "href": "https://owid.cloud/wp-json/wp/v2/media?parent=48378"
            }
        ],
        "version-history": [
            {
                "href": "https://owid.cloud/wp-json/wp/v2/posts/48378/revisions",
                "count": 29
            }
        ],
        "wp:featuredmedia": [
            {
                "href": "https://owid.cloud/wp-json/wp/v2/media/48379",
                "embeddable": true
            }
        ],
        "predecessor-version": [
            {
                "id": 56713,
                "href": "https://owid.cloud/wp-json/wp/v2/posts/48378/revisions/56713"
            }
        ]
    },
    "author": 47,
    "format": "standard",
    "status": "publish",
    "sticky": false,
    "content": {
        "rendered": "\t<div class=\"wp-block-owid-summary\">\n\t\t<h2>Summary</h2>\n\t\t\n\n<p>In order to tackle the world&#8217;s largest problems, we need to be able to identify their causes and effective ways to solve them. How would we know about the effects of a new idea, treatment or policy? Answering this question can be difficult. Some research is poorly done and there are incentives for scientists and funders to exaggerate their findings.</p>\n\n\n\n<p>In this post, I look at a key tool to study causal effects reliably: randomized controlled trials (RCTs). What makes RCTs useful is the set of procedures they are expected to follow, such as placebo controls, concealment, randomization and pre-registration. These procedures are layers of protection against challenges that scientists usually face in research. </p>\n\n\n\n<p>To be able to read published research, we need to understand why these procedures matter and why they are not always followed in practice. With this knowledge, we have a greater ability to identify the causes and solutions to the largest problems that affect us.</p>\n\n\n\t</div>\n\n\n<p>At Our World in Data, we bring attention to the world&#8217;s largest problems. We explore what these problems are, why they matter and how large they are. Whenever possible, we try to explain why these problems exist, and how we might solve them.</p>\n\n\n\n<p>To make progress, we need to be able to identify real solutions and evaluate them carefully. But doing this well is not simple. It&#8217;s difficult for scientists to collect good evidence on the costs and benefits of a new idea, policy, or treatment. And it&#8217;s challenging for decision-makers to scrutinize the evidence that exists, to make better decisions.</p>\n\n\n\n<p>What we need are reliable ways to distinguish between ideas that work and those that don&#8217;t.</p>\n\n\n\n<p>In this post, I will explain a crucial tool that helps us do this \u2013 randomized controlled trials (RCTs). We will see that RCTs matter for three reasons: when we don&#8217;t know about the effects of interventions, when we don&#8217;t know how to study them, and when scientific research is affected by biases.</p>\n\n\n\n<h4>What are randomized controlled trials?</h4>\n\n\n\n<p>To begin with, what are RCTs? These are experiments where people are given, at random, either an intervention (such as a drug) or a control and then followed up to see how they fare on various outcomes.</p>\n\n\n\n<p>RCTs are conducted by researchers around the world. In the map, you can see how many RCTs have ever been published in high-ranked medical journals, by the country where the first author was based. Over 18,000 of these were from the United States, but most countries have had fewer than a hundred.{ref}Catal\u00e1-L\u00f3pez, F., Aleixandre-Benavent, R., Caulley, L., Hutton, B., Tabar\u00e9s-Seisdedos, R., Moher, D., &amp; Alonso-Arroyo, A. (2020). Global mapping of randomised trials related articles published in high-impact-factor medical journals: a cross-sectional analysis. <em>Trials</em>, <em>21</em>(1), 1-24.{/ref} RCTs have also become more common over time.{ref}Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., &amp; Otte, W. M. (2021). The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. <em>PLOS Biology</em>, <em>19</em>(4), e3001162. <a href=\"https://doi.org/10.1371/journal.pbio.3001162\">https://doi.org/10.1371/journal.pbio.3001162</a>{/ref}</p>\n\n\n\n<p>It&#8217;s easy to take RCTs for granted, but these trials have transformed our understanding of cause and effect. They are a powerful tool to illuminate what is unknown or uncertain; to discern whether something works and how well it works.</p>\n\n\n\n<p>But it&#8217;s also important to recognize that these trials are not always perfect, and understand why.</p>\n\n\n\n<p>The strengths of RCTs are subtle: they are powerful because of the set of procedures that they are expected to follow. This includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.</p>\n\n\n\n<p>In this post, we will explore why these procedures matter \u2013 how each one adds a layer of protection against complications that scientists face when they do research.&nbsp;</p>\n\n\n\n<iframe src=\"https://ourworldindata.org/grapher/total-number-of-rcts-published-in-high-ranked-medical-journals?tab=map&amp;country=GBR~USA~IND~DNK~NGA~BOL~MEX~FRA~JPN~CHN\" loading=\"lazy\" style=\"width: 100%; height: 600px; border: 0px none;\"></iframe>\n\n\n\n<h4>The fundamental problem of causal inference</h4>\n\n\n\n<p>We make decisions based on our understanding of how things work \u2013 we try to predict the consequences of our actions.&nbsp;</p>\n\n\n\n<p>But understanding cause and effect is not just crucial for personal decisions: our knowledge of what works can have large consequences for people around us.&nbsp;</p>\n\n\n\n<p>An example is antiretroviral therapy (ART), which is used to treat <a href=\"http://ourworldindata.org/hiv-aids\">HIV/AIDS</a>. The benefits of these drugs were surprising. One of the first ART drugs discovered was azidothymidine, which had previously been abandoned as an ineffective treatment for cancer.{ref}National Research Council (U.S.) (1993). The social impact of AIDS in the United States. 4. Clinical Research and Drug Regulation. <em>National Academy Press.</em>{/ref} The discovery that azidothymidine and other antiretroviral therapies worked, and their use worldwide, has prevented millions of deaths from HIV/AIDS, as the chart shows.</p>\n\n\n\n<p>Discovering what works can save lives. But discovering what <em>doesn\u2019t</em> work can do the same. It means we can redirect our time and resources away from things that don&#8217;t work, towards things that do.</p>\n\n\n\n<p>Even when we already know that something works, understanding how <em>well</em> it works can help us make better decisions.&nbsp;</p>\n\n\n\n<p>An example for this is <a href=\"https://ourworldindata.org/grapher/bcg-immunization-coverage-for-tb-among-1-year-olds?country=~OWID_WRL\">the BCG vaccine</a>, which reduces the risk of tuberculosis. The efficacy of this vaccine is different for different people around the world, and the reasons for this are unclear.{ref}Dockrell, H. M., &amp; Smith, S. G. (2017). What Have We Learnt about BCG Vaccination in the Last 20 Years? <em>Frontiers in Immunology</em>, <em>8</em>, 1134. <a href=\"https://doi.org/10.3389/fimmu.2017.01134\">https://doi.org/10.3389/fimmu.2017.01134</a> <br>Mangtani, P., Abubakar, I., Ariti, C., Beynon, R., Pimpin, L., Fine, P. E. M., Rodrigues, L. C., Smith, P. G., Lipman, M., Whiting, P. F., &amp; Sterne, J. A. (2014). Protection by BCG Vaccine Against Tuberculosis: A Systematic Review of Randomized Controlled Trials. <em>Clinical Infectious Diseases</em>, <em>58</em>(4), 470\u2013480. <a href=\"https://doi.org/10.1093/cid/cit790%7B/ref\">https://doi.org/10.1093/cid/cit790</a>{/ref} Still, knowing this is informative, because it tells us that more effort is needed to protect people against the disease in places where the benefit of the vaccine is low.</p>\n\n\n\n<p>If there was a reliable way to know about the effects of the measures we took, we could prioritize solutions that are most effective.</p>\n\n\n\n<p>So, how would we understand cause and effect without trials?</p>\n\n\n\n<p>One way is through observation: we can observe what different people did and track their outcomes afterwards. But it&#8217;s possible that the events that followed their actions were simply a coincidence, or that they would have happened anyway.</p>\n\n\n\n<p>The biggest challenge in trying to understand causes and effects is that we are only able to see one version of history.&nbsp;</p>\n\n\n\n<p>When someone makes a particular decision, we are able to see what follows, but we cannot see what would have happened if they had made a different decision. This is known as \u201cthe fundamental problem of causal inference.&#8221;{ref}Holland, P. W. (1986). Statistics and Causal Inference. <em>Journal of the American Statistical Association</em>, <em>81</em>(396), 945\u2013960. <a href=\"https://doi.org/10.1080/01621459.1986.10478354\">https://doi.org/10.1080/01621459.1986.10478354</a><br>Imbens, G. W., &amp; Rubin, D. B. (2010). Rubin causal model. In <em>Microeconometrics</em> (pp. 229\u2013241). Springer.<br>Rubin, D. B. (1977). Assignment to Treatment Group on the Basis of a Covariate. <em>Journal of Educational Statistics</em>, <em>2</em>(1), 1\u201326. <a href=\"https://doi.org/10.3102/10769986002001001%7B/ref\">https://doi.org/10.3102/10769986002001001</a>{/ref}</p>\n\n\n\n<p>What this means is that it is impossible to predict the effects that an action will have for an individual person, but we can try to predict the effects it would have on average.</p>\n\n\n\n<iframe src=\"https://ourworldindata.org/grapher/hivaids-deaths-and-averted-due-to-art?country=~OWID_WRL\" loading=\"lazy\" style=\"width: 100%; height: 600px; border: 0px none;\"></iframe>\n\n\n\n<h4>Why randomized controlled trials matter</h4>\n\n\n\n<div class=\"wp-block-columns\">\n<div class=\"wp-block-column\">\n<p>Sometimes we do not need an RCT to identify causes and effects. The effect of smoking on lung cancer is one example, where scientists could be confident as early as the 1960s that the large increase in lung cancer rates could not have been caused by other factors.&nbsp;</p>\n\n\n\n<p>This was because there was already knowledge from many different lines of evidence.{ref}Hill, G., Millar, W., &amp; Connelly, J. (2003). \u201cThe Great Debate\u201d: Smoking, Lung Cancer, and Cancer Epidemiology. <em>Canadian Bulletin of Medical History</em>, <em>20</em>(2), 367-386.{/ref} Experiments, biopsies and population studies all showed that cigarette smoke was associated with specific changes in lung tissue and with the incidence of lung cancer. The association was so large and consistent that it could not be explained by other factors.{ref}Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B., &amp; Wynder, E. L. (1959). Smoking and Lung Cancer: Recent Evidence and a Discussion of Some Questions. <em>JNCI: Journal of the National Cancer Institute</em>. <a href=\"https://doi.org/10.1093/jnci/22.1.173\">https://doi.org/10.1093/jnci/22.1.173</a>{/ref}</p>\n\n\n\n<p>Even now, smoking is estimated to cause a large proportion of cancer deaths in many countries, as you can see in the chart.</p>\n\n\n\n<p>But in other situations, RCTs have made a huge difference to our understanding of the world. Prioritizing the drugs that were shown to be effective in trials saved lives, as we saw with the example of antiretroviral therapy to treat HIV/AIDS.</p>\n\n\n\n<p>People sometimes refer to RCTs as the &#8220;gold standard of evidence&#8221;, but it&#8217;s more useful to recognize that their strengths emerge from a <em>set of procedures</em> that they aim to follow.</p>\n</div>\n\n\n\n<div class=\"wp-block-column\">\n<iframe src=\"https://ourworldindata.org/grapher/share-of-cancer-deaths-attributed-to-tobacco\" loading=\"lazy\" style=\"width: 100%; height: 600px; border: 0px none;\"></iframe>\n</div>\n</div>\n\n\n\n<div class=\"wp-block-columns\">\n<div class=\"wp-block-column\">\n<h4>Why do RCTs matter?&nbsp;</h4>\n\n\n\n<p>In my opinion, even though we do have lots of knowledge about various topics, these trials matter for three reasons. They matter because we may not know enough, because we can be wrong, and because we might see what we want to see.</p>\n\n\n\n<p>First, they matter when <em>we don&#8217;t know</em>.&nbsp;</p>\n\n\n\n<p>Randomized controlled trials can illuminate our understanding of effects, especially when there is uncertainty around them. They can help resolve disagreements between experts.</p>\n\n\n\n<p>In some cases, scientists can use other methods to investigate these topics. But when there is insufficient knowledge, other methods may not be enough.&nbsp;</p>\n\n\n\n<p>This is because, in a typical study, scientists need to be able to account for the possibility that other factors are causing the outcome.&nbsp;</p>\n\n\n\n<p>They need to have knowledge about why an effect usually occurs and whether it will happen anyway. They need to know about which other risk factors can cause the outcome and what makes it likely that someone will receive the treatment. They also need to know how to measure these factors properly, and how to account for them.</p>\n\n\n\n<p>The second reason they matter is when <em>we are wrong</em>.&nbsp;</p>\n\n\n\n<p>Even when scientists think they know which risk factors affect the outcome, they might be incorrect. They might not account for the right risk factors or they might account for the wrong ones.</p>\n\n\n\n<p>In an RCT where scientists use randomization, blinding and concealment, they can minimize both of these problems. Despite people&#8217;s risks before the trial, the reason that someone will receive the treatment is the random number they are given. The reason that participants in each group differ at the end of the study is the group they were randomized to.&nbsp;</p>\n\n\n\n<p>Even if we don&#8217;t know about other risk factors, or even if we are wrong about them, these procedures mean that we can still find out whether a treatment has an effect and how large the effect is.</p>\n\n\n\n<p>The third reason is when <em>we see what we want to see</em>.&nbsp;</p>\n\n\n\n<p>When participants expect to feel different after a treatment, they might report that they feel different, even if they didn&#8217;t receive the treatment at all.&nbsp;</p>\n\n\n\n<p>When scientists want to see people improve after a treatment, they might decide to allocate the treatment to healthier participants. They might measure their data differently or re-analyze their data until they can find an improvement. They might even decide against publishing their findings when they don&#8217;t find a benefit.</p>\n\n\n\n<p>If scientists use concealment, blinding, and pre-registration, they can reduce these biases. They can protect research against their own prejudices and the expectations of participants. They can also defend scientific knowledge from poor incentives, such as the desires of scientists to hype up their findings and the incentives for pharmaceutical companies to claim that their drugs work.</p>\n</div>\n\n\n\n<div class=\"wp-block-column\"></div>\n</div>\n\n\n\n<h3>The layers of protection against bias</h3>\n\n\n\n<p>Previously, I described how the strengths of RCTs emerge from the <em>set of procedures</em> that they aim to follow.</p>\n\n\n\n<p>In medicine, this commonly includes the use of controls, placebos, experimentation, randomization, concealment, blinding, intention-to-treat analysis, and pre-registration.{ref}Although these procedures are also used outside of medicine, it can be difficult to apply them elsewhere. For example, in a trial that tests the effectiveness of talking therapy, it would be known to the participants that they are receiving it; it may not be possible to find a placebo control version to disguise the procedure. Due to constraints in length and focus, I will not detail the advantages of intention-to-treat analysis or experimentation.{/ref} Each of these helps to protect the evidence against distortions, which can come from many sources. But they are not always enforced. Below, we will explore them in more detail.</p>\n\n\n\n<h4>The control group gives us a comparison to see what would have happened otherwise</h4>\n\n\n\n<p>The most crucial element to understand causes and effects is a control group.&nbsp;</p>\n\n\n\n<p>Let&#8217;s use antidepressants as an example to see why we need them.</p>\n\n\n\n<p>If no one in the world received antidepressants, we wouldn\u2019t know what their effects were. But the opposite is also true. If <em>everyone</em> received antidepressants, we wouldn&#8217;t know how things would be without them.&nbsp;</p>\n\n\n\n<p>In order to understand the effect of an antidepressant, we need to see which other outcomes are possible. The most important thing we need to understand effects is a <em>comparison</em> \u2013 we need some people who receive the treatment and some who don\u2019t. The people who don\u2019t could be our control group.</p>\n\n\n\n<p>An ideal control group should allow us to <em>control</em> the biases and measurement errors in a study completely.&nbsp;</p>\n\n\n\n<p>For example, different doctors may diagnose depression differently. If someone had symptoms of depression, then the chances they get diagnosed should be equal in the antidepressant group and the control group. If they weren&#8217;t equal, we could mistake differences between the groups for an effect of antidepressants, even if they didn&#8217;t have any effect.</p>\n\n\n\n<p>In fact, an ideal control group should allow us to control the total effects of <em>all of the other factors</em> that could affect people&#8217;s mood, not just control for the biases and errors in studies.&nbsp;</p>\n\n\n\n<p>As an example, we know that the symptoms of depression tend to change over time, as I explained in <a href=\"https://ourworldindata.org/what-is-depression\">this earlier post</a>.&nbsp;</p>\n\n\n\n<p>You can see this in the chart. This shows that the symptoms of depression tend to decline over time, among people who are diagnosed with depression but not treated for it. This is measured by seeing how many patients would still meet the threshold for a diagnosis of depression later on.&nbsp;</p>\n\n\n\n<p>Almost a quarter of patients with depression (23%) would no longer meet the threshold for depression after three months, despite receiving no treatment. Just over half (53%) would no longer meet the threshold after one year.{ref}Whiteford, H. A., Harris, M. G., McKeon, G., Baxter, A., Pennell, C., Barendregt, J. J., &amp; Wang, J. (2013). Estimating remission from untreated major depression: A systematic review and meta-analysis. <em>Psychological Medicine</em>, <em>43</em>(8), 1569\u20131585. <a href=\"https://doi.org/10.1017/S0033291712001717\">https://doi.org/10.1017/S0033291712001717</a>{/ref} This change is known as &#8220;regression to the mean.&#8221;</p>\n\n\n\n<p>If we didn\u2019t have a control group in our study, we might misattribute such an improvement to the antidepressant. We need a control group to know how much their symptoms would improve anyway.</p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https://lh5.googleusercontent.com/khK-H11avMDGkTpCLAqCJIsIyAJSX63_Blm-15GL-0fkGYoDK-47Q1Ancs-i06algU5S3GRWTTeL4hNMQgfezt2_DlxwUm1kTykSx6oc3Unzbnz0UNaKUbGZdQF4fagJFIwrXvCu\" alt=\"\"/></figure>\n\n\n\n<h4>Placebos allow us to account for placebo effects</h4>\n\n\n\n<p>Some types of controls are special because they resemble the treatment without actually being it \u2013 these controls are called placebos. An ideal placebo has all the qualities above and also allows us to account for &#8220;placebo effects.&#8221; In this case, the placebo effect refers to when people&#8217;s moods improve from the mere <em>procedure</em> of receiving the antidepressant.&nbsp;</p>\n\n\n\n<p>For example, taking a pill might improve people&#8217;s mood because they believe that taking a pill will give them some benefit, even if the pill does not actually contain any active ingredient.&nbsp;</p>\n\n\n\n<p>How large is the placebo effect?&nbsp;</p>\n\n\n\n<p>Some clinical trials have tried to estimate this by comparing patients who received a placebo to patients who received no treatment at all. Overall, these studies have found that placebo effects are small in clinical trials for many conditions, but are larger for physical treatments, such as acupuncture to treat pain.{ref}Hr\u00f3bjartsson, A., &amp; G\u00f8tzsche, P. C. (2010). Placebo interventions for all clinical conditions. <em>Cochrane Database of Systematic Reviews</em>. https://doi.org/10.1002/14651858.CD003974.pub3 &nbsp;<br>Hr\u00f3bjartsson, A., &amp; G\u00f8tzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. <em>New England Journal of Medicine</em>, <em>344</em>(21), 1594\u20131602. <a href=\"https://doi.org/10.1056/NEJM200105243442106\">https://doi.org/10.1056/NEJM200105243442106</a>{/ref}&nbsp;</p>\n\n\n\n<p>Placebo effects tend to be larger when scientists are recording symptoms reported by patients, rather than when they are measuring something that can be verified by others, such as results from a blood test or death.&nbsp;</p>\n\n\n\n<p>For depression, the placebo effect is non-significant. Studies do not detect a difference in the moods of patients who receive a placebo and those who do not receive any treatment at all.{ref}Hr\u00f3bjartsson, A., &amp; G\u00f8tzsche, P. C. (2010). Placebo interventions for all clinical conditions. <em>Cochrane Database of Systematic Reviews</em>. <a href=\"https://doi.org/10.1002/14651858.CD003974.pub3\">https://doi.org/10.1002/14651858.CD003974.pub3</a> &nbsp;<br>Hr\u00f3bjartsson, A., &amp; G\u00f8tzsche, P. C. (2001). Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment. <em>New England Journal of Medicine</em>, <em>344</em>(21), 1594\u20131602. <a href=\"https://doi.org/10.1056/NEJM200105243442106%7B/ref\">https://doi.org/10.1056/NEJM200105243442106</a>{/ref} Instead, the placebo group serves as a typical control group, to show what would happen to these patients even if they did not receive treatment.</p>\n\n\n\n<h4>Randomization ensures that there are no differences between control and treatment group apart from whether they received the treatment</h4>\n\n\n\n<p>Before participants are enrolled in a trial, they might already have different levels of risk of developing the outcomes.&nbsp;</p>\n\n\n\n<p>Let&#8217;s look at an example to illustrate this point.&nbsp;</p>\n\n\n\n<p>Statins are a type of drug commonly used to prevent stroke. But strokes are more common among people who use statins. Does that mean that statins <em>caused</em> an increase in the rates of stroke?&nbsp;</p>\n\n\n\n<p>No \u2013 people who are prescribed statins are more likely to have cardiovascular problems to begin with, which increases the chances of having a stroke later on. When this is accounted for, researchers find that people who take statins are actually <em>less likely</em> to develop a stroke.{ref}Orkaby, A. R., Gaziano, J. M., Djousse, L., &amp; Driver, J. A. (2017). Statins for Primary Prevention of Cardiovascular Events and Mortality in Older Men. <em>Journal of the American Geriatrics Society</em>, <em>65</em>(11), 2362\u20132368. <a href=\"https://doi.org/10.1111/jgs.14993\">https://doi.org/10.1111/jgs.14993</a><br>Makihara, N., Kamouchi, M., Hata, J., Matsuo, R., Ago, T., Kuroda, J., Kuwashiro, T., Sugimori, H., &amp; Kitazono, T. (2013). Statins and the risks of stroke recurrence and death after ischemic stroke: The Fukuoka Stroke Registry. <em>Atherosclerosis</em>, <em>231</em>(2), 211\u2013215. <a href=\"https://doi.org/10.1016/j.atherosclerosis.2013.09.017\">https://doi.org/10.1016/j.atherosclerosis.2013.09.017</a><br>N\u00ed Chr\u00f3in\u00edn, D., Asplund, K., \u00c5sberg, S., Callaly, E., Cuadrado-Godia, E., D\u00edez-Tejedor, E., Di Napoli, M., Engelter, S. T., Furie, K. L., Giannopoulos, S., Gotto, A. M., Hannon, N., Jonsson, F., Kapral, M. K., Mart\u00ed-F\u00e0bregas, J., Mart\u00ednez-S\u00e1nchez, P., Milionis, H. J., Montaner, J., Muscari, A., \u2026 Kelly, P. J. (2013). Statin Therapy and Outcome After Ischemic Stroke: Systematic Review and Meta-Analysis of Observational Studies and Randomized Trials. <em>Stroke</em>, <em>44</em>(2), 448\u2013456. <a href=\"https://doi.org/10.1161/STROKEAHA.112.668277\">https://doi.org/10.1161/STROKEAHA.112.668277</a>{/ref}</p>\n\n\n\n<p>If researchers simply compared the rates of stroke in those who used statins with those who did not, they would miss the fact that there were other differences between the two groups, which could have caused differences in their rates of stroke.&nbsp;</p>\n\n\n\n<p>Important differences such as these \u2013 which affect people&#8217;s likelihood of receiving the treatment (using statins) <em>and also</em> affect the outcome (the risk of a stroke) \u2013 are called \u2018confounders\u2019.</p>\n\n\n\n<p>But it can be difficult to know what all of these confounders might be. It can also be difficult to measure and account for them properly. In fact, scientists can actually <em>worsen</em> a study by accounting for the wrong factors.{ref}Cinelli, C., Forney, A., &amp; Pearl, J. (2020). A crash course in good and bad controls. <em>Available at SSRN, 3689437:</em> <a href=\"https://dx.doi.org/10.2139/ssrn.3689437\">http://dx.doi.org/10.2139/ssrn.3689437</a>{/ref}</p>\n\n\n\n<p>What happens when participants are randomized in a trial?&nbsp;</p>\n\n\n\n<p>Randomization is the procedure of allocating them into one of two groups at random.</p>\n\n\n\n<p>With randomization, the problems above are minimized: everyone has the possibility of receiving the treatment. Whether people receive the treatment is not determined by the risks they have, but whether they are randomly selected to receive the treatment.</p>\n\n\n\n<p>So, the overall risks of developing the outcome in one group become comparable to the risks in the other group.</p>\n\n\n\n<p>Randomization means that it is not a problem when there are confounders that are not known or not measured. Researchers don&#8217;t have to know about why or how the outcome usually occurs.{ref}Aronow, P., Robins, J. M., Saarinen, T., S\u00e4vje, F., &amp; Sekhon, J. (2021). Nonparametric identification is not enough, but randomized controlled trials are. <em>ArXiv Preprint ArXiv:2108.11342</em>.{/ref}</p>\n\n\n\n<h4>Concealment and blinding limit the biases of researchers and expectations of participants</h4>\n\n\n\n<p>In a clinical trial, participants or scientists might realize which groups they are assigned to. For example, the drug might smell, taste or look different from the placebo. Or it might have obvious benefits or different side effects compared to the placebo.{ref}Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., Cochrane Bias Methods Group, &amp; Cochrane Statistical Methods Group. (2011). The Cochrane Collaboration\u2019s tool for assessing risk of bias in randomised trials. <em>BMJ</em>, <em>343</em>(oct18 2), d5928\u2013d5928.<a href=\"https://doi.org/10.1136/bmj.d5928\"> https://doi.org/10.1136/bmj.d5928</a>{/ref}</p>\n\n\n\n<p>Concealment is a first step in preventing this: this is the procedure of preventing scientists from knowing which treatment people <em>will be</em> assigned to.&nbsp;</p>\n\n\n\n<p>Blinding is a second step: this is the procedure of preventing participants and scientists from finding out which treatment group people <em>have been</em> assigned to.{ref}Schulz, K. F., Chalmers, I., &amp; Altman, D. G. (2002). The Landscape and Lexicon of Blinding in Randomized Trials. <em>Annals of Internal Medicine</em>, <em>136</em>(3), 254. <a href=\"https://doi.org/10.7326/0003-4819-136-3-200202050-00022\">https://doi.org/10.7326/0003-4819-136-3-200202050-00022</a>{/ref}&nbsp;</p>\n\n\n\n<p>When blinding is incomplete, it can partly reverse the benefits of randomization. For example, in clinical trials for oral health, the benefits of treatments appear larger when patients and the scientists who assess their health are not blinded sufficiently.{ref}Saltaji, H. <em>et al.</em> Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions. <em>BMC Med Res Methodol</em> <strong>18</strong>, 42 (2018).{/ref}&nbsp;</p>\n\n\n\n<p>If randomization was maintained, the only reason that groups would differ on their outcomes was the treatment they received. However, if the treatments were not hidden from scientists and participants, other factors could cause differences between them.</p>\n\n\n\n<p>Sometimes, blinding is not possible \u2013 there might not be something that is safe and closely resembles the treatment, which could be used as a placebo.&nbsp;</p>\n\n\n\n<p>Fortunately, this does not necessarily mean that these trials cannot be useful. Researchers can measure verifiable outcomes (such as changes in blood levels or even deaths) to avoid some placebo effects.</p>\n\n\n\n<p>But even when blinding occurs, participants and researchers might still make different decisions, because of the effects of the treatment or placebo.&nbsp;</p>\n\n\n\n<p>For example, some participants might decide to withdraw from the trial or not follow the protocols of the trial closely. Similarly, scientists might guess which groups people are in and therefore treat them differently or measure their outcomes in a biased way.</p>\n\n\n\n<p>It&#8217;s difficult to predict how this might affect the results of a trial. It could cause us to overestimate or underestimate the benefit of a treatment. For example, in the clinical trials for <a href=\"https://ourworldindata.org/covid-vaccinations\">Covid-19 vaccines</a>, some participants may have guessed that they received the vaccine because they experienced side effects.&nbsp;</p>\n\n\n\n<p>So, they may have believed that they were more protected and took fewer precautions. This means they may have increased their risk of catching Covid-19. This would make it appear as if the vaccines gave less protection than they actually did: it would result in an underestimate of their efficacy.</p>\n\n\n\n<h4>Preregistration allows us to hold researchers accountable to their original study plans</h4>\n\n\n\n<div class=\"wp-block-columns\">\n<div class=\"wp-block-column\">\n<p>Some of the procedures we&#8217;ve explored so far are used to safeguard research against errors and biases that scientists can have. Pre-registration is another procedure that contributes to the same goal.&nbsp;</p>\n\n\n\n<p>After the data in a study is analyzed, scientists have some choice in which results they present to their colleagues and the wider community. This opens up research to the possibility of cherry-picking.</p>\n\n\n\n<p>This problem unfortunately often arises in studies that are sponsored by industry. If a pharmaceutical company is testing their new drugs in trials, disappointing results can lead to financial losses. So, they may decide not to publish them.&nbsp;</p>\n\n\n\n<p>But this problem is not limited to trials conducted by pharmaceutical companies.</p>\n\n\n\n<p>For many reasons, scientists may decide not to publish some of their studies. Or they might re-analyze their data in a different way to find more positive results. Even if scientists want to publish their findings, journals may decide not to publish them because they may be seen as disappointing, controversial or uninteresting.&nbsp;</p>\n\n\n\n<p>To counter these incentives, scientists can follow the practice of &#8220;pre-registration.&#8221; This is when they publicly declare which analyses they plan to do in advance of collecting the data.</p>\n\n\n\n<p>In 2000, the United States Food and Drug Administration (FDA) established an online registry for the details of clinical trials. The FDA required that scientists who were studying therapies for serious diseases needed to provide some details of their study designs and the outcomes that they planned to investigate.{ref}Dickersin, K., &amp; Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. <em>Jama</em>, <em>307</em>(17), 1861\u20131864.{/ref}</p>\n\n\n\n<p>This requirement reduced the bias towards positive findings in published research. We see evidence of this in the chart, showing clinical trials funded by the National Heart, Lung and Blood Institute (NHLBI), which adopted this requirement. </p>\n\n\n\n<p>Before 2000 \u2013 when pre-registration of studies was not required \u2013 most candidate drugs to treat or prevent cardiovascular disease showed large benefits. But most trials published after 2000 showed no benefit.{ref}Kaplan, R. M., &amp; Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. <em>PLOS ONE</em>, <em>10</em>(8), e0132382. <a href=\"https://doi.org/10.1371/journal.pone.0132382\">https://doi.org/10.1371/journal.pone.0132382</a>{/ref}</p>\n</div>\n\n\n\n<div class=\"wp-block-column\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" width=\"1414\" height=\"1169\" src=\"https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2.jpg\" alt=\"\" class=\"wp-image-48462\" srcset=\"https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2.jpg 1414w, https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2-400x331.jpg 400w, https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2-665x550.jpg 665w, https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2-150x124.jpg 150w, https://owid.cloud/app/uploads/2022/02/Efficacy-in-trials-before-and-after-registration-requirement2-768x635.jpg 768w\" sizes=\"(max-width: 1414px) 100vw, 1414px\" /></figure>\n</div>\n</div>\n\n\n\n<div class=\"wp-block-columns is-style-sticky-right\">\n<div class=\"wp-block-column\">\n<p>Over the last two decades, this practice was strengthened and expanded. In 2007, the FDA required that most approved trials must be registered when people are enrolled into the study. They introduced notices and civil penalties for not doing so. Now, similar requirements are in place in Europe, Canada and some other parts of the world.{ref}Dickersin, K., &amp; Rennie, D. (2012). The evolution of trial registries and their use to assess the clinical trial enterprise. <em>Jama</em>, <em>307</em>(17), 1861\u20131864. <a href=\"https://doi.org/10.1001/jama.2012.4230\">https://doi.org/10.1001/jama.2012.4230</a>{/ref}</p>\n\n\n\n<p>Importantly, sponsors of clinical trials are required to share the results online after the trial is completed.&nbsp;</p>\n\n\n\n<p>But unfortunately many still violate these requirements. For example, less than half (40%) of trials that were conducted in the US since 2017 actually reported their results on time.{ref}DeVito, N. J., &amp; Goldacre, B. (2021). Evaluation of Compliance With Legal Requirements Under the FDA Amendments Act of 2007 for Timely Registration of Clinical Trials, Data Verification, Delayed Reporting, and Trial Document Submission. JAMA Internal Medicine, 181(8), 1128. <a href=\"https://doi.org/10.1001/jamainternmed.2021.2036\">https://doi.org/10.1001/jamainternmed.2021.2036</a>{/ref}&nbsp;</p>\n\n\n\n<p>Here, you can see patterns in reporting results in Europe. According to EU regulations, all clinical trials in the European Economic Area are required to report their results to the European Clinical Trials Register (EU-CTR) within a year of completing the trial. But reporting rates vary a lot by country. By January 2021, most clinical trials by non-commercial sponsors in the UK (94%) reported their results in time, while nearly none (4%) in France had.{ref}Note: Data is shown until January 2021 for trials. After the UK left the European Union in January 2021, clinical trials in the UK were then no longer required to report their results to the EU-CTR. Only data from trials by non-commercial sponsors is shown. This includes trials sponsored by institutions such as universities, hospitals, research foundations and so on.<br>Dal-R\u00e9, R., Goldacre, B., Mahillo-Fern\u00e1ndez, I., &amp; DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. <em>Journal of Clinical Epidemiology</em>, S0895435621003577. <a href=\"https://doi.org/10.1016/j.jclinepi.2021.11.005\">https://doi.org/10.1016/j.jclinepi.2021.11.005</a>{/ref}</p>\n\n\n\n<p>Although a large share of clinical trials fail to report their results when they are due, this has improved recently: only half (50%) of clinical trials by non-commercial sponsors reported their results in time in 2018, while more than two-thirds (69%) did in 2021.{ref}Dal-R\u00e9, R., Goldacre, B., Mahillo-Fern\u00e1ndez, I., &amp; DeVito, N. J. (2021). European non-commercial sponsors showed substantial variation in results reporting to the EU trial registry. <em>Journal of Clinical Epidemiology</em>, S0895435621003577. <a href=\"https://doi.org/10.1016/j.jclinepi.2021.11.005\">https://doi.org/10.1016/j.jclinepi.2021.11.005</a>{/ref}</p>\n</div>\n\n\n\n<div class=\"wp-block-column\"></div>\n</div>\n\n\n\n<div class=\"wp-block-columns is-style-side-by-side\">\n<div class=\"wp-block-column\" style=\"flex-basis:50%\">\n<iframe src=\"https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-in-time-by-country?country=GBR~FRA~BEL~NLD~DEU~ITA~ESP~SWE~AUT~DNK\" loading=\"lazy\" style=\"width: 100%; height: 600px; border: 0px none;\"></iframe>\n</div>\n\n\n\n<div class=\"wp-block-column\" style=\"flex-basis:50%\">\n<iframe src=\"https://ourworldindata.org/grapher/share-of-clinical-trials-that-report-results-within-a-year-over-time?country=~European+Economic+Area\" loading=\"lazy\" style=\"width: 100%; height: 600px; border: 0px none;\"></iframe>\n</div>\n</div>\n\n\n\n<h4>Pulling the layers together</h4>\n\n\n\n<p>Together, these procedures give us a far more reliable way to distinguish between what works and what doesn&#8217;t \u2013 even when we don&#8217;t know enough, when we&#8217;re wrong and when we see what we want to see.</p>\n\n\n\n<p>Unfortunately, we have seen that many clinical trials do not follow them. They tend not to report the details of their methods or test whether their participants were blinded to what they received. These standards cannot just be expected from scientists \u2013 they require cooperation from funders and journals, and they need to be actively enforced with penalties and incentives.</p>\n\n\n\n<p>We&#8217;ve also seen that trials can still suffer from remaining problems: that participants may drop out of the study and not adhere to the treatment. And clinical trials tend to not share their data or the code that they used to analyze it.</p>\n\n\n\n<p>So, despite all the benefits they provide, we shouldn&#8217;t see these layers as a fixed checklist. Just as some of them were introduced recently, they may still evolve in the future \u2013 open data sharing, registered reports and procedures to reduce dropouts may be next on the list.{ref}Chambers, C., &amp; Tzavella, L. (2020). The past, present, and future of Registered Reports.{/ref}</p>\n\n\n\n<p>The procedures used in these trials are not a silver bullet. But when they are upheld, they can make these trials a more powerful source of evidence for understanding cause and effect.&nbsp;</p>\n\n\n\n<p>To make progress, we need to be able to understand the problems that affect us, their causes and solutions. Randomized controlled trials can give scientists a reliable way to collect evidence on these important questions, and give us the ability to make better decisions. At Our World in Data, this is why they matter most of all.</p>\n\n\n\n<hr class=\"wp-block-separator\"/>\n\n\n\n<h5><em>Keep reading on Our World in Data</em></h5>\n\n\n    <block type=\"prominent-link\" style=\"is-style-thin\">\n        <link-url>https://ourworldindata.org/what-is-depression</link-url>\n        <title></title>\n        <content></content>\n        <figure></figure>\n    </block>\n\n    <block type=\"prominent-link\" style=\"is-style-thin\">\n        <link-url>https://ourworldindata.org/mental-health</link-url>\n        <title></title>\n        <content></content>\n        <figure></figure>\n    </block>\n\n\n<p><strong>Acknowledgements:</strong> I would like to thank Darren Dahly, Nathaniel Bechhofer, Hannah Ritchie and Max Roser for reading drafts of this post and their very helpful guidance and suggestions to improve it.</p>\n\n\n\n<hr class=\"wp-block-separator\"/>\n\n\n\t<block type=\"additional-information\" default-open=\"false\">\n\t\t<content>\n\n<h3>Appendix: When are randomized controls ethical?</h3>\n\n\n\n<div class=\"wp-block-columns is-style-sticky-right\">\n<div class=\"wp-block-column\">\n<p>Trials are useful to test new ideas or treatments, when people do not have enough knowledge to recommend them on a larger scale.&nbsp;</p>\n\n\n\n<p>But people also have ethical concerns surrounding trials of treatments that are already used by doctors. For one, they might believe that it isn&#8217;t justified to give patients a placebo when a drug is available, even if it is unclear how well the drug works.&nbsp;</p>\n\n\n\n<p>How can we balance these concerns with the benefits of trials? When are trials important?</p>\n\n\n\n<p>A leading view is that they are justified when there is disagreement or uncertainty about the benefits of an intervention.{ref}Wendler, D. The Ethics of Clinical Research. in The Stanford Encyclopedia of Philosophy (ed. Zalta, E. N.) (Metaphysics Research Lab, Stanford University, 2021).<br>London, A. J. Equipoise in Research: Integrating Ethics and Science in Human Research. JAMA 317, 525 (2017).{/ref}</p>\n\n\n\n<p>Outside of a trial, some doctors might prescribe a particular treatment, while other doctors would not, because they believe that same treatment is ineffective. In this case, the options in a trial \u2013 an intervention or control&nbsp;\u2013 would not be so different from what people could already encounter.&nbsp;</p>\n\n\n\n<p>When experts disagree on a topic, an RCT is useful because it can resolve these conflicts and inform us about which option is better.&nbsp;</p>\n\n\n\n<p>So, to make the most of trials, they should be planned in line with the questions that people have. For example, if we wanted to understand whether vaccines gave less protection to people who were immunocompromised, then we should plan a trial with enough participants with these conditions.</p>\n\n\n\n<p></p>\n</div>\n\n\n\n<div class=\"wp-block-column\"></div>\n</div>\n\n</content>\n\t</block>\n\n\n<p></p>\n",
        "protected": false
    },
    "excerpt": {
        "rendered": "Randomized controlled trials are a key tool to study cause and effect. Why do they matter and how do they work?",
        "protected": false
    },
    "date_gmt": "2022-02-10T11:00:00",
    "modified": "2023-04-12T13:57:29",
    "template": "",
    "categories": [
        46,
        1
    ],
    "ping_status": "closed",
    "authors_name": [
        "Saloni Dattani"
    ],
    "modified_gmt": "2023-04-12T12:57:29",
    "comment_status": "closed",
    "featured_media": 48379,
    "featured_media_paths": {
        "thumbnail": "/app/uploads/2022/02/RCT-thumbnail-1-150x53.jpg",
        "medium_large": "/app/uploads/2022/02/RCT-thumbnail-1-768x269.jpg"
    }
}

Links from other tables

2 rows from post_id in post_tags
32 rows from sourceId in posts_links
0 rows from wordpressId in posts_unified
29 rows from postId in post_links
6 rows from postId in post_charts
0 rows from postId in post_broken_chart_links