Sometimes Randomized Controlled Trials are not the golden trophies of science that some say they are. (Like yesterday’s Ivermectin study). While some say that nothing less than an RCT will do, actually, sometimes it’s just silly to do one. And all RCT’s will wallow on the rocks of confirmation bias if they try to study something people already know the answer to.
For a long time there were no RCT studies showing benefits from parachutes — yet people continued to use them. It wasn’t until 2018 that a paper was finally published, surprisingly showing that there was no statistical difference in jumping with or without one from an average altitude of 4,000m.
This satirical paper makes real points about the flaws of the hallowed RCT’s:
- It’s easy to design an RCT to show the opposite of the truth.
- Reading the abstract of a paper is rarely enough. Reading a news headline, even less so. Details are everything.
- RCT’s are not random if the test subjects have opinions and can self-select themselves out of the experiment.
These authors are black-belt professionals at medico-lingo. Those that appreciate the wit will want to read the whole thing rather than just the snippets below (I really can’t do it justice):
Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial
Robert Yeh et al, 2018, British Medical Journal
Parachutes are routinely used to prevent death or major traumatic injury among individuals jumping from aircraft. However, evidence supporting the efficacy of parachutes is weak and guideline recommendations for their use are principally based on biological plausibility and expert opinion. Despite this widely held yet unsubstantiated belief of efficacy, many studies of parachutes have suggested injuries related to their use in both military and recreational settings, and parachutist injuries are formally recognized in the World Health Organization’s ICD-10 (international classification of diseases, 10th revision). This could raise concerns for supporters of evidence-based medicine, because numerous medical interventions believed to be useful have ultimately failed to show efficacy when subjected to properly executed randomized clinical trials.
The researchers were diligent at collecting data which was essentially useless:
Participants were randomized to wear either a parachute (National 360, National Parachute industries …) or an empty backpack (The North Face, Inc, Alameda, CA…). The interventions were not blinded to either participants or study investigators.
We collected data on basic demographic characteristics during screening by using paper forms or the survey app. Characteristics included age, sex, ethnic group, height, and weight. We also collected information on participants’ medical history including a history of broken bones, acrophobia (fear of heights), previous parachute use, family history of parachute use, and frequent flier status.
At the time of each jump, researchers recorded the altitude and velocity of the aircraft, and conducted a follow-up interview with each participant to ascertain vital status and to record any injuries sustained from the free fall within five minutes of impact with the ground, and again at 30 days after impact.
Details are everything:
Results: Parachute use did not significantly reduce death or major injury (0% for parachute v 0% for control; P>0.9). This finding was consistent across multiple subgroups. Compared with individuals screened but not enrolled, participants included in the study were on aircraft at significantly lower altitude (mean of 0.6 m for participants v mean of 9146 m for non-participants; P<0.001) and lower velocity (mean of 0 km/h v mean of 800 km/h; P<0.001).
So the control group with parachutes jumped from 9,000m at 800 km per hour, and participants were more likely to have jumped with backpacks from 0.6m and 0 kilometers per hour.
UPDATE: Before anyone criticizes the trial design, know that the researchers boarded a commercial passenger jet and tried to recruit people for the trial for the 9000m jump with and without a parachute, but were unable to get any participants to sign on to that randomized intervention. However they were able to recruit participants for 0.6m jump with and without said intervention.
Obviously the design of the experiment was entirely fair and even handed, it was just that the participants held strong preconceived and prejudicial conclusions, without evidence, and selected themselves out of the experiment.
The primary efficacy analysis tested the hypothesis that parachute use is superior to the control in preventing death and major traumatic injury. Based on an assumption of an average jump altitude of 4000 meters (typical of skydiving) and the anticipated effect of impact with the Earth at terminal velocity on human tissue, we projected that 99% of the control arm would experience the primary outcome at ground impact with a relative risk reduction of 95% in the intervention arm. A sample size of 14 (7 in each arm) would yield 99% power to detect this difference at a two sided α of 0.05
No deaths were recorded from people jumping from planes with backpacks.
Conclusions Parachute use did not reduce death or major traumatic injury when jumping from aircraft in the first randomized evaluation of this intervention. However, the trial was only able to enroll participants on small stationary aircraft on the ground, suggesting cautious extrapolation to high altitude jumps. When beliefs regarding the effectiveness of an intervention exist in the community, randomized trials might selectively enroll individuals with a lower perceived likelihood of benefit, thus diminishing the applicability of the results to clinical practice.
The study also has several limitations. First and most importantly, our findings might not be generalizable to the use of parachutes in aircraft traveling at a higher altitude or velocity.
Should our results be reproduced in future studies, the end of routine parachute use during jumps from aircraft could save the global economy billions of dollars spent annually to prevent injuries related to gravitational challenge.
Confirmation bias (and ethics) means both patients and doctors are likely to opt not to take part in an RCT when they believe the intervention is particularly useful. This means many RCT trials are filled with people at low risk who may not benefit or miss out much either way. True RCT’s may be very useful but they are inherently cruel.