to uncover causal relationships, stop trying to invent a time machine and run an experiment instead! Understanding causal relationships offers the knowledge needed to produce desired results through action. In this article, I’m going to illustrate the power of experimental design by using a time-machine-based conceptual exercise. My goal is to convince you that more can be learned about causality through experimentation than using a time machine.
Why are time machines useful in a causal thought experiment?
Using a time machine for a thought experiment feels ridiculous, and in many ways it is. But it also has a characteristic that makes it valuable for exploring hypothetical outcomes. Time machines could give us something that we, in our time-bound state, cannot see – counterfactuals. As the name implies, a counterfactual is something that didn’t happen. They are not observable by definition because they never occurred. Counterfactuals what would’ve happened under different circumstances. They give answers to questions like – “Would I have gotten sick if I didn’t eat that gas station sushi?” If we had a time machine however, we could reverse the clock, do something different and see what happens. In the case of the sushi, I could restart the day, not eat the sushi and see if I still get sick. In other words, we could observe the otherwise unobservable counterfactuals.
The counterfactuals learned by the time machine could then be compared to what actually happened (we could call it a ‘factual’ I suppose…) to understand the impact of an intervention. For our unfortunate sushi example, me getting sick is the ‘factual’ – it actually happened. If I had a time machine, I could rewind time, not eat the sushi and observe what would’ve happened, this is the counterfactual. I could then compare the factual with the counter factual to establish causality. Let’s say that I went back in time, kept everything in my day the same except eating the sushi. If I still got sick (factual = counterfactual), I know that the sushi didn’t cause the illness because I would’ve been sick either way. If I didn’t get sick however (factual ≠ counterfactual), then I can conclude that the sushi caused my illness. With a time machine, establishing causality for individual events would be that easy!
At first glance, it seems like our time machine is also an awesome causality deducing machine! Being able to observe counterfactuals would be very powerful, but we can actually make more useful causal deductions using well-designed experiments. Which is great because, time machines don’t exist, but well-designed experiments do! Let’s get into how designed experiments can be better than using a time machine.
The causality of individual events is not generalizable
While a time machine would answer a lot of curiosity-driven ‘what if’ causal questions, the learnings we would gain from observing counterfactuals would not be generalizable to other, similar (but not the same) situations. In my sushi example, I would satisfy my curiosity by understanding if the sushi made me sick – but the knowledge I gained wouldn’t serve any pragmatic purpose for future decisions. All I know is that on that specific day, at that specific gas station, at that specific time, that specific serving of sushi made me sick. I don’t know what would happen if I changed any of the bolded circumstances.
We can gain generalizable knowledge, which we wouldn’t get from the time machine, by designing an experiment. Generalizable knowledge is very useful because it can help us make good decisions in the future!
Imagine that I ran an experiment that randomly assigned multiple brave souls to eat gas station sushi or restaurant sushi. This experiment would tell me if on average, gas station sushi makes people sicker than restaurant sushi. This is already an improvement from the ‘time machine’ approach because the results apply to the population of people that I sampled instead applying to me only.
But, I could be smarter about the design of the experiment to get even more knowledge! Instead of simply assigning people to gas station or restuarant sushi, I could assign people specifc gas stations at specific times or the restuarant at specific times. By adding these two new variables (time and gas station location) I can not only learn if gas station sushi makes people sick more often, I can also learn if there are differences between the three gas stations that serve sushi in my town and if time of day also has an impact.
In this experiment, I don’t directly observe counterfactuals, but the randomized assignment helps confounders average out so I can estimate the average treatment effect (ATE) almost as if I could observe counterfactuals.
How do the experiment learnings differ from my time machine learnings? The experiment is (1) using multiple people, (2) multiple sushi servings, (3) multiple gas stations and (4) multiple times of day. As a result, I can take away a lot of causal insights that I and other people can use. For example, I would understand if generally, gas station sushi makes people sicker than restaurant sushi in my town. I would also learn if some gas stations make people more sick than others and if buying sushi at some times is worse than others. This knowledge can help me, and other people make future decisions. It is much more useful than knowing that the sushi from one gas station and one time made me sick!
In addition to all of the variables that we can control, we can include covariates in our analysis. Covariates are factors that we cannot control but are important. In this example, covariates could be things like previous medical conditions or age. By including covariates in the analysis, we can also learn if there are any interaction effects between the covariates and the treatments.
Below is a summary that compares what we could learn with a time machine to what we can learn with experiments.
Now that we understand the rich depth of causal relationships that we can understand using experimentation, let’s transition to discussing how the variety of outcomes under an experiment is more powerful than a single outcome (the one counterfactual) that we would observe with a time-machine run.
Designed experiments quantify the causal relationships; single counterfactuals do not
Direct observation of a single counterfactual doesn’t give any idea of the strength of the general causal relationship. If I go back in time after I got sick once to test if the sushi made me sick, I would learn that it did, or it did not cause my illness. I still wouldn’t have any idea of the probability that I’ll get sick if I satisfy my sushi craving at a gas station again in the future! Is it deterministic, i.e., will get sick every single time I eat gas station sushi? Is it probabilistic, will I get sick fifty percent of the time? I just don’t have enough information to know.
The experiment we designed in the previous section would not only help us understand if gas station sushi makes people sick, it would also help quantify the relationship. For example, the experiment might find that on average, eating gas station sushi makes you five times more likely to get sick than restaurant sushi.
Experimental design generalizes better, and it also quantifies the causal relationship better! If we go back in time and look at one counterfactual, we can’t know the probability of observing the same result under similar conditions, with experimentation we can!
Wrapping it up
My goal in writing this article was to discuss why I would still use experimental design to learn about causal relationships even if I had a time machine that allowed me to observe counterfactuals.
The main reasons experimental design is better is because:
- It generates generalizable causal learnings (as opposed to one specific case)
- It provides the strength of relationships to inform future decisions
I hope this thought experiment deepened your understanding of the strengths of experimental design!
