Jul 7, 2018
The Randomistas- from James Lind to Esther Duflo
Book of the day — Randomistas- How Radical Researchers Are Changing Our World by Aussie MP economist Andrew Leigh.
In the book Andrew Leigh narrates the story of Scottish physician James Lind and his remarkable experiment to find the cause of the dangerous disease scurvy.
Then, in 1747, a 31-year-old ship’s surgeon by the name of James Lind ran a remarkable experiment. Several months into the voyage of the HMS Salisbury, most of the crew were affected by scurvy. Lind decided to try different treatments on twelve sailors with advanced scurvy, ‘as similar as I could have them’. He took other people’s theories and found a way to test them. Lind’s randomised trial tested six treatments, with each being given to a pair of sailors. The first group got a litre of cider, the second 4 millilitres of sulphuric acid (‘elixir of vitriol’) and the third 80 millilitres of vinegar. If those last two treatments weren’t bad enough, the fourth group were made to drink 250 millilitres of seawater, while the fifth group got a mixture of nutmeg, garlic, mustard-seed, horseradish, balsam of Peru and gum myrrh. The sixth group received two oranges and one lemon. Apart from the treatments, all the patients were fed the same diet and kept in the same part of the ship.
It didn’t take long for the experiment to show results. Lind reported that ‘the most sudden and visible good effects were perceived from the use of oranges and lemons’, to the extent that one of the patients who had been given citrus fruit was ready to return to duty in less than a week. By contrast, the patients given acid, vinegar and seawater did not improve. Given that sulphuric acid was the British Navy’s main treatment for scurvy, this was an important finding.
….
The British may have been slow to adopt Lind’s findings, but they were faster at curing scurvy than their main naval opponents. An end to scurvy was a key reason why the British, under the command of Admiral Lord Nelson, were able to maintain a sea blockade of France and ultimately win the 1805 Battle of Trafalgar against a larger force of scurvy-ridden French and Spanish ships. Unlike Commodore Anson six decades earlier, Lord Nelson didn’t have to fight while scurvy ravaged his crew.
In the age of sail, over 160,000 convicts and many more free settlers made their way to Australia. Had James Lind not found a way of preventing scurvy, many of those settlers — ancestors of present-day Australians — would have died at sea. In fact, it is possible that if the scurvy randomised trial had been conducted by another colonial superpower, Australia’s national language might be French, Dutch or Portuguese.
For Discussion: Given the many public policy implications of RCTs does the public sector need an Evaluator-General ? Here is a proposal from Australia;
Rather than simply being talked about by prime ministers as if it were all commonsense, evidence-based policy and its organisational underpinning — monitoring and evaluation — becomes a key function of any concerted endeavour in the public sector as accounting is. (I’ll make some limiting comments on this later, but for the purposes of exposition assume a concerted endeavour is a program — like chaplains in schools, the R&D tax credit or a police program to reduce domestic violence.)
No new program could be introduced without a properly worked up monitoring and evaluation plan. Existing programs would be systematically exposed to this regime over time. Evaluation should be done at the level of delivery. As this occurs, further synthesis and analysis at various levels will typically occur.
Evaluation would be done by people with domain skills in both evaluation and in the service delivery area who were formally officers of the Office of the Evaluator-General. They would report to both the portfolio agency delivering the program and to the evaluator-general with the EG being the senior partner in the event of irreconcilable disagreement. All data and information gathered would travel to the centre of both the EG’s and the departmental systems. Meanwhile, the portfolio agency would report to their minister but the EG would report to Parliament — as the auditor-general does.
The monitoring and evaluation system would be built from the start to maximise the extent to which its outputs can be made public and the public could be given access to query the system, though the system itself would only provide public information outputs that met strict privacy safeguards.
Related:
Nicholas Gruen: evaluation knowledge comes not from numbers, but questions
Review of Randomistas: How Radical Researchers Changed Our World
Several books over the last years have been reviewed here that have highlighted the leading role of RCTs in development economics including Karlan and Appel’s More Than Good Intentions and Tim Ogden’s excellent book Experimental Conversations. So what does Randomistas offer a development economist with plenty of exposure to the RCT world?
What grabbed my attention the most were the chapters that included engaging reviews of RCTs outside of development economics. In the chapter on RCTs in education, Leigh reviews a series of randomized trials that produce a sequence of results so counter-intuitive to prevailing wisdom that it might deserve its own TV episode on MythBusters. One example: After-school programs not only don’t seem to improve academic outcomes; they appear to lead to significantly worse school behavior and a higher probability of suspension. In short, after-school programs introduce marginal kids to the “wrong crowd,” probably neglected kids whose parents don’t pay enough attention to them.
Leigh’s chapter on controlling crime reviews the randomized trial on the city of Chicago’s creative Becoming a Man program. Guided by a philosophy that “A boy has problems. A man finds solutions to his problems,” the RCT showed BAM reduced arrests by one-third to one-half. I also liked the review of results from the British “Nudge Unit,” which carries out a myriad of low-budget, behavioral economics experiments in the U.K. These RCTs carried out on nudge treatments induce people to pay their taxes, attend job-training seminars, and donate their organs. They are fun to read about, and for development researchers they may stir up the mental broth for running similar experiments overseas.
Randomistas through the ages
The final chapter, ‘Building A Better Feedback Loop’, indeed makes a strong case that it would make for better outcomes if policymakers and politicians were able to change course on the basis of evidence. One of the practical advantages of RCTs is perhaps that they leave open the decision — they can be described as pilot schemes in the policy context. There is far too little evaluation in policymaking — it can be too embarrassing, the decisions are water under the bridge — so setting up an evaluation in advance by design, as it were, is attractive.
Experimental approaches are surely welcome as one more addition to the toolbox of policy evaluation, more useful in some contexts than others, vulnerable (like all empirical methods) to not being well carried out or interpreted. They can easily become a means of manipulating people — as in the behavioural testing done by online marketers — so should always be deployed with caution in the world of policy and politics. After all, people are wising up to the testing methods used by Facebook, Amazon, etc and not necessarily liking them. So I would be less of a randomista enthusiast than Andrew Leigh; other methods of evaluation are available.
