Use of Prediction Markets to Forecast Infectious Disease Activity

Abstract

Prediction markets have accurately forecasted the outcomes of a wide range of future events, including sales of computer printers, elections, and the Federal Reserve's decisions about interest rates. We propose that prediction markets may be useful for tracking and forecasting emerging infectious diseases, such as severe acute respiratory syndrome and avian influenza, by aggregating expert opinion quickly, accurately, and inexpensively. Data from a pilot study in the state of Iowa suggest that these markets can accurately predict statewide seasonal influenza activity 2–4 weeks in advance by using clinical data volunteered from participating health care workers. Information revealed by prediction markets may help to inform treatment, prevention, and policy decisions. Also, these markets could help to refine existing surveillance systems.

Physicians, nurses, pharmacists, and microbiologists have access to unique information. In some instances, this information provides insight into future infectious diseases activity. But because this information is subjective and asymmetrically distributed across multiple professions, the collection and analysis of the information is not amenable to standard statistical methods.

Focus groups and surveys are used by businesses to capture subjective information from both customers and employees to improve business decisions. Often, these decisions relate to future expectations (i.e., sales forecasts and industry trends). Recently, both academic researchers and businesses have used prediction markets to guide decisions regarding future events. Prediction markets provide real-time information about the probability of forthcoming events by aggregating expert opinions. We propose that these markets can provide useful information about future activity for infectious diseases. Such information would allow time for planning and allocation of resources to help with treatment and preventative interventions. Prediction markets might also be used to estimate the success of interventions or to predict the likelihood of new drug or vaccine developments.

Background Information About Prediction Markets

In recent years, specialized markets have been developed solely for their utility in forecasting. These “prediction markets” allow trade in artificial financial instruments that are defined to have a value to be determined by the outcome of the event of interest. Participants in the markets are experts with information regarding that event (e.g., an election). They buy contracts that are, according to their private information, undervalued by the market and sell contracts that are overvalued. The prices at which these instruments trade reflect a consensus belief about the future value of the instruments and thus can be used as a prediction of the future event.

Research using markets for the sole purpose of aggregating beliefs regarding a future event originated at the University of Iowa in 1988. This idea evolved into the Iowa Electronic Markets (IEM), and in 1993, the Commodity Futures Trading Commission granted the IEM an exemption from federal regulatory oversight over its operation of a futures exchange to run political markets for educational and research purposes. The IEM has conducted >100 cash markets to predict the results of elections around the world. Also, it has run other markets for a variety of events, including interest rate decisions of the Federal Reserve, foreign exchange rate changes, movie box office receipts, and the capitalized value of initial public offerings. More than 15,000 students in >100 colleges and universities have participated in IEM markets. The prediction record achieved by the election markets of the IEM is substantially superior to alternative mechanisms, such as opinion polls [1–3], and the IEM is cited as one of the best demonstrations of how efficiently markets aggregate information about uncertain future events [4]. For presidential elections, the average prediction error across 6 elections has been <1.5%, whereas opinion polls for those same elections had an average error of ∼2.5%.

Examples of Different Prediction Markets

Although the IEM was the first prediction market, several other successful examples exist. Eli Lilly and Company has used internal markets to help predict which developmental drugs might have the best chance of advancing though clinical trials [5]. Hewlett-Packard has experimented with markets to forecast sales of its printers, and these markets have outperformed statistical sales forecasts [6]. Similar private markets have been conducted by Google, Microsoft, and other firms. Three publicly available prediction market sites, which do not use real money, are the Hollywood Stock Exchange, the Foresight Exchange, and NewsFutures. The Hollywood Stock Exchange has accurately predicted Oscar nominees and repeatedly forecasts opening-weekend box office returns more accurately than does the movie industry. The Foresight Exchange operates markets to predict a variety of events, such as whether specific scientific conjectures will be proven [7].

How Prediction Markets Work

Many situations involving uncertain future events offer the prospect of successful application of prediction markets. Such markets can aggregate the disparate and diffuse information held by participants to predict the event outcome. A simple example, patterned after one provided by Eisenberg and Gale [8], illustrates how markets can aggregate information. Suppose there are 2 investors and 3 competing technologies that could be used to produce a new product. Only 1 technology will ultimately succeed. With no other information, each investor might regard each technology's likelihood of success as one-third and thus make offers to buy the rights to all 3. Suppose instead that different information is given to each investor. Investor 1 is told that technology A has an inherent flaw, and investor 2 is told that technology B cannot succeed. Each investor still is uncertain about the outcome. However, investor 1 believes that technologies B and C each have a 50% chance of success, and investor 2 believes that technologies A and C each have a 50% chance of success. On the basis of their privileged knowledge, investor 1 pursues technologies B and C, and investor 2 pursues technologies A and C. Competition drives the price of technology C higher, whereas the lack of competition drives the price of technologies A and B to 0. Thus, the prices investors are willing to pay for the rights to the technologies will reveal the successful outcome.

Prediction Market Requirements and Limitations

Successful application of a prediction market requires both uncertainty about an outcome and differing opinions about the outcome probabilities. If everyone participating has the same information and the same opinions, then no one will trade, no prices will be generated, and no information will be aggregated [9]. Also required is an outcome that can be verified after the fact. For example, a market for the recurrence of severe acute respiratory syndrome (SARS) might base contracts on whether human-to-human transmission of SARS occurs in Hong Kong by January 2008, as documented by the World Health Organization. However, as with all forecasting methods, the accuracy of prediction markets is greater for near-term events than for those far in the future, and that accuracy will depend critically on the information available to participating traders.

There are 3 important considerations for all prediction markets [3].

1. A diverse pool of traders. Successful markets require data from diverse sources. Specifically, traders must have different information and opinions. Thus, for an influenza market, the goal would be to identify a diverse group of traders from a variety of health care professions and geographical locations.

2. A sufficient number of traders. We know from both theory and experience that increasing the number of traders increases the accuracy of the predictions. Although the minimum number of traders needed for an accurate prediction is still an open research question, the experience of the IEM suggests that the number of participants need not be as large as the number required to obtain comparable accuracy with surveys. For example, election prediction markets with 200 active traders routinely yield predictions with smaller errors than those from opinion polls involving 10 times as many respondents (e.g., Gallup). In the Hewlett-Packard example described above, the markets seemed to work well with a dozen traders [6].

3. Incentives to trade. Transactions made for the purpose of maximizing profits are the means by which traders reveal their private information to the marketplace. The importance of using real money in prediction markets, however, has yet to be determined; some experimental evidence shows that using play money can yield similar results [10]. Moreover, the opportunity to participate in some markets may be enough of an incentive, and the use of real money in some health care settings may not be appropriate. However, the markets operated by the IEM, which serve as the research basis for the markets proposed here, involve real money. One alternative would be for market managers to provide small educational grants to encourage participation.

Advantages of Prediction Markets Over Surveys

Prediction markets offer a number of advantages over surveys. They are continuous and ongoing, allowing immediate revelation of new information. Although some surveys offer a small incentive in return for participation, the incentives earned by traders in a prediction market increase in proportion to the quality of the information provided. Unlike surveys, a market provides immediate feedback to participants, allowing them opportunities to reassess their own information and to respond. The market interface is interactive, in marked contrast to most surveys, providing further incentives for participation.

Most surveys rely on random samples for validity and accuracy. In prediction markets, on the other hand, those with the best information are the best participants—the very individuals who are most likely to self-select into the market. With surveys, this process would introduce a sampling bias, but with markets, the incentive structure tends to make the forecasts more accurate.

An Example of A Prediction Market for Infectious Diseases: The FLU Market

Although the influenza season occurs annually, unique characteristics particular to each influenza season make forecasting difficult. Each year, the geographical locations, rates of increase and decrease, duration, and size of each outbreak vary considerably [11–13]. Statistical models using historical data may accurately describe the typical pattern for a particular year, but they do not predict departures from the norm [14]. However, many health care workers have access to information regarding future influenza activity. For example, pediatricians and microbiologists are the first to know when the season is starting. Also, information about influenza activity in some regions may help predict what will happen in others. To use this information to predict future influenza activity, we conducted a pilot study for the 2004–2005 influenza season in Iowa to determine whether health care workers with no experience using futures markets could trade in an influenza prediction market [15]. The market opened on 20 September 2004 and closed on 23 April 2005. Sixty-two health care workers from a variety of backgrounds participated. Traders bought and sold contracts over the Internet at prices that were in accordance with their individual expectations regarding future influenza activity. The interaction of all these traders created a set of market prices that reflected the traders' consensus of the probability that future influenza activity would be at a particular level. The market was open 24 h each day; prices were updated with each new trade on our Web site and could be viewed by all traders.

Contracts in this market were based on the level of influenza activity in Iowa, as reported by the Centers for Disease Control and Prevention's (CDC's) color-coded system (red, widespread; blue, regional; purple, local; green, sporadic; yellow, no activity) [16]. A new set of contracts was introduced for each even-numbered week throughout the influenza season. Trading in contracts for a particular target week opened 8 weeks in advance and ended at noon on the last day of the target week. Contracts were liquidated after the CDC released its activity report the following week. The contract (color) that denoted the actual outcome for a particular week had a liquidation value of FLU$1.00; all other contracts for that week had a value of FLU$0.00. Each trader started with FLU$100 in a trading account. At the end of the influenza season, the remaining balance in each account was converted to US dollars at the rate of 1 : 1, and the trader received an educational grant equal to that amount.

Sixty-two traders were enrolled in the market; 47 of the participants made at least 1 transaction. The maximum number of log-ins to the market was 210, and the mean number was 33.3. When the final market closed, balances ranged from $44.70 to $213.19. If any of the traders ran out of funds, they were no longer able to participate. Seventy contracts were offered for trading over the course of the market (5 color-specific contracts for each of 14 selected target weeks), and the mean number of trades per contract was 110.

In general, as the time before the target week decreased, the prediction accuracy of the market increased. Although the actual color was not revealed until 1 week after the end of the target week, the market correctly predicted that color by the end of the target week 71% of the time, 1 week in advance 50% of the time, and 2 weeks in advance 43% of the time (table 1). The market predicted a color within 1 of the actual color >90% of the time as far out as 2 weeks in advance and >84% of the time as far out as 5 weeks in advance.

Table 1

Accuracy of a prediction market to forecast influenza activity.

No publicly available forecasts for influenza activity are available, but a comparison with predictions implied by historical experience can provide a benchmark for the quality of the predictions from the market. For each week, using the historical average level of activity would have correctly predicted the correct color only 36% of the time, and the prediction would have been within one color only 79% of the time (table 1). According to these raw counts, therefore, the market outperformed the historical prediction through at least 4 weeks in advance, with the single exception of exact prediction 3 weeks in advance. The number of observations generated by this pilot experiment, however, is insufficient to show a statistically significant difference. Assuming the same prediction advantage over history is constant, twice as many observations would be needed to approach statistical significance.

Market prices at any point in time can be taken as a prediction, as of that moment in time, of future influenza activity. Figure 1 is a graphical summary of predictions from the market at 3 such moments: the end of each target week and the end of the periods 2 and 4 weeks before each target week. For comparison, the figure includes the actual activity levels for 2004–2005 as reported by the CDC and the reported activity levels for the corresponding calendar weeks over the previous 5 years. This figure demonstrates the success of the market in predicting the epidemic curve at least 2 weeks in advance.

Figure 1

Market predictions and actual and historical levels of 2004–2005 influenza activity in Iowa. The top 3 panels show the market's predictions 4 weeks in advance, 2 weeks in advance, and at the end of the target week. The bar for each week is divided into ⩾2 colored sections; the length of each section reveals the probability associated with that activity level. For example, the “4 weeks in advance” prediction for week 42 comes from the week 38 market prices for the week 42 contracts. The interpretation of the colored bars for that prediction is an ∼90% chance of yellow, ∼10% chance of green, and a 0% chance of colors purple, blue, or red. Graphs with predictions from 0–8 weeks in advance are available from the authors. Of note, the figure does not reveal information about trading volume, just closing prices. The fourth panel shows the actual color-coded activity level in Iowa for each week of the 2004–2005 season, as reported by the Centers for Disease Control and Prevention (CDC). The fifth panel shows, in a comparable format, the breakdown of activity reports over the previous 5 years. The week-48 activity level, for examples, was yellow in one year, blue in another, and green in the other 3 years.

The accuracy of market predictions is particularly impressive given the variability of the onset, duration, and intensity of influenza activity in Iowa from year to year (figure 2). During the past 6 years, influenza appeared as early as the second week of October and as late as the second week of January [16]. In 2 of the 6 years, the season ended by the first week in March, whereas in 2 others, it extended into May. Influenza was widespread in the state for 6 consecutive weeks during 2 of the 6 years, whereas in 1 year, it never reached the widespread level [16]. Figure 3 illustrates information in the public domain regarding influenza activity in Iowa during the 2004–2005 season that traders could have used.

Figure 2

Six-year history of influenza activity in Iowa. Influenza occurs on a predictable, seasonal basis during the winter months. Despite the seasonal predictability of influenza, considerable variability exists regarding the extent of disease activity and the timing and geographical location of influenza outbreaks. The typical course of an influenza outbreak runs ∼10 weeks. The columns in the table represent the 6 most recent influenza seasons; the 30 rows represent weeks of the year during the season. Each cell in the table is coded with the Centers for Disease Control and Prevention's (CDC's) color representation of influenza activity in Iowa during that week of the respective year. Note that 4 entries were not available; either no map or the wrong map appeared for that week on the CDC influenza activity Web site. NA, not available.

Figure 3

Publicly available information regarding influenza activity. The top panel illustrates the percentage of influenza cultures that yielded positive results, school absenteeism rates, and the weekly percentage of influenza like illnesses (ILIs) reported during the 2004–2005 influenza season for Iowa. For comparison, the Centers for Disease Control and Prevention's (CDC's) activity levels appear in panel 2, and the forecasts from 2 weeks in advance from the market appear in panel 3.

There were several limitations to this market. First, the CDC's weekly color-coded map may not be the most accurate measure of influenza activity and, consequently, is not an ideal basis for contracts in our market. However, the color-coded activity levels are the only state-level measure of influenza activity available, and they have the advantages of accessibility and ease of interpretation. The market predictions seemed to track influenza-like illness and culture data more closely than the geographic distribution (color). Second, the trading level throughout the season was low; 20% of the traders made 80% of the trades, for example. Thirdly, our pool of traders was not widely disbursed geographically; most were located in the eastern part of the state, and 70% were associated with the University of Iowa. Participants included infection control practitioners, nurses, physicians, pharmacists, and microbiologists, all of whom were recruited by word of mouth and e-mail. Because only a few traders were on the “front line”—several were academic physicians who did not see patients on a regular basis—our traders may not have always had up-to-date information about influenza. However, those with inferior information may have been less likely to participate. In general, the more active traders outperformed those who were less active. Finally, more data from future influenza markets beyond these pilot data will be needed to demonstrate that infectious diseases markets can outperform statistical forecasts, as prediction markets have done in previous applications.

Applications Beyond Predicting Influenza Activity

Prediction markets for infectious diseases have applications beyond predicting influenza activity; for example, markets could be used to predict the future dominant strain of influenza by aggregating information from experts in the fields of virology, epidemiology, and clinical microbiology. The most effective way to initiate infectious diseases prediction markets might be to work with existing sentinel networks, such as ProMED-mail or the Emerging Infections Network. With a pool of traders in place, new prediction markets could be created almost overnight to address emerging infectious diseases, even before a specific etiological agent is identified. For example, information about SARS existed in the early spring of 2003 [17], and a prediction market could have been started to forecast human-to-human transmission in the United States (or any other particular country) by a certain date. This market could have quickly, accurately, and inexpensively aggregated expert opinion about this new infection. Such a prediction market would have been especially useful, given the absence of preexisting SARS surveillance systems. Similarly, a prediction market for avian influenza could help public health officials plan ahead by reducing uncertainty about the timing of a future pandemic. Markets to forecast next season's supply and demand for influenza vaccine could help to reduce uncertainty regarding the future vaccine supply, helping patients, health care workers, vaccine manufacturers, and public health policy makers (table 2).

Table 2

Examples of potential applications of prediction markets to the field of infectious diseases.

Prediction markets will never replace traditional infectious diseases surveillance; we view them instead as a potential supplement. If prediction market information is used in tandem with existing surveillance systems, it can guide the deployment of resources and increase the efficiency of professionals who are monitoring emerging infectious diseases. Prediction markets can also be used to refine existing surveillance systems. For example, prices can reveal the timing and geographic spread of information. An inspection of trading behavior by groups might reveal which types of participants are the best informed. Such information could then be used to design more sensitive and efficient traditional surveillance systems.

Acknowledgments

We thank Loreen Herwaldt and Daniel Diekema for their help and support, as well as all of the Influenza Market participants for trading in this market.

Financial support. These markets were funded in part by an unrestricted educational award from Sanofi Pasteur, and our current markets are funded by a grant from the Robert Wood Johnson Foundation (An Influenza Prediction Market; 053352).

Potential conflicts of interest. P.M.P., F.D.N., and G.R.N. received recent research funding from Sanofi Pasteur.