Decision-making in the multiphase optimization strategy: Applying decision analysis for intervention value efficiency to optimize an information leaflet to promote key antecedents of medication adherence

Abstract Advances in the multiphase optimization strategy (MOST) have suggested a new approach, decision analysis for intervention value efficiency (DAIVE), for selecting an optimized intervention based on the results of a factorial optimization trial. The new approach opens possibilities to select optimized interventions based on multiple valued outcomes. We applied DAIVE to identify an optimized information leaflet intended to support eventual adherence to adjuvant endocrine therapy for women with breast cancer. We used empirical performance data for five candidate leaflet components on three hypothesized antecedents of adherence: beliefs about the medication, objective knowledge about AET, and satisfaction with medication information. Using data from a 25 factorial trial (n = 1603), we applied the following steps: (i) We used Bayesian factorial analysis of variance to estimate main and interaction effects for the five factors on the three outcomes. (ii) We used posterior distributions for main and interaction effects to estimate expected outcomes for each leaflet version (32 total). (iii) We scaled and combined outcomes using a linear value function with predetermined weights indicating the relative importance of outcomes. (iv) We identified the leaflet that maximized the value function as the optimized leaflet, and we systematically varied outcome weights to explore robustness. The optimized leaflet included two candidate components, side-effects, and patient input, set to their higher levels. Selection was generally robust to weight variations consistent with the initial preferences for three outcomes. DAIVE enables selection of optimized interventions with the best-expected performance on multiple outcomes.


Introduction
Multicomponent interventions are used to promote a wide range of health behaviors [1][2][3][4].For example, in secondary cancer prevention, complex interventions may aim to support medication adherence, with individual intervention components targeting different barriers to adherence [5].The intervention components that make up a complex intervention incur costs, such as money, time, participant burden, or cognitive load.Removing ineffective or underperforming components improves intervention efficiency, saving resources that could be allocated elsewhere [6].
The multiphase optimization strategy (MOST) is an engineering-inspired framework for optimizing complex interventions such that they are not only effective but also readily implementable, e.g. because they are efficient [6,7].In MOST, intervention components are considered candidates for inclusion in an eventual intervention package.The individual and combined effects of the candidate components are estimated in an optimization randomized control trial (ORCT), often using a design from the factorial family of experiments [8].Based on the empirical results of the ORCT, an optimized intervention-i.e. a combination of the candidate components that is, ideally, effective, and efficient, or that otherwise accomplishes the objective for optimizationis selected [7,9].A key purpose of the ORCT is to provide information useful in identifying and removing ineffective or underperforming candidate components.
A component screening approach (CSA) was initially recommended as a way to select an optimized intervention based on empirical ORCT data [9,10].However, CSA has limitations [11].First, CSA relies on systematic interpretation of certain important main and interaction effects estimated in the ORCT using a process that is difficult to apply when there is more than one outcome variable of interest.This is an important limitation, as multicomponent interventions frequently have objectives that involve multiple valued outcomes [11].For example, in the context of medication adherence, outcomes such as quality of life and physical functioning may also be important, in addition to adherence [12].Second, the systematic screening process used in CSA, detailed and demonstrated in Green et al. [13], relies on arbitrary thresholds (usually, but not always, statistical significance thresholds) to determine which estimated main and interaction effects are "important" enough to contribute to decision-making.This can make the approach prone to missing relevant information from near-threshold effects.Finally, the systematic interpretation of main and interaction effects in CSA frequently requires difficult judgment calls.This can occur when two different important effects indicate contradictory decisions about components (e.g. one indicating a component should be included and another indicating a component should not be included).CSA does not dictate how contradictory screening decisions should be dealt with.
Recent methodological advances in MOST have suggested an alternative approach for selecting optimized interventions [11,14].Using the concept of posterior expected value from the Bayesian decision sciences [15], this alternative approach, decision analysis for intervention value efficiency (DAIVE), overcomes limitations of CSA.DAIVE can readily incorporate multiple outcome variables; outcomes are combined using a value function that reflects the decision-maker's preferences (e.g.giving some outcomes more importance than others, as appropriate) [11].As explained more in the Methods section, DAIVE does not rely on arbitrary significance cutoffs, and therefore makes use of all available empirical information.Moreover, DAIVE does not rely so heavily on interpretation of individual effects, as expected outcomes can be estimated without preliminary screening of components.An optimized intervention can be chosen that maximizes expected value, or that strategically balances expected values with other relevant criteria (e.g.resource constraints).
Green et al. [13] previously applied CSA to optimize an information leaflet intended to support women with earlystage breast cancer who are prescribed medication called adjuvant endocrine therapy (AET).AET can reduce breast cancer recurrence by up to 30%, and mortality by up to 40% compared with no endocrine therapy [13,16,17].However, maintaining adherence to AET can be challenging, e.g.due to side-effects such as hot flushes and joint pains, which are commonly reported in women taking AET [18].The information leaflet components were designed to target key barriers to AET adherence [19][20][21][22][23][24], e.g. the presence of concerns about AET or low perceived necessity AET.A 2 5 factorial trial estimated the individual and combined effects of factors representing five candidate leaflet components: (i) diagrams about how AET works; (ii) visual representations of the benefits of AET; (iii) detailed information about the prevalence of AET side-effects; (iv) answers to common concerns about AET; and (v) quotes and pictures from breast cancer survivors [13].Effects were estimated on three outcomes: (i) women's medication beliefs about AET, reflecting the balance between concerns and necessity beliefs [25,26]; (ii) their objective knowledge about AET; and (iii) their satisfaction with information received about AET.Of these three outcomes, medication beliefs was considered primary, given well-established relationships between beliefs and adherence [21,25,27].
Objective knowledge and satisfaction with information are also hypothesized antecedents of eventual adherence to AET, but these relationships are less consistently reported.Lower knowledge about AET has been associated with lower adherence [28], and women prescribed AET frequently report receiving insufficient information about the medication and its side-effects [22,23,29,30].
When Green et al. [11] initially identified an optimized leaflet (i.e.just before the latest advances in optimization decision-making were published), CSA required the use of a single outcome variable for selection of an optimized intervention.As a result, Green et al. [11] identified a leaflet that was optimized for effects on only the primary outcome, medication beliefs.With this approach, components with strong effects on, say, knowledge but not medication beliefs would not be identified.To base decisions about component performance on only one of three-valued outcomes (i.e.medication beliefs only) could mean overlooking potentially useful candidate components.
The purpose of the present study was to apply DAIVE to optimize an information leaflet for effectiveness on multiple valued hypothesized antecedents of adherence to AET.In a secondary analysis of the 2 5 factorial trial [11], we incorporated empirical information about component performance on all three outcome variables in the 2 5 factorial trial [13]: medication beliefs, objective knowledge, and satisfaction with information.As described in more detail below, we gave the primary outcome, medication beliefs, greater importance Decision-making in the multiphase optimization strategy relative to the other two outcomes, and we further determined that objective knowledge was relatively more important than satisfaction with information.To examine the robustness of the decision made, we systematically varied the outcome weights used to assign differential importance to the three outcomes, and observed whether this changed the decision about identification of the optimized leaflet.

Experimental design
The trial used a full 2 5 factorial design (2 × 2 × 2 × 2 × 2) to estimate the individual and combined effects of five candidate intervention components [13].Each candidate component was operationalized as a two-level factor: (i) diagrams detailing the mechanisms of AET (factor levels: on/off); (ii) benefits, with visual icon arrays detailing the benefits of AET (factor levels: enhanced/basic); (iii) side-effects, with detailed information about the prevalence of AET side-effects (factor levels: enhanced/basic); (iv) concerns, with answers to common concerns about AET (factor levels: on/off); and (v) patient, with quotes and pictures from breast cancer survivors ( factor levels: on/off) (full descriptions available in Supplementary Appendix 1).Factor levels for benefits and side-effects were enhanced versus basic, as we felt any optimized version of the leaflet should contain at least a basic level of information about the benefits and side-effects of AET.A sample of n = 1603 healthy women were randomized to 32 experimental conditions, each comprising a unique combination of factor levels (Table 1).Participants were required to be over 18 and able to read English, and did not require a diagnosis of breast cancer to participate.Full details about the candidate components, factors, sample recruitment, and demographics are available elsewhere [13].Study materials are available online (DOI 10.17605/OSF.IO/AG7YK).

Medication beliefs
At two time points, before and after being shown a version of the information leaflet (containing a unique combination of candidate intervention components), participants responded to the Beliefs about Medicine Questionnaire-AET [26], with 10 items assessing specific medication beliefs on a 5-point scale ranging from "strongly disagree" to 'strongly agree' [26].There were two subscales: necessity beliefs and concerns, with five items each.Cronbach's alpha (α) scores, assessing internal consistency reliability, for the necessity subscale were 0.852 and 0.884 pre-and post-leaflet respectively, and 0.808 and 0.831 for the concerns subscale pre-and post-leaflet, respectively.For the medication beliefs outcomes (baseline and post-leaflet), a beliefs differential score was calculated by subtracting concern from necessity scores (range −20 to +20).The differential follows recommended scoring [31], and has more consistently predicted nonadherence than necessity beliefs or concerns subscales alone across conditions [32].

Objective knowledge
The objective knowledge outcome was assessed after participants viewed a version of the information leaflet using eight "true or false" items about AET.The eight statements, which involved the mechanisms of AET (two statements), the benefits of taking AET (two statements), the prevalence of side-effects (three statements), and the management of side-effects (one statement), were written to reflect key information in the leaflet.Four statements were true, and four statements were false.

Satisfaction with information
After viewing a version of the information leaflet, participants responded to a modified version of the original Satisfaction with Information about Medicines Scale [33].Participants were asked to rate their satisfaction with the information about AET provided across 11 domains, such as how hormone therapy works and what the benefits of the medication are.Participants responded on a 5-point scale: "too much," "about right," "too little," "none received," or "none needed."An overall satisfaction score ranging from 0 to 11 was calculated, whereby responses of "about right" or "none needed" (indicating satisfaction) were scored as 1, while all other responses (indicating dissatisfaction) were scored as 0 (Cronbach's α = 0.875).

Overview
In contrast to CSA, DAIVE makes use of a Bayesian paradigm [11,14].First, factorial trial data on one or more outcome variables are analyzed using Bayesian factorial analysis of variance, which summarizes evidence about main and interaction effects using posterior distributions.These posterior distributions for main and interaction effects are then used to obtain posterior distributions for the expected outcomes ( Ŷ s) associated with each version of the intervention under consideration (i.e. each of the 32 versions of the information leaflet comprising unique combinations of the factor levels).
In further contrast to CSA, expected outcomes are estimated with no need for preliminary screening.When multiple outcomes matter, outcomes are combined using a value function, such as a simple linear weighted sum.For example, with J outcomes: where each weight w i indicates the relative importance of that outcome Y i .Depending on the optimization objective [7], the optimized intervention may be the intervention that maximizes expected value.Alternatively, expected values may be strategically balanced with additional criteria (e.g.intervention delivery cost), when this fits the objective of optimization.

Identifying an optimized leaflet
We analyzed factorial data using R version 4.2.2 [34] and the brms package [35].We fit a regression model (see code in Appendix 2) for each of the three outcomes: medication beliefs, objective knowledge, and satisfaction with information.Each model included all 31 main and interaction effects and was equipped with non-informative N(0,5) priors.To estimate main and interaction effects on the primary outcome, medication beliefs after viewing the leaflet, we also controlled for baseline medication beliefs.
Using the steps modeled by Strayhorn et al. [14], we scaled and assigned weights to outcomes using a swing weighting exercise [36].First, we identified the swing in medication beliefs as the most important, as medication beliefs are one of the most consistently reported barriers to AET adherence [21].Following conventions in swing weighting, this meant that we assigned medication beliefs 100 points.We identified the swing in objective knowledge as the next-most important-i.e.half as important as medication beliefs-given some (less consistently reported) evidence suggesting that knowledge of AET is associated with nonadherence to AET [28].We therefore gave objective knowledge 50 points.Finally, we identified the swing in satisfaction with information as half as important as objective knowledge (1/4 as important as medication beliefs), given potential subjectivity in the satisfaction outcome.We therefore gave satisfaction with information 25 points.We calculated outcome weights by dividing the points for a given outcome (100, 50, or 25) by the total number of points for all outcomes (175).This gave us the following value function (rounded to the nearest 0.1): We then put all outcomes on a 0-1 scale, where 0 was worst and 1 was best.
We applied the value function to estimate expected values for each alternative intervention (i.e. each information leaflet, containing a unique combination of factor levels), and we identified the leaflet with the largest expected value as the optimized information leaflet.In other words, we chose to advance intervention efficiency by using the estimated expected values to identify components that were-or were not-worth including in the optimized leaflet, with the goal of removing components that were redundant (or worse, harmful) in contributing to expected value.

Exploring robustness
By systematically varying the points assigned to outcomes in the swing weighting exercise, we repeated the identification of an optimized leaflet using value functions reflecting different preferences about the three outcomes.In general, we wanted to uphold our initial determinations about the relative importance of outcomes (i.e. that all three outcomes are important, and medication beliefs > objective knowledge > satisfaction with information), particularly regarding medication beliefs as the primary outcome.Our main goal in exploring the robustness of the choice of optimized intervention was to examine the extent to which the decision might change as a function of variation in the weights assigned to the additional two outcomes, objective knowledge and satisfaction with information.Therefore, we approached this systematic variation using the following steps: Step 1: Systematically reducing the importance of secondary outcomes Decision-making in the multiphase optimization strategy Beginning with our initial preferences (medication beliefs: 100 points; objective knowledge: 50 points; satisfaction with information: 25 points) as a starting point, we reduced the importance of objective knowledge and satisfaction with information, relative to medication beliefs.We did this by incrementally reducing the points for satisfaction with information (i.e. the less important of the secondary outcomes) first, holding all other points constant, until satisfaction with information had very small importance (one-twentieth the importance of medication beliefs).We then incrementally reduced the points for objective knowledge, again holding all other points constant, until objective knowledge also had very small importance (one-twentieth the importance of medication beliefs).We made the reductions in this order to preserve the ranking of the importance of the three outcomes.Across the incremental changes, we observed whether the selected optimized intervention changed as the importance of the secondary outcomes decreased (eventually, to the point that the secondary outcomes were both minimally important).
Step 2: Systematically increasing the importance of the secondary outcomes Beginning again with our initial preferences (medication beliefs: 100 points; objective knowledge: 50 points; satisfaction with information: 25 points) as the starting point, we increased the importance of objective knowledge and satisfaction with information, relative to medication beliefs.This time we incrementally increased the points for objective knowledge (i.e. the more important of the secondary outcomes) first, holding all other points constant, until objective knowledge had importance equal to the primary outcome, medication beliefs.
We then incrementally increased the points for satisfaction with information, holding the other points constant, until all three outcomes had equal importance.Again, we observed whether the selected optimized intervention changed as the importance of the secondary outcomes increased (eventually, to the point that the three outcomes were equally important).Finally, to contextualize these systematic variations, we also considered select preference scenarios that differed markedly from our initial preferences for the three empirical outcomes, such as one scenario in which the relative importance of the secondary outcomes were reversed.

Results
Credible intervals for the estimated main and interaction effects on each of the three outcomes are provided in Figs.1-3.Of particular note is a main effect for the patient factor on beliefs (which was also reported by Green et al. [13]).There are also multiple interaction effects for which the credible interval does not include zero, suggesting that components are involved in complex synergistic and antagonistic effects on all three outcomes.

Identifying an optimized leaflet
Table 2 provides expected values for all 32 alternative versions of the information leaflet, in order of magnitude.The optimized leaflet-i.e. the one with the largest expected value-is highlighted; in this leaflet, side-effects and patient were set to their higher levels, while all other factors were set to their lower levels.

Exploring robustness
When we upheld the spirit of our initial preferences, in which medication beliefs was the primary outcome, followed in relative importance by objective knowledge and then satisfaction with information, we observed consistency in the choice of optimized intervention (Table 3).In Step 1, when we systematically decreased the importance of satisfaction with informa-tion, first, and then objective knowledge, the choice remained the same (i.e.side-effects and patient set to their higher levels) across all levels of relative importance, even when both satisfaction with information and objective knowledge had

Figure 3
Credible intervals for the estimated main and interaction effects on satisfaction with information.Decision-making in the multiphase optimization strategy minimal importance.Similarly, in Step 2, when we systematically increased the importance of objective knowledge, first, and then satisfaction with information, the choice remained the same (side-effects and patient set to their higher levels) across all levels of relative importance, even when all three outcomes had equal importance (or points = 100 for all 3).This level of consistency was not observed in the selected scenarios that deviated more markedly from our initial preferences.Table 4 reports the decisions under these scenarios.In general, these results suggest a pattern in which the choice of optimized intervention differs when the satisfaction with information importance outcome is relatively more important than the objective knowledge outcome.Furthermore, these results highlight the similarity in performance for the top two contenders for the optimized leaflet, the version with sideeffects and patient set to their higher levels and the version with diagrams, benefits, and patient set to their higher levels; we return to this in the discussion.

Discussion
Using empirical data from a 2 5 factorial trial [13] and the latest recommendations for decision-making in MOST [11], we identified an optimized information leaflet for women with breast cancer that included two of our five candidate intervention components set to their higher levels: information about side-effects of the medication and patient input (quotes and photos) from survivors explaining their motivations for taking the medication.To identify this optimized leaflet, we applied the newly recommended approach for optimization decision-making, DAIVE, which offers advantages relative to the previously recommended approach, CSA-especially in terms of the ready accommodation of multiple outcome variables.The identified optimized leaflet demonstrated the best-expected performance in terms of three-valued outcomes: medication beliefs, objective knowledge, and satisfaction with information.These outcomes were valued as hypothesized antecedents of eventual adherence to AET for women with breast cancer.
When Green et al. [13] previously applied CSA to identify an optimized information leaflet, that leaflet version differed from the optimized leaflet we identified here.The previous selected leaflet included four of the five candidate components set to their higher levels (D, B, C, and P)-and notably, included the lower level of the side-effects component.Since CSA can only accommodate a single outcome, decision-making   in Green et al. [13], was based on medication beliefs only, so the differences between these two identified leaflets are not necessarily surprising.Instead, the differences are likely due in part to varying patterns of component performance on the three outcomes, suggesting a different choice when three outcomes versus one outcome are considered.However, one of the alternative preference scenarios we considered (Scenario 2, in which medication beliefs was the single primary outcome), suggests that these differences may also reflect other points of contrast between CSA and DAIVE.With the same outcome, CSA (as applied in Green et al. [13]) and DAIVE (in Scenario 2) identified different optimized interventions (diagrams, benefits, concerns and patient, and diagrams, benefits and patient, respectively).This is also not surprising; simulated testing of the two approaches across varied trials with one outcome variable, found that CSA and the methodology DAIVE uses (a posterior expected value approach) frequently but not always arrived at the same optimized interventions [11].In this case, differences may reflect the judgment calls that went into interpreting higher-order interaction effects [13], as is common in applications of CSA.
To examine whether our choice of optimized intervention was susceptible to differences in the weights that assigned relative importance to the three outcomes, we systematically varied the points for the three outcomes ("points" referring to those used in a swing weighting exercise [36]).In general, our choice of optimized intervention was robust to systematic variations in weights for the three outcome variables that upheld the spirit of our initial preferences, supporting the idea that an optimized leaflet with side-effects and patient set to their higher levels is appropriate given our initial preferences.In contrast, when we considered scenarios that deviated from those we set initially-e.g. by giving satisfaction with information more importance than objective knowledge-there were differences in the choice of optimized intervention.This emphasizes the importance of making initial determinations about the relative importance of outcomes; for investigators using MOST, it may even be best to outline such determinations during the Preparation phase, e.g. as a conceptual model is being built.
Across the full range of relative importance we considered for the three outcomes, there were two top contenders for the optimized leaflet: the version with side-effects and patient set to their higher levels and the version with diagrams, benefits, and patient set to their higher levels.Even under our precise initial preferences for the three outcomes, the expected values of these two leaflet versions were close in magnitude, especially relative to the other possible combinations of component levels.This finding is not unique to our use of DAIVE for decision-making; in Green et al.'s previous application of CSA, alternative versions of the information leaflets also performed similarly [13].We respond to this finding with the following observations: First, the similarity in performance is unsurprising given the context, as components distinguished elements of an information leaflet.Furthermore, in two cases, components were operationalized as factors with higher versus lower levels (e.g.side-effects), meaning that the main effect estimated was for the higher level of the factor relative to the lower level, not for the presence versus absence of that component, potentially leading to smaller differences in performance across leaflet versions.Still, results suggest particular support for the patient component; the factor for patient had a significant main effect on medication beliefs and is included in both contenders for the optimized leaflet (side-effects, patient and diagrams, benefits, patient) at its higher level.The question appears to be which synergies with patient should be chosen, and there appears to be an either/or, perhaps due to small antagonisms between sideeffects and either diagrams or benefits: the choice is either synergies of patient with side-effects or synergies of patient with diagrams and benefits, but not of patient with sideeffects, diagrams, and benefits.Such antagonisms may have implications for health communication strategies, suggesting that certain types of messaging, effecting change via certain mechanisms, work better together (or worse) than others.In future work, factorial mediation analysis [37] could help to elucidate the mechanisms underlying communication strategies' effects (e.g.reflecting different theoretical constructs, such as gain versus loss framing [38]).Second, our objective in this trial was not to prove that one version of the information leaflet is definitively better than another version.In general, that is not the objective in any prototypical ORCT; "optimized" does not necessarily mean "best."In other ORCTs similar performance for two alternative intervention versions may be further contextualized with cost data; if two interventions that perform similarly in terms of expected value have sufficiently different costs, a choice between them may be clear-and the choice may favor the version with slightly lower expected value, given large cost savings.In such a case, the ORCT provides very useful information that an alternative intervention version produces comparable expected benefits at a lower cost.We refer interested readers to Strayhorn et al. [14] for an example and further discussion of this.In the present trial to optimize the information leaflet, we did not consider monetary costs, primarily given the granularity in components and almost identical costs across leaflet versions.However, there may still be reason to think carefully about the benefits of smaller versus larger intervention packages, given very similar performance.Based on constraints of attention and time, participants could be more likely to engage in all components of a leaflet that contains fewer components set to their higher levels (i.e.side-effects and patient input) than they are to engage in all components of a leaflet that contains more components set to their higher levels (i.e.diagrams, benefits, and patient input set to their higher levels).This could be important, given the extent to which leaflet benefits seem to depend on synergies with the patient input component.In any case, the next step for this work is to test a selected optimized leaflet further, not to proceed directly to implementation [27].In a subsequent, larger-scale ORCT, an optimized information leaflet will serve as one of four candidate intervention components (e.g.alongside text messages, a therapy program, and a side-effect self-management website).For this ORCT, the primary outcome of interest will be adherence to AET, and the target of the information leaflet will be medication beliefs [5]; enabling the hypotheses linking the antecedents we consider here (e.g.medication beliefs) to adherence to be tested.
Third, these results highlight the idea that interpreting ORCT results to mean that certain components are "inactive" may be unhelpful.DAIVE's standard output (Table 3) offers a reminder that, when activity is defined in terms of desirable main effects and/or synergies in interaction effects, the differences between an intervention version that contains a particular component and another version that does not may be small.Therefore, to interpret such a difference as a signal of "inactivity" may be incorrect.

Limitations and future directions
The sample of healthy women could limit generalizability of findings to women with breast cancer, who may have had differing beliefs about the medication, more prior knowledge about AET, and differing satisfaction with information.However, the baseline beliefs differential score was not significantly different between those who did (n = 79) and did not report a diagnosis of breast cancer, as discussed elsewhere [13].Furthermore, the leaflet will now undergo further testing in a larger ORCT in women with breast cancer [27].Additionally, we did not assess participant's baseline AET knowledge, which could have impacted our findings.
It is currently usual practice in MOST for the preferences for multiple outcomes to be decided by the intervention scientists leading an ORCT [14]; our initial preferences for the three outcomes were defined by two investigators (S.M.C.G. and S.G.S.).It is plausible that, if other interested parties (e.g.clinicians and women with breast cancer) had been included in assigning relative importance to outcomes, we could have come to a different choice for the optimized intervention.Future work should explore the best strategies for incorporating wider perspectives in the determination of preferences in intervention optimization, likely drawing on a large literature in stakeholder engagement [39,40], and preference elicitation [41,42].
Furthermore, we relied on the number of candidate components in a given leaflet to indicate the efficiency (fewer components) versus complexity (more components) of that leaflet.While this strategy may be appropriate given the inevitable constraints on patients' time and attention, it may also not fully capture patients' subjective experiences.For example, it could be that candidate components (say, benefits versus concerns) produce different degrees of perceived burden.If we had quantified the burden associated with different leaflet versions more directly, we might have been able to strategically balance expected value with burden.It could be that this would have further affirmed the choice of a leaflet with fewer components (side-effects and patient input, versus diagrams, benefits, and patient input) set to their higher levels; of course, this could also have informed selection of a different optimized leaflet, especially since the top two contenders for the optimized leaflet performed so similarly in terms of expected value.In general, investigators using MOST may benefit from devoting some thought in the Preparation phase to how they might anticipate proceeding with selection of an optimized intervention in the event that two interventions perform similarly.One option would be to quantify uncertainty more explicitly than we did here; future work could expand on these possibilities.

Conclusions
Based on our preferred combination of medication beliefs, objective knowledge, and satisfaction with information, the optimized leaflet contained the higher levels of the side-effects and patient input components.Consistent with the decision-making reported by Green et al. [13], the patient input component showed the strongest evidence for activity, with further evidence of synergies with other components.Using DAIVE to consider multiple valued outcomes highlighted potential in the side-effects component, which may have been otherwise overlooked.

Figure 1
Figure1Credible intervals for the estimated main and interaction effects on beliefs about AET.

Figure 2
Figure 2 Credible intervals for the estimated main and interaction effects on objective knowledge.
This report is independent research supported by the National Institute for Health Research NIHR Advanced Fellowship, Professor S.G.S. NIHR300588.S.G.S. also acknowledges funding support from a Yorkshire Cancer Research University Academic Fellowship.S.M.C.G. acknowledges receipt of a Health and Behavior International Collaborative Research Award, sponsored by the International Behavioral Trials Network.The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the National Institute for Health Research or the Department of

Table 1
Experimental conditions in 25factorial design and number randomized to each condition [13].Each component had two levels: on versus off, or enhanced versus basic.Table adapted from Green et al.[13].

Table 2
Expected values for all 32 versions of the information leaflet using multiple valued outcomes Note.D = Diagrams; B = Benefits; SE = Side-effects; C = Concerns; P = Patient.Cells highlighted in green indicate factors set to their higher level.The cell highlighted in grey identifies the optimized leaflet, or the version with the largest expected value.

Table 3
Systematic variations in relative importance for three outcomes

Table 4
Preference scenarios that deviate from our initial preferences