Effects of an Increased Financial Incentive on Follow-up in an Online, Automated Smoking Cessation Trial: A randomized Controlled Study Within a Trial

Abstract Introduction Poor retention in clinical trials can impact on statistical power, reliability, validity, and generalizability of findings and is a particular challenge in smoking cessation studies. In online trials with automated follow-up mechanisms, poor response also increases the resource need for manual follow-up. This study compared two financial incentives on response rates at 6 months follow up, in an online, automated smoking cessation feasibility trial of a cessation smartphone app (Quit Sense). Aims and Methods A study within a trial (SWAT), embedded within a host randomized controlled trial. Host trial participants were randomized 1:1 to receive either a £10 or £20 voucher incentive, for completing the 6-month questionnaire. Stratification for randomization to the SWAT was by minimization to ensure an even split of host trial arm participants and by 6-week response rate. Outcome measures were: Questionnaire completion rate, time to completion, number of completers requiring manual follow-up, and completeness of responses. Results Two hundred and four participants were randomized to the SWAT. The £20 and £10 incentives did not differ in completion rate at 6 months (79% vs. 74%; p = .362) but did reduce the proportion of participants requiring manual follow-up (46% vs. 62%; p = .018) and the median completion time (7 days vs. 15 days; p = .008). Measure response completeness rates were higher among £20 incentive participants, though differences were small for the host trial’s primary smoking outcome. Conclusions Benefits to using relatively modest increases in incentive for online smoking cessation trials include more rapid completion of follow-up questionnaires and reduced manual follow-up. Implications A modest increase in incentive (from £10 to £20) to promote the completion of follow-up questionnaires in online smoking cessation trials may not increase overall response rates but could lead to more rapid data collection, a reduced need for manual follow-up and reduced missing data among those who initiate completing a questionnaire. Such an improvement may help to reduce bias, increase validity and generalizability, and improve statistical power in smoking cessation trials. Trial Registration Host trial ISRCTN12326962, SWAT repository store ID 164.


Introduction
Poor retention of participants in clinical trials and the resulting missing outcome data are consistently documented challenges. 1,2The impacts include potential selection bias in results, reduced statistical power and reduced reliability, validity, and generalizability of the trial findings.Schultz et al. 3 propose that a loss of greater than 20% potentially poses a threat to the validity of the study. 3However, all gains in collection of outcome data can have a meaningful impact 4 so it is important that options are explored to improve retention in trials.
Retention of participants can be more challenging when trials are conducted remotely or wholly online 5 and when there is minimal contact with participants due to noninterpersonal treatment delivery.For smoking cessation treatment, there are perceptions from participants that if they have not stopped smoking and their data are no longer useful, hence, the common application of a "missing = smoking" rule among those lost to follow-up. 6This further compounds the challenge of ensuring representative outcome data and retention rates in smoking cessation trials.Some online smoking cessation trials investigating remote treatments that have used incentives have achieved relatively high positive 6-month response rates, such as 77% of returning a saliva sample who received a £20 incentive, 7 and 88% retained who received a $25 incentive plus a $10 bonus if done within 24 hours of notification. 8However, a Cochrane review (2017) of webbased smoking cessation trials identified that the majority of included studies achieved between 50% and 80% follow-up rates. 9The extent to which incentives promote response in smoking cessation trials remains largely unknown, particularly for remote and online trials.1][12] For example, Bailey et al. 13 found postal response rates increased by 6%-10% through increasing the offered incentive from £10 to £20.
We embedded a randomized study within A trial (SWAT) 14 in an online and automated feasibility randomized controlled trial of a stop-smoking smartphone app called Quit Sense, to understand the impact of offering a £20 incentive compared to a £10 incentive on: (1) participant response on their smoking status at 6 months follow-up by all methods, (2) the number of participants requiring manual follow up, (3) completeness of response from all follow-up questions asked.The trial results will inform which incentive rate to use in a future definitive trial of the Quit Sense app.

Design and Participants
A two-arm parallel randomized controlled study within a host trial 15 was undertaken.The host trial recruited 209 patients to a randomized controlled trial comparing a Just-In-Time Adaptive Intervention smoking cessation smartphone app (Quit Sense), to online standard care for smoking cessation (NHS SmokeFree website) only. 16In the host trial, randomization was stratified by smoking rate (<16 vs. ≥ 16 cigarettes per day) and socioeconomic status (low vs. high).Recruitment, enrollment, baseline data collection, allocation, and intervention access delivery were automated through the study website.Outcomes were collected at 6 weeks and 6 months follow-up via the study website via a text message or email prompt or, if an initial response was not received, by telephone.
All participants of the host trial who had not withdrawn by the 6-week follow-up were randomized into the SWAT using minimization. 17Minimization ensured an equal balance between the SWAT (incentive) groups, accounting for the participants' host trial allocation group, and whether they responded at 6-week follow-up.Allocation to the incentive groups was carried out in R using the minirand package, and then the results were imported into REDCap. 18,19he researcher was unaware of incentive group allocation when undertaking manual follow-up calls although was not blinded to incentive group as they needed to allocate vouchers when follow-up was completed using the REDCap database.Participants were blinded to their incentive allocation up until the incentive was offered at 6-month follow-up and were not informed at any point that the incentive amount offered was different for some participants.
Participants were offered a £5 Amazon gift voucher for completing the 6-week follow-up.Participants were informed in the participant information sheet they would receive a £10 voucher code for completing 6-month follow-up, but as the SWAT was conceived after all participants were recruited, they were not aware of the additional £10 incentive offered to some participants.
Additional consent to the SWAT was not sought as this was deemed unnecessary as all participants would still receive the £10 6-month incentive as indicated and informing the £10 incentive SWAT arm that they received less than £20 arm could have biased the findings.The SWAT was approved by the Wales NHS Research Ethics Committee 7 (19/WA/0361) as an amendment to the host trial ethics application.
The detailed plan for the SWAT and incentive rates were reviewed and agreed upon by a patient and public representative on the host trial management group and the SWAT was registered on the SWAT Repository Store (ID 164).
The sample size of this SWAT was dependent on the sample size used in the host trial, and so a formal power calculation was not undertaken.
Refer to the host trial outcomes paper 16 for a description of host trial measures.

Procedure
Six and a half months after enrollment (referred to as the 6-month follow-up, allowing 2 weeks for setting and preparing for a quit attempt), all participants were sent an automated text message inviting them to complete the final questionnaire.This message differed between groups only by the incentive amount they would receive (£10 or £20) (see Figure S1 for text message wording).
An automated reminder text message and a text message explaining the researchers would contact them by telephone if they did not complete the questionnaire online were sent 4 and 7 days later, respectively.Participants stating E-mail as a preferred contact method were sent the questionnaire link and reminders by E-mail instead, although very few did.Two weeks after the questionnaire's due date, the Researcher attempted manual follow-up by telephone and, if unsuccessful, a final text message inviting them to respond to the primary smoking outcome question was sent.
Other strategies to try and improve overall response rates in the trial were used for all participants, including text reminders about participation and a "thank you" postcard.

Analysis
Sample characteristics at baseline (by incentive arm) and completeness of responses are summarized, though no statistical comparisons were planned or undertaken.For the overall response rate analysis, a logistic regression model was run, using Wald confidence intervals for the odds ratio (OR), including stratification variables as covariates in the model and any participant characteristics that appeared imbalanced between the different incentive groups (prognostic variables), with response (binary) as the dependent variable.For the other analyses, further logistic regression models were run using an analogous approach for assessing the effect of incentives on manual follow-up rates.Results from regression analyses are presented separately as unadjusted, adjusted for stratification variables, and then adjusted for stratification and potential prognostic factors.Finally, a Cox proportional hazards analysis was undertaken with the log-rank test to assess the effects of incentives on completion time.Outcomes are described as proportions or summary statistics with 95% CIs.

Results
Two hundred and four participants were included in the Quit Sense SWAT, 101 were randomized to the £20 incentive arm and 103 to the £10 arm.
The SWAT sample had a mean age of 41 years (range 18-61), 55% were female, 29% were in the lowest socioeconomic status category, 9% had other-than-White ethnicity and mean baseline smoking rate was 15 cigarettes per day (Table 1; Supplementary Table 2).
In the host trial, more participants provided data at 6 months than at 6 weeks.At 6 weeks, 149 (71%; 95% CI: 65%, 77%) were followed up and at 6 months, when the incentive rate was manipulated, this was 160 (77%; 95% CI: 71%, 82%).There were six host trial withdrawals, with five withdrawing prior to SWAT randomization (and so excluded) and one between 6 weeks and 6 months.
The log-rank test, which compares expected and observed questionnaire completions over time by incentive group, shows moderate evidence that there was a difference in survival probability between incentive groups, with the £20 incentive group showing on average a lower completion time (median: 7.0 days, IQR 0.08, 18.13) than the £10 incentive group (median: 14.9 days, IQR 0.15, 24.55), which was statistically significant (unadjusted p = .008;Table 2).This is confirmed by a Cox proportional hazards analysis showing that at any point in time, the £20 incentive group was 55% more likely to respond to the 6-month questionnaire, compared to those participants in the £10 incentive group (unadjusted HR = 1.55, 95% CI: 1.12, 2.15, p = .008).The failure plot (Figure S2) suggests that failure probability increased over time, with a higher probability of completing the 6-month questionnaire when close to 0 days.Adjusting for stratification and prognostic variables for the above analyses did not alter the findings meaningfully.
Completeness of data for questionnaire items other than the overall response rate were also higher in the £20 incentive group compared to the £10 group (Table S1).For example, 78% versus 68% of participants reported urge items data and 78% versus 66% completed EQ-5D-5L questions in the £20 and £10 incentive groups respectively, though between-group differences for response to the primary outcome were smaller (79% vs. 74%).

Discussion
Increasing the incentive for completing follow-up from £10 to £20 did not increase overall response rates at 6 months.However, it did reduce the proportion of participants requiring manual follow-up and the median completion time and may increase the completeness of some data.It is possible that incentives influence response to remote or non-interpersonal follow-up prompts more than response to interpersonal efforts.This is suggested by the beneficial effect of increased incentives on manual follow-up rates, where, if manual follow-up was not undertaken, a difference in overall response rates would potentially have been observed.Speeding up completion times could contribute to enhanced High et al.
validity through a in variability in response delay and would likely lead to other potential benefits such as a more rapid collection of saliva samples, when verifying those reporting abstinence.If higher completeness of data are achieved, this would enhance statistical power and validity through reduced missing data.While higher incentives increase the overall incentive costs, these could be, at least in part, offset by reduced researcher costs with fewer participants to manually follow up.Furthermore, it is possible that an incentive higher than £20 may increase the response rate to a satisfactory level that would avoid the need for any manual follow-up, though further research would be needed to determine this, and the optimum level and type of incentive, noting that there is likely to be a ceiling effect. 20The overall response rate in the £20 incentive arm was very similar to another UK digital cessation intervention trial with a comparable incentive 7 and the nonsignificant increase in response rates observed was in line with increased postal response rates found from increasing an incentive from £10 to £20 for an online intervention. 13his trial was automated, though where necessary participants were followed up by phone call.It is unclear whether the same incentive amounts would work as well in a different healthcare setting or population, particularly where motivations may be different.Response rates were in-line with or perhaps slightly higher than literature-reported rates for similar trials, 9 indicating that appropriate incentives may help to achieve an acceptable response and retention rate.
Strengths of this SWAT include robust randomization and intervention delivery fidelity.Arms were well balanced for most baseline characteristics and key imbalances accounted for in the adjusted analyses.Participants were also effectively blinded from incentive allocation, so their response was likely to reflect the effect of the incentive within the context of the trial.However, it is possible some participants may have recalled the original incentive from the participant information sheet before the SWAT was embedded in the trial, and those offered £20 may have felt the increased incentive was a bonus or upgrade, which could have influenced the effects observed.

Conclusion
While overall response rates were not increased, the potential benefit of a more rapid completion of follow-up questionnaires, a reduced need for manual follow-up, and potentially increased measure completion rates from a relatively modest increase in incentive costs may be worthwhile, particularly in the context of the high costs of running smoking cessation clinical trials and value of follow up data.a Logistic regression model was used, modeling the odds of response to smoking status at 6 months (self-reported smoking status: where nonsmoking is smoking no more than 5 cigarettes within the 6-month study period) and modeling the odds of manual follow-up at 6 months.Both analyses adjusted for differences in number of cigarettes smoked per day at baseline, socioeconomic status at baseline (stratification variables), heaviness index at baseline, gender (prognostic variables), and incentive group.
b Odds ratio (OR; the odds of response/manual follow-up for participants in the £20 incentive group is (OR) times that of the odds of response for participants in the £10 incentive group).c 95% Wald confidence interval for odds ratio.
The p-value is based upon a null hypothesis of zero difference.
e Hazard ratio (HR; at any particular point in time, the £20 incentive group are (HR) times more likely to respond to the 6-month questionnaire, compared to those participants in the £10 incentive group.If HR < 1 (or HR > 1), the £20 incentive group experiences a "risk" reduction (or "risk" increase) compared to the £10 incentive group).

1260
High et al.

Table 1 .
Participant Characteristics at Baseline a Low socioeconomic status was classified as individuals with a semiroutine or routine and manual occupation (grade five in the National Statistics Socio-Economic Classification), or who have never worked or are long-term unemployed.High socioeconomic status included all those not classed as low status.

Table 2 .
Effects of incentive value on response rate of the smoking status question at 6 months (self-reported prolonged abstinence), rate of manual follow up at 6 months and completion time (days) to completing the smoking status question at 6 months (in days)