A Comprehensive Qualitative Review of Studies Evaluating the Impact of Local US Laws Restricting the Sale of Flavored and Menthol Tobacco Products

Abstract

Objectives

To assess the quality of evidence on the effectiveness of local US laws restricting the sale of flavored tobacco products.

Methods

We conducted a systematic search and qualitative scoping review of English-language papers published through May 2020 that evaluated flavored tobacco sales policies implemented by US jurisdictions during 2010–2019. We constructed a conceptual model for flavored and menthol tobacco sales restriction outcomes, assigned GRADE quality of evidence ratings to policy outcomes evaluated through the included studies, and summarized factors that might explain weak or inconsistent findings.

Results

We found moderate to high quality of evidence associating policy implementation with reduced availability, marketing, and sales of policy-restricted products, and decreased youth and adult tobacco use of these products; however, policy exclusions and exemptions, implementation challenges, tobacco industry actions (e.g., marketing of concept-named flavored products; exploiting policy exemptions for certain store types), and consumer responses (e.g., cross-border or illicit purchasing) might undermine or mitigate intended policy effects.

Conclusions

Flavored and menthol tobacco product sales restrictions implemented and evaluated in US jurisdictions appear to have achieved some of their intended outcomes; however, deficiencies in study designs, methods, and metrics could contribute to equivocal findings on quality of evidence associating policy implementation and outcomes. Gaps in the evidence are beginning to be filled with research using more rigorous study designs, improved measurement and analytic methods, and longer-term follow-up.

Implications

In the absence of comprehensive federal action, US jurisdictions have the obligation to restrict flavored and menthol product sales to protect vulnerable populations from tobacco-related harms. The considerable expenditure of financial resources, political will, and time dedicated to policy adoption and implementation argue for evaluation studies designed to maximize the quality of evidence. This review offers generalizable insights into evaluation findings that can inform efforts to enhance tobacco control policy implementation and impact in the US and globally.

Introduction

Flavored tobacco products, including menthol-flavored tobacco products, are widely available for sale in US retail settings and are especially attractive to youth.¹ Electronic nicotine delivery systems (i.e., e-cigarettes or “vapes”) are the tobacco products most commonly used by US youth. The majority of middle and high school students use flavored e-cigarettes,^1–3 with fruit (73.1%; 75.6% prevalence, respectively); mint (55.8%; 46.5%); menthol (37.0%; 23.5%); and candy, desserts, or other sweets (36.4%; 47.2%) reported as the most common flavors.⁴ In the past decade, US sales of menthol cigarettes have increased, as have sales of flavored and menthol cigarillos and little cigars, flavored chewing tobacco, and flavored e-cigarettes.^5,6 Disparities in use of menthol cigarette and other favored and menthol tobacco, exacerbated by tobacco industry marketing practices, are especially problematic for certain US subpopulations, notably Black/African American smokers, as well as for LQBTQ and other vulnerable populations, who disproportionately suffer from tobacco-related disease and death.^7–10

In the absence of comprehensive US federal actions to remove flavored (including menthol) non-cigarette tobacco products, such as cigars and e-cigarettes, and menthol cigarettes from the marketplace, many jurisdictions have adopted policies to restrict the sales of flavored and menthol tobacco products. As of March 31, 2021, 336 jurisdictions in the United States (states and local counties and municipalities) had passed restrictions on the sale of flavored tobacco products some of which restrict sale of menthol tobacco products.¹¹ Moreover, in response to the e-cigarette, or vaping, product-use associated lung injury (EVALI) outbreak which began in August 2019,¹² several US states implemented temporary e-cigarette sales restrictions in 2019.¹³ Additionally, to address the dramatic growth in reported use of e-cigarettes among US youth,⁴ more than 40 US jurisdictions have banned the sale of all e-cigarettes, regardless of their flavor characteristics.¹¹

Coincident with the proliferation of these restrictions is the growing evidence demonstrating the impact these policies have on retailer compliance, product availability, marketing, sales, and population health outcomes. While a recent scoping review summarized the effects of implemented and hypothetical policies restricting the sale of menthol tobacco products,¹⁴ there has yet to be a comprehensive review evaluating intended and unintended outcomes of implemented US flavor and menthol restrictions, including those with exclusions of certain tobacco products or exemptions of certain retailer types or locations.

The purpose of our review is to provide a comprehensive analysis of published papers reporting results of evaluation studies of actual, implemented flavored product sales laws, including menthol tobacco product sales restrictions, in the United States. This review is designed to inform advocates, regulators, public health practitioners, and policymakers about the degree to which policy implementation has achieved intended outcomes and, in some cases, prompted actions by the tobacco industry and consumers that could undermine policy effectiveness.

We examine the strength of current evidence on the effectiveness of flavored and menthol tobacco sales restrictions to achieve intended policy outcomes. To organize our review, we developed a conceptual model (CM) for flavored/menthol tobacco sales restrictions, illustrating the presumed causal pathways linking policy implementation and intended immediate, short-term, intermediate, and long-term public health outcomes (Figure 1). The model hypothesizes that retailer education, compliance checks, and enforcement actions will result in decreased flavored and menthol tobacco product availability, decreased youth access to these products, decreased marketing and advertising, and decreased exposure to marketing and advertising. These immediate outcomes are expected to lead to decreased appeal, susceptibility, initiation, and established use of tobacco products among individuals in areas subject to the policy, especially youth. The CM also predicts that decreased availability, access, and marketing and advertising of flavored and menthol tobacco products will lead to increased quit attempts and successful quitting among users.

Figure 1.

Conceptual model for flavored tobacco sales restrictions.

By illustrating how policy implementation is associated with outcomes, we identify where there is sufficient evidence of intended policy effects and examine gaps in the literature. Our analysis also identifies how policy exclusions and exemptions, implementation challenges, industry reactions, and consumer responses might be explanatory factors for the outcomes reported.

Methods

We conducted a qualitative scoping review,¹⁵ which was not pre-registered with PROSPERO but was guided by the Preferred Reporting Items for Systematic Reviews, and Meta-Analyses (PRISMA) statement.¹⁶

Data Sources

We searched PubMed, EBSCO CIHNAL, and Web of Science databases for peer-reviewed articles published from January 1, 2000 to May 31, 2020, an expansive time frame informed by publication dates of early scientific discussions of the use flavorings in tobacco products to enhance product attractiveness to youth and vulnerable populations.^8,17–19 We used a combination of search terms (listing abbreviated here for space considerations): policy, law, ordinance, rule, restriction, ban, regulation, legislation; adoption, implementation, enforcement, compliance; flavor, menthol; tobacco products, cigarettes, cigars, electronic cigarettes, smokeless tobacco; [outcomes] sale, availability, marketing, advertising, susceptibility, appeal, use; and retail, store, consumer, industry. We also reviewed reference lists of key review articles identified through database searches to detect additional candidate papers.

Inclusion/Exclusion Criteria

Two authors reviewed every abstract to assess which candidate papers to include in the full-text review based upon pre-determined inclusion criteria: English language, peer-reviewed paper; published between January 2000 and May 2020; reporting on evaluation of policy-intended outcomes or unintended policy consequences of state or local laws implemented in the United States. We excluded studies of non-US policies to reduce the variation in influence of national tobacco control policy, history and context. We also excluded studies that evaluated the impact of national policies (e.g., the 2009 US federal ban on flavored cigarettes) or projected the impact of hypothetical policies (e.g., surveys asking smokers to predict their response to a possible ban on menthol cigarettes).

Data Synthesis

We reviewed the methods and findings for each study and categorized these based on the CM outcome(s) measured. Four authors (E.B., A.F., L.S.R., B.R.) reviewed each paper that met criteria for full review (see below) and prepared summary statements. The first author conducted a thorough review of all data extracted from the articles, confirmed measures, and resolved differences in summary wording across articles. We assigned qualitative ratings based on the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach²⁰ to characterize the quality of evidence regarding the association between policy implementation and each CM outcome. Although this approach was originally designed to guide evidence-based clinical decision-making, we determined that the well-documented GRADE rating process and qualitative ratings had utility for assessment and labeling of evidence quality in our tobacco policy evaluation scoping review. To assign GRADE rating and adjustments, we considered the factors that could contribute to strength-of-evidence ratings for the relevant CM outcome, as well as factors that could explain weak or inconsistent findings. These factors included: features of the policies themselves; study design and measurement methods; variations in policy implementation and enforcement; evidence of policy effect; and evidence of unintended policy effects (industry actions or consumer responses). Finally, we summarized findings of unintended policy consequences detected by studies.

Results

Study Sample

Our systematic literature search yielded 138 abstracts for review. We identified nine other eligible papers by reviewing citations in key papers. Of the 147 abstracts screened, 126 records were excluded, each for one or more reasons: not an evaluation of specific flavored/menthol tobacco sales policies (n = 89); not an evaluation of a US state or local policy (n = 16); not peer-reviewed (n = 4); and/or laboratory, hypothetical policy survey, or simulation study (n = 16). We conducted full-text reviews of the remaining 21 papers, excluding five that evaluated only policy implementation (not any CM outcomes), which yielded 16 papers eligible for full review and synthesis. Of these, 15 papers reported on studies associating policy implementation with intended outcomes (11 of which also report on unintended policy consequences), and one paper reported only on unintended policy consequences (Figure 2).

Figure 2.

Flowchart for selection of outcome studies evaluating US state or local laws restricting sales of flavored tobacco.*Exclusion reasons (papers could have multiple reasons for exclusion): not an evaluation of specific flavored/menthol tobacco sales policies (n = 89); not an evaluation of a US state or local policy (n = 16); not peer-reviewed (n = 4); and/or laboratory, hypothetical policy survey, or simulation study (n = 16).

Policy Jurisdictions

The included studies cover sales restriction polices in 48 US jurisdictions (Table 1). All policies were implemented between 2015 and 2019, with the exception of the 2010 policy in New York, NY (NYC) and the 2013 policy in Providence, RI. As of the date of this review, policies in New York City, Providence, and Massachusetts excluded certain flavors (e.g., menthol) or product types (e.g., e-cigarettes). Most policies have exemptions for some adult-only establishments such as tobacco shops or smoking bars, and the Chicago policy has a place-based exemption for retailers more than 500 feet from a high school. Only the San Francisco policy applies to all flavors, product types, and retailers. (See citation 11 for policy coverages, exemptions, and exclusions current as of March 31, 2021.)

Table 1.

US Local Flavored and Menthol Tobacco Product Sales Restrictions with Published Outcome Evaluation Studies

Policy location	Implementation date(month year)	Policy-restricted tobacco products	Excluded tobacco products	Exempted retailers	Published evaluation studies[reference #]
New York, NY	Nov 2010	Flavored non-cigarettes	Menthol tobacco; ENDS	Tobacco bars with >10% gross income from tobacco sales	Farley (2017, 2018, 2020); Kurti (2019); Rogers (2017) [16; 24; 26; 27; 30]
Providence, RI	Jan 2013	Flavored non-cigarettes	Menthol tobacco	Smoking bars	Pearlman (2019); Rogers (2020) [20; 28]
Minneapolis, MN	Jan 2016	Flavored non-cigarettes	Menthol tobacco	Adult-only tobacco shops with ≥90% tobacco sales revenue	Brock (2019) [14]
St. Paul, MN	Apr 2016	Flavored non-cigarettes	Menthol tobacco	Adult-only tobacco shops with ≥90% tobacco sales revenue	Brock (2019) [14]
Lowell, MA	Oct 2016	Flavored non-cigarettes	Menthol tobacco	Adult-only retail tobacco stores with ≥90% of sales from tobacco products	Kingsley (2019) [18]
Massachusetts (38 municipalities)	Oct–Jun 2016	Flavored non-cigarettes	Menthol tobacco	Adult-only establishments such as bars, vape shops; tobacconists	Kingsley (2020); Usidame (2019) [19; 25]
Boston, MA	Jan 2017	Flavored non-cigarettes	Menthol tobacco	Retail tobacco store or smoking bar	Kephart (2019) [17]
Chicago, IL	Feb 2017	All flavored and menthol	None	Retailers >500 feet from high schools or ≥80% sales from tobacco products	Czaplicki (2018) [23]
Duluth, MN	Jun 2018	All flavored and menthol	None	Adult-only tobacco shops with ≥90% tobacco sales revenue	D’Silva (2020) [15]
Falcon Heights, MN	Jul 2018	All flavored and menthol	None	Adult-only tobacco shops with ≥90% tobacco sales revenue	D’Silva (2020)[15]
Minneapolis, MN	Aug 2018	All flavored and menthol	None	Adult-only tobacco shops with ≥90% tobacco sales revenue; adult-only liquor stores	D’Silva (2020) [15]
St. Paul, MN	Nov 2018	All flavored and menthol	None	Adult-only tobacco shops with ≥90% tobacco sales revenue; all liquor stores	D’Silva (2020) [15]
San Francisco, CA	Jan 2019	All flavored and menthol	None	None	Vyas (2020); Yang (2020) [21; 29]

Studies of Policy-Intended Impact on Conceptual Model Outcomes

Each paper addressed one or more indicators of CM outcomes, with 14 papers evaluating immediate outcomes, seven papers reporting on the association between policy implementation and intermediate outcomes, and four papers reported on evaluations of long-term outcomes. Details on data sources, main outcome measures, study designs, and evidence of intended effects are shown in Supplemental Table 1, with studies nested under one or more CM outcome areas.

Immediate Outcomes

Ten studies evaluating policy effects on flavored or menthol product availability used retail observational methods (n = 8)^21–28 or product purchase attempts (n = 2)^29,30 to measure this outcome. One study measured flavored product availability through collection of discarded product packages in public areas of a city with a flavored product sales restriction policy.³¹ Eight retail observational studies operationalized availability as the proportion of policy-affected retailers displaying for sale at least one policy-restricted flavored or menthol tobacco product (hereafter, “store-level availability”).^21–28 Three studies also operationalized availability as the number or proportion of restricted flavored or menthol tobacco products observed in the store’s displayed inventory (hereafter, “product-level availability”).^21,24,25 Five studies used retail observational methods to evaluate policy effects on marketing/advertising of flavored or menthol tobacco products, typically operationalized as the presence of tobacco marketing materials or advertisements within or outside of stores.^{21,22,24,30,32}

Intermediate Public Health Outcomes

Four studies evaluated policy effects on intermediate public health outcomes, three of which evaluated policy effects on consumption of restricted flavored or menthol products, as measured by tobacco product sales at the retail level in policy-affected areas.^33–35 No studies measured changes in self-reported, individual-level product consumption in areas subject to the flavored/menthol product sales restrictions. A single study measured self-reported, post-policy quitting behavior among a sample of tobacco users in the affected jurisdiction.³⁶

Long-term Public Health Outcomes

Four studies in our review evaluated the impact of flavored-only or flavored including menthol tobacco product sales restrictions on tobacco use prevalence. Two of these studies used pre-post designs, surveying high school students in communities subject to the sales policy before and after policy implementation, but these studies lacked no-policy comparison samples.^27,33 A third study in this group surveyed high school students drawn from representative cohorts of school classrooms in a policy-affected community and a matched, non-policy comparison community; however, both survey waves were conducted after the policy was implemented.²⁵ In the fourth study, a retrospective design was used to assess changes in self-reported tobacco product use among a convenience sample of young adult tobacco users living or working in a city covered by a comprehensive sales restriction.³⁶

Unintended Consequences of Policy Implementation

Eleven papers reported on indicators of unintended policy consequences, offering insights into why certain policy-intended outcomes were not fully achieved.

Industry Actions

Several studies explored the association between policy implementation and availability or use of products with ambiguous or modified package labeling. In studies of the NYC flavored sales policy, for example, investigators raised anecdotal reports of some retailers selling flavored products labeled with concept names rather than explicit names, such as “purple” instead of “grape,” following policy implementation, although instances of products with color descriptors were not detected in scanner sales data.³³ Nonetheless, a sample of non-cigarette tobacco products without explicit flavor names purchased in NYC two years post-policy had flavor chemical profiles similar to those seen in products with explicit flavor names, suggesting that the tobacco industry exploited policy loopholes, which allowed products without characterizing flavor names to be sold.³⁷ Seven years after policy implementation, 70.9% of NYC retailers were observed selling non-cigarette tobacco products with concept flavor descriptors that were not available previously, suggesting that these products were introduced as a response to the growing proliferation of local flavored sales restrictions.²³ Other retail sales data analyses provide evidence of substantial growth in availability and sales of non-cigarette tobacco products with concept-named flavor descriptors starting in 2008.^38,39 Similarly, a 2016 study found close to 10% of discarded cigar packages collected were “implicitly flavored” (i.e., had concept names, such as, “Blue-Mixx”), which suggests widespread use of concept-named flavored products in NYC years after the policy went into effect.³¹ Moreover, no study has demonstrated the complete, sustained elimination of restricted product availability or sales in policy-affected areas. Studies suggest that sales of restricted products remain stubbornly high relative to pre-policy periods and policy intent,^34,35 and other studies suggest that some retailers might be continuing to sell restricted products illegally.³⁶

There is some evidence of increased promotional efforts for products not restricted by flavored or menthol policies. For example, in one study, the prevalence of exterior retail ads for unflavored products in some cities was higher than the prevalence seen in comparison non-policy communities.²¹ Exemptions for certain retailer types have led to some unintended outcomes, including an increase in the number of tobacco shop license applications and efforts to qualify as an adult-only tobacco shop by altering the physical space of the retail establishment, exempting these locations from the local sales restriction policy.^22,40

Studies have identified other industry actions related to policy implementation, including distributors or manufacturers refusing to take back unsellable flavored product stock after policy implementation, hindering retailers who attempted to comply with policies.^24,25 As a result, retailers identified as being non-compliant with their local flavored product sales policy tend to be smaller, independent operators who have less leverage with distributors than larger chain retailers.²⁶

Consumer Responses

Two of the 12 studies examined unintended consumer responses to implementation of flavored-only or flavored including menthol sales policies, including product substitution of non-flavored or non-menthol products for flavored products. Retail scanner sales data collected in one city covered by a sales restriction showed an increase in average dollar sales of non-flavored cigars and pipe/roll-your-own tobacco following policy implementation.³³ Countering the argument that the observed changes in dollar sales could reflect price increases for these products, average weekly unit sales of non-flavored (e.g., tobacco-flavored) cigars grew from the pre- to post-policy period in this city, at the same time that average weekly unit sales of these products declined in a non-policy comparison area.³⁴

One study provides evidence that implementation of a flavored including menthol sales policy is associated with cross-border or alternative-source purchasing behavior by tobacco users. Among flavored or menthol tobacco users living or working in one city with a comprehensive flavored including menthol sales restriction, 15% reported purchasing these products online, and 12% purchased from retailers outside of city boundaries.³⁶

Quality of Evidence

Our review reveals variation in the quality of evidence supporting the association between implementation of flavored or menthol tobacco product sales policies in the US and intended outcomes (Table 2). The table displays our GRADE ratings of evidence quality and the factors that reduce or increase the quality of evidence offered by the published studies evaluating each outcome.

Table 2.

Quality of Evidence* from Outcome Evaluation Studies of US Local Flavored and Menthol Tobacco Product Sales Restrictions

Outcome	Quality of evidence	Factors that reduce the quality of evidence	Factors that increase the quality of evidence	Published evaluation studies [reference #]
Immediate
Decreased availability of flavored/ menthol tobacco products	Moderate-High	• Inconsistency of controls for bias risk (study designs) • Inconsistency in measurement rigor • Short follow-up	• Large magnitude of effect across studies • Does–response gradient (enforcement variation)	Brock (2019); Czaplicki (2019); D’Silva (2020); Farley (2020); Kephart (2019); Kingsley (2019, 2020); Kurti (2019); Pearlman (2019); Vyas (2020) [14–21; 23–24]
Decreased marketing/advertising of flavored/menthol tobacco products	Moderate	• Inconsistency of controls for bias risk (study designs) • Inconsistency in measurement rigor • Short follow-up	• Moderate magnitude of effect across studies	Brock (2019); Czaplicki (2019); D’Silva (2020); Kephart (2019); Usidame (2019) [14–15; 17; 23; 25]
Intermediate
Decreased consumption of flavored/menthol tobacco products	Moderate-high	• Inconsistency of controls for bias risk (study designs) • Indirectness of outcome measures (sales as proxy for consumption)	• Large magnitude of effect across studies	Farley (2017); Rogers (2017, 2020) [26–28]
Increased quit intentions and quit attempts	Low	• No controls for bias risk (study design) • Imprecision • Selective outcome reporting	• None	Yang (2020) [29]
Long-term
Decreased tobacco use prevalence	Moderate	• Inconsistency of controls for bias risk (study designs) • Inconsistency in measurement rigor	• Moderate magnitude of effect across studies	Farley (2017); Kingsley (2019); Pearlman (2019); Yang (2020) [18; 20; 26; 29]

* Quality of evidence ratings and adjustment factors based on Grading of Recommendations, Assessment, Development and Evaluation (GRADE) Working Group approach ²⁰.

Immediate Outcome: Product Availability

Several studies provide evidence of intended policy impact on store-level product availability. All relevant studies demonstrated significant reductions in availability, and/or significant differences between policy and non-policy areas. Product-level availability measures in these studies also showed evidence of intended policy effects. The evidence is of moderate-high quality because most observational studies of product availability reflect short-term effects occurring only a few months after full policy implementation, and used methods with only modest reliability and validity.⁴¹ One study showed high rates of policy-restricted product availability, including those labeled with explicit flavor names, years after policy implementation in this city,²³ a finding consistent with other longer-term studies of this policy^31,33,34 This suggests that observed short-term effects on restricted product availability might not be sustained, especially in light of weak policy provisions, inadequate enforcement, and industry actions to counteract these policies.

Immediate Outcome: Marketing/Advertising

In the studies measuring policy impact on the marketing of restricted tobacco products, there is generally positive, but inconsistent, evidence of an association. Among studies with stronger designs and analytic methods, results suggest that tobacco retailers in localities with flavored tobacco product sales restrictions are less likely to feature flavored marketing materials as compared to those not affected by these policies.^21,22,32 The absence of a strong, consistent effect on flavored tobacco product marketing in communities subject to sales restrictions, coupled with an increase in non-flavored tobacco advertisements in some communities,²² raises questions about whether these policies are sufficient to reduce exposure to ubiquitous tobacco marketing in the retail environment. We conclude that these studies show moderate quality of evidence of policy effect on marketing indicators because of concerns about study designs and measurement methods, and inconsistencies in the findings across the studies addressing this outcome.

Intermediate Outcome: Consumption

Although retail scanner sales data have been used for national or regional surveillance of flavored or menthol tobacco product consumption trends,^5,6,38,42 the three papers included in our review were the first to use scanner sales data to associate policy implementation with changes in product consumption (sales) at the local level.^33–35 None of these studies of community-level consumption of policy-restricted and total tobacco products using localized retail sales data demonstrate that policy implementation is associated with complete suppression of restricted flavored or menthol product sales following policy implementation. These studies offer moderate-high evidence quality on the impact of flavored or menthol policy restrictions given the limitations of retail scanner data. And, although trends in area sales data are considered a reasonable proxy for changes in individual-level consumption, they do not include online sales or sales from vape or specialty tobacco stores. Moreover, tobacco users may modify their purchasing behaviors, seeking retailers outside the policy-affected area or illicit sources. Thus, these studies using retail scanner sales data likely overestimate the policy-intended impacts on consumption of restricted products, even as they generally show significant, sustained levels of sales of restricted products in policy-affected areas.

Intermediate Outcome: Quitting

We identified only one study of the impact of a US flavored or menthol sales policy on quitting behavior among tobacco users.³⁶ Findings from this uncontrolled, retrospective study of tobacco users offers only low quality evidence that a comprehensive local flavored and menthol sales restriction can have intended effects on quit attempts and cessation among consumers.³⁶ No studies provided evidence on the impact of these policies on other intermediate outcomes (reducing youth access to tobacco products, susceptibility to using tobacco products, or initiation of tobacco use).

Long-term Outcome: Tobacco Use Prevalence

Four studies offer moderate quality of evidence that implementation of flavored or menthol product sales restrictions is associated with the ultimate intended outcome of these policies, decreased tobacco use prevalence.^25,27,33,36 Although most study designs were weak (e.g., post-policy only, or pre-post without comparisons) and insufficient to control bias risk, and some studies used unstable measures of self-reported tobacco use, in the aggregate these studies offer evidence that implementation of the evaluated policies is moderately associated with decreased tobacco use prevalence among youth.^25,27,33

Possible Explanatory Factors

A number of factors could explain the quality of evidence regarding flavored and menthol policy outcomes, as well as some unintended consequences.

Policy Features and Context

Variation in policy language, exemptions and loopholes, and poorly operationalized definitions of product restrictions can affect policy implementation and impact. Policies vary in their level of clarity, strength, and comprehensiveness^11,43,44 and may be unclear as to product definitions. Exemptions for certain retail settings or specific types of tobacco products increase the potential for youth to obtain restricted products through social sources or lead to increases in stores seeking exempt status.^22,40 Litigation surrounding flavor policies can also contribute to confusion regarding policy implementation.⁴⁵ Although there is some evidence of retailer noncompliance (i.e., sustained availability of restricted products)²⁸ and unintended consequences (e.g., illicit sales)³⁶ in San Francisco, implementation of similar comprehensive policies that avoid product exclusions or retailer exemptions and are actively supported by engaged governmental agencies and community advocates⁴⁶ appears likely to achieve greater impacts than non-comprehensive policies.

Policy Enforcement

Challenges in enforcement may also mitigate the impact of these policies; for example, how enforcement responsibility is assigned, funded, detailed in protocols, and conducted can all influence policy effects. Enforcement officials conducting inspections in the field typically rely on package labels that are sometimes ambiguous.^23,47 The proliferation of concept-named flavors and overall flavor definitions contribute to confusion among both retailers and enforcement agents.^23,24,47 Retailers have expressed uncertainty about which products are restricted and have encountered distributor resistance to taking back products.²⁴ Enforcement responsibility, training, and protocols vary with respect to proactive product inspections, retailer educational approaches, and violation consequences (citations or training).⁴⁸ Together, these factors can affect policy-intended and unintended outcomes, as well as how evaluation findings are interpreted.

Research Study Features

Our analysis suggests that imprecision or inconsistency in measurement of outcomes and the absence of rigorous study designs in the reviewed evaluations may obscure potential policy effects. Most studies, for example, assessed restricted product availability at the store level using observance of at least one restricted product as an indicator of product availability. Few studies used product-level measures of availability (i.e., inventory counts; retail sales data), a key proximal indicator of successful policy implementation. Likewise, no studies have precisely measured changes in individual-level flavored or menthol product consumption. One study reported tobacco use data from cross-sectional samples of youth surveyed prior to and after policy implementation, but survey items grossly measured flavored product use and were not stable across survey waves.³³ Other studies measuring tobacco use among youth or young adults have collected only post-policy data,^25,27,36 which limits our ability to assess policy impacts on the ultimate intended outcome. While a few studies have inferred changes in product consumption using community-level retail sales data,^33–35 these measures do not capture sales online or by all retailers (e.g., vape shops). This gap is especially important given that local flavored or menthol product sale policies appear to have unfavorable reach equity to various vulnerable subgroups, especially youth.⁴⁹

Studies in our review used non- or pre-experimental designs, such as single group, post-policy surveys, which provide minimal controls for sources of internal invalidity.⁵⁰ Few studies used simple quasi-experimental comparison designs, which, depending on the design, might better control for certain sources of invalidity, especially the risk posed by concurrent, non-study events and conditions (e.g., other policies implemented by higher levels of government) that could have confounding influences on policy-intended outcomes. This is of special concern given the real-world conditions under which these evaluation studies are conducted—an environment of rapid policy proliferation at all levels of government, an evolving product market, and unpredictable changes in consumer preferences. At the very least, evaluation studies of flavored and menthol tobacco product policies should include pre-post cross-sectional, or preferably cohort or time-series, designs. Two time-series and follow-up studies have documented decays in immediate outcome effects and/or the emergence of unintended consequences long after the start of policy implementation,^23,35 while others have detected policy-intended outcomes only after a resurgence in enforcement efforts years after the policy went into effect.²⁷ Thus, even changes in indicators of “immediate” intended outcomes (e.g., reduced flavored or menthol product availability) might need time to become apparent, arguing for longer-term evaluation studies. The significant expenditure of financial resources, political will, and time dedicated to policy advocacy, creation, adoption, defense, and implementation argue that policy evaluation studies be designed to maximize the quality of evidence and honor these investments in the policy process.

Discussion

This review of evaluation studies reveals that flavored and menthol tobacco product sales restriction policies implemented in US jurisdictions over the past decade have achieved some of their intended outcomes. Studies have also documented industry actions and consumer responses that may have undermined or mitigated policy impact. Moreover, while we identified many promising findings, especially for immediate policy outcomes, as of the date of our review published evaluation studies of US flavored and menthol sales policies had not yet provided unequivocal evidence of effects on ultimate intended outcomes of these policies (e.g., reduction in tobacco use prevalence among youth).

This review also identified several gaps in the literature. While most studies to date have evaluated policy effects on only a few immediate or intermediate outcomes, studies are also needed to evaluate the impact of flavored and menthol sales policies on youth and young adult access, initiation, and progression to established use for flavored or menthol tobacco products. Various studies have examined the impact of policy implementation across multiple papers addressing a given policy, such as in NYC,^23,33,45 Providence,^27,35 and Lowell,²⁵ yet no studies have evaluated how variations in policy implementation factors (e.g., the amount of retailer education, or enforcement activities) affect policy-intended outcomes. Our review highlights the need for a comprehensive study of the association between policy mechanisms (e.g., policy provisions, exemptions, implementation actions, and challenges) and outcomes, which could provide evidence-based guidance for improving policy development and implementation,^51,52 especially to address endemic health inequities associated with target marketing of flavored and menthol tobacco products.^8,53,54 Indeed, few studies have specifically assessed the health equity impacts of sales restrictions among population groups who disproportionately suffer from tobacco-related disease and death, such as Black Americans.⁵⁵ Recent studies that have investigated health equity impacts of flavored and menthol sales restrictions reveal, for example, that: (a) the rationale for menthol tobacco sales restrictions was not well-understood or universally supported by African American smokers in Minneapolis and St. Paul, MN⁵⁶; (b) compared to White, non-Hispanic youth, Black, Hispanic, Asian, and LGBTQ youth in various California localities with flavor policies reported greater difficulty in accessing policy-restricted flavored tobacco products⁵⁷; and (c) Black young adults in San Francisco were more likely than other young adults to continue using flavored cigars after implementation of the flavored and menthol sales restriction in that city.³⁶ Our review highlights the need for rigorously designed evaluation studies of flavored or menthol tobacco product sales restrictions that could inform improvements in policy scope and implementation and address persistent health inequities exacerbated by the ubiquitous availability and target marketing of flavored and menthol tobacco products.⁵⁸

Although this review has many strengths, it is not without limitations. First, because of the heterogeneity in measures, samples, and time periods used, we did not employ meta-analyses or other quantitative assessment of pooled effects across studies. Rather than calculating quantitative averages of effect sizes, we used qualitative GRADE ratings to characterize the quality of evidence generated by the specific studies that evaluated policy implementation on outcomes.²⁰ Using quality of evidence summary rating labels is consistent with descriptive, qualitative scoping reviews¹⁵ and enhances our study, which was focused on providing insights into how on real-world policy evaluations have assessed intended outcomes and illuminated unintended consequences, thereby offering a more in-depth, practical appraisal of evaluation findings to guide practitioners and researchers in future efforts. Second, the authors of this review have been directly or indirectly involved in several of the included studies. To help reduce the threat of any bias, multiple coauthors reviewed and rated each study using a standardized methodology. Third, this review excludes several evaluation studies reported at scientific sessions and relevant research submitted for publication but were not yet available for review. Moreover, since the end date for our systematic review (May 2020), several high-quality studies of flavor policy evaluations have been published, which—had they been available for our review and synthesis—might have raised our quality of evidence ratings on the associations between policy implementation and certain outcomes. These include recent outcome studies of flavor and/or menthol policy effects on: tobacco product availability in Providence⁵⁹; tobacco product sales in San Francisco⁶⁰; retail access to policy-restricted products in multiple California jurisdictions^57,61; and, importantly, reported use of tobacco products by youth in Massachusetts^62,63 and San Francisco.⁶⁴ Lastly, our review is limited to papers evaluating flavored and menthol tobacco product sales restrictions that have actually been implemented in the US. Had studies of policy impacts in other countries been included in our review—notably, several well-designed studies evaluating impacts of the 2017 Ontario, Canada ban on certain menthol tobacco products—our quality of evidence ratings might have been higher for some outcomes, such as product availability, sales, consumer quitting behaviors, and longer-term effects on tobacco product use.^65–71 Nonetheless, our synthesis of the evidence associating US policy implementation with intended outcomes, and our observations about how the tobacco industry attempts to interfere with, avoid, or subvert local sales restriction policies,⁷² are consistent with other global flavored tobacco policy reviews and are internationally generalizable.⁷³

Conclusions

Rigorous evaluation of real-world, local flavored or menthol tobacco product sales restrictions can inform the design and implementation of policies and regulatory actions at all levels of government in the US and elsewhere.⁷⁴ As the evidence base associating the implementation of flavored and menthol tobacco sales policies expands, data will be available to help optimize policy effectiveness. Although national and state-level policies are more visible and can trigger aggressive political and legal reactions,⁷⁵ these governments typically have more resources than local jurisdictions to defend and enforce policies and measure outcomes. This review provides further justification for nations and states to adopt, implement, and evaluate comprehensive flavored and menthol tobacco product sales restriction policies, such as recent actions by the US states of Massachusetts and California to ban the sale of most flavored and menthol tobacco products with limited exemptions.

While states and localities are often seen as “laboratories” for policy action, in the US only the Food and Drug Administration can set national product standards regarding ingredients such as flavors, which would help reduce some of the unintended consequences of ambiguous product labeling and cross-border purchasing. Until comprehensive national policies are in place, however, US jurisdictions must continue their efforts to create and implement policies restricting the sale of flavored and menthol tobacco products to protect the most vulnerable from the deadly toll of tobacco-related disease and death.

Supplementary Material

A Contributorship Form detailing each author’s specific involvement with this content, as well as any supplementary data, are available online at https://academic.oup.com/ntr.

Declaration of Interests

The authors have no conflicts of interest to report.

Funding

This review was supported by a contract from Truth Initiative to RTI International. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of Truth Initiative or RTI International.