Medical devices pose unique challenges for economic evaluation and associated decision-making processes that differ from pharmaceuticals. We highlight and discuss these challenges in the context of cardiac device therapy, based on a systematic review of relevant economic evaluations. Key challenges include practical difficulties in conducting randomized clinical trials, allowing for a ‘learning curve’ and user characteristics, accounting for the wider organizational impacts of introducing new devices, and allowing for variations in product characteristics and prices over time.
Since the mid-1990s, many health care systems in Europe and elsewhere have begun using economic evaluation to make decisions about which new technologies should be funded from the systems’ collective resources. Requirements for economic analysis has largely related to the need for manufacturers to submit economic studies to decision makers in order for their new technology or intervention to be reimbursed.1 In the United Kingdom, for example, economic evaluation plays a key part in the technology appraisal process of the National Institute for Health and Clinical Excellence (NICE).2 In this process, evidence on clinical and cost effectiveness is used to make assessments of value for money and to inform the decision about whether a given technology should be adopted by the National Health Service (NHS). Other countries in Europe, including France, Germany, and Sweden, operate national health technology assessment (HTA) systems similar to NICE to inform coverage and pricing decisions, and in more decentralized countries, such as Italy, regional authorities are increasingly adopting HTA.
To date, HTA, especially with regard to resource allocation decisions, has been applied largely in the context of pharmaceuticals.3 NICE's technology appraisal programme, with its broad consideration of all health technologies, including medical devices, is an exception, although there is growing discussion on the evaluation of devices in a number of jurisdictions. While the general methods of economic evaluation are well established,4,5 it is often their specific application that raises methodological challenges. In particular, most international guidelines for economic evaluation have been predominately written with pharmaceuticals in mind.6 Against this backdrop, the growing role of economic evaluation in policy making and its application to medical devices raises a number of practical and methodological issues. Indeed, Drummond et al.7 argue that the economic evaluation of devices raises additional (and unique) challenges that existing guidelines and HTA programmes frequently overlook.
To deepen the understanding of the differences between drugs and devices in economic evaluation and the implications for decision making, we examine some of the challenges outlined by Drummond et al.7 as applied to cardiac device therapy, based on a systematic review of available economic evaluations in the field of cardiology and other relevant evidence. The article is structured as follows. First, we outline and discuss the methods used in the review. We then turn to an examination of key challenges to economic evaluation faced by cardiac device therapy and conclude with a brief discussion on potential ways forward.
To identify key evaluation challenges with regard to cardiac devices, we reviewed all relevant full economic evaluations contained in the NHS Economic Evaluation Database (NHS EED), which is compiled and maintained by the Centre for Reviews and Dissemination in the United Kingdom. National Health Service EED conducts comprehensive literature searches of health and social science databases (e.g. Medline, EMBASE, Scopus), identifying studies exploring the economic aspects of health care treatments and programmes. Studies that are considered to be full economic evaluations (i.e. those comparing the costs and consequences of alternative health care programmes and treatments) are reviewed and a structured abstract produced.8 Given its systematic review of the literature and robust classification of studies, NHS EED represents a comprehensive and authoritative source of economic evaluations in health care; it contains around 7000 quality-assessed abstracts of full economic evaluations. Therefore, the NHS EED was considered to be a reliable and efficient way of identifying studies for the review.
In particular, we searched the NHS EED based on the following search terms: ‘cardiology’, ‘coronary’, ‘implantable cardioverter defibrillators (ICDs)’, ‘stents’, ‘cardiac resynchronization therapy (CRT)’, and ‘pacemakers’. No restrictions on year of publication were employed. Once the relevant abstracts were identified, the full published papers were obtained and reviewed to identify methodological challenges. To supplement the analysis of economic evaluations contained in NHS EED, we also drew upon NICE technology appraisals of relevant cardio-pacing devices, namely ICDs, pacemakers and CRT. The main practical and methodological issues identified in the literature are discussed in turn below.
Problems in conducting clinical trials
Based on the review, one of the most evident and frequently mentioned methodological hurdles is the difficulty in undertaking randomized controlled trials (RCTs). Although RCTs are considered as the ‘gold standard’ for evaluating health technologies and are typically pursued in the case of many major categories of devices in cardiac care, such as ICDs, CRT and stents, the predominance of non-randomized studies was one of the most frequently noted methodological challenges. Even in the case where RCTs are available and used in economic evaluations, available studies are frequently characterized by small sample sizes and short-term follow-up. For example, it was noted in several studies on ICDs that the evaluation was focused solely on selected patient populations, namely those deemed high risk, and that this limited generalization of evidence of benefit to other patient groups. In turn, this hindered evidenced-based selection of patients appropriate for the use of the technology.9–11 This was also an issue noted in studies on left ventricular assist devices and pacemakers.12,13 Moreover, smaller trials used in economic evaluation posed an additional challenge, in that although the studies were often powered to detect differences in mortality, they were not large enough to demonstrate statistically significant differences in cost effectiveness.11,14
The prevalence of short-term trials across a variety of cardiac devices was considered a barrier to fully substantiating the value of the technology, making it difficult to detect important differences in outcomes (e.g. survival, quality of life), and increased the level of uncertainty in the analysis.15–17 Zwanzinger et al.16 noted that short-term trials are problematic when the incremental costs of an intervention are largely the result of the initial intervention, but the benefits continue to accrue over time. This means that the incremental cost-effectiveness ratio (ICER) declines over time and could have important policy implications as decision makers often base coverage or adoption decisions on the ICER. A lack of longer-term data was identified as an important evidence gap by NICE in its appraisal of pacemakers, where the Appraisal Committee called for further studies to evaluate the effectiveness of single- and dual-chamber pacing at follow-up beyond 5 years (and, where possible, up to 10 years) for various outcomes (e.g. mortality, stroke, pacemaker syndrome).18 Furthermore, in the Institute's assessment of CRT, the Assessment Group analysis required extrapolation from the 36-month trial data to use in an economic model in order to assess lifetime cost effectiveness.19 Modelling is frequently used to project costs and benefits across a longer-term time horizon.
While some of these issues are also prevalent in the context of pharmaceuticals, issues of small samples and shorter-term studies are a particular challenge for medical devices. For example, RCTs for devices are often small because the initial patient population is not as large as many target populations for drug treatment. In addition, it is frequently much more difficult to obtain patient consent to enter RCTs, particularly if an invasive surgical procedure is involved.7 Another influential factor relates to the way in which devices diffuse into the health system. Unlike pharmaceuticals, there is no formal requirement to undertake RCTs to obtain market approval; rather, evidence on performance and safety are required for CE approval and are completed at the point of market entry. Consequently, for many devices, including cardiac therapies, their uptake into clinical practice occurs quickly and, at times, at a point much earlier than the completion (or initiation) of clinical studies. Although not explicitly investigated in our review, the case of transcatheter aortic valve implantation (TAVI) provides a good example, where its use is well established in many countries, despite the fact that there is a paucity of published RCTs.20 Once a technology becomes common or standard practice, it is no longer ethical to conduct, or prolong the duration of, an RCT. The fact that physicians have a general proclivity towards the use of new innovations only serves to encourage early diffusion, making long-term follow-up in randomized studies problematic.
Another key challenge in undertaking RCTs is that, unlike drugs, devices frequently undergo product modifications over time, some of which impact on efficacy and other important endpoints (e.g. costs, quality of life). As Drummond et al.7 maintain, at the point where a drug reaches Phase III of clinical development, its dosage, and mode of administration is typically established and, although there is always some degree of uncertainty, the trial results provide a fairly robust basis for conducting an economic evaluation. However, in the case of devices, products evolve quickly and modifications are often made on an incremental basis, based on either emerging clinical evidence or use in regular clinical practice. Several examples of this challenge were evident in the review.11,21–26 Mushlin et al.,11 for example, noted that their study results only served as indirect estimates of the value for money of ICDs, as the study took place during a ‘technical evolution’ of the technology when transthoracic placement and single-purpose defibrillators were replaced by transvenous, multipurpose devices. In recognition that CRT was not yet a mature technology at the time of their study, Calvert et al.26 highlighted that ongoing technological developments would likely impact estimates of cost (and, in turn, cost effectiveness) by reducing implantation times and failures, and by delivery more effective resynchronization to more eligible patients. Finally, in its appraisal of pacemakers, NICE deemed the available literature on the cost effectiveness of dual- and single-chamber pacing in relevant patient populations of limited relevance, in part due to the ongoing technology developments in dual-chamber pacing.18 These documented challenges suggest that in the case of devices, there is unlikely to be a substantial ‘steady-state’ period during which the device could be best evaluated in an RCT.7
Allowing for the ‘learning curve’ and user characteristics
Another unique characteristic of some devices influencing their economic evaluation is the fact that the efficacy of a device depends not only on the device itself, but also on how it is used.7 There is often a ‘learning curve’ associated with a device; for example, user skills and training with the new technology can have important impacts on a therapy's performance. The learning curve is particularly evident in the context of newer or particularly innovative technologies or techniques and in those requiring more surgical intervention (e.g. TAVI). The issue of the learning curve has important implications for the conduct of clinical trials, as discussed previously, and the robustness of their findings. In particular, the challenge of longer-term studies of devices makes it difficult to measure the impact of the learning curve, especially considering that physicians are often already using the device in practice concurrent with the conduct of RCTs. Drummond et al.7 contrast this situation with pharmaceuticals, where as long as the drug is provided in the correct dose, its efficacy relates specifically to the drug itself, not to the end user.
Hand in hand with the learning curve, the need to adjust for user characteristics can further complicate the design of RCTs and, if not, evidence on performance may not accurately reflect what will be achieved in actual practice. For example, Nichol et al.27 commented that the experience with CRT observed in the study population may not be applicable to other settings, since only experienced physicians participated in the RCTs. Moreover, several analysts noted that the nature of a single-centre study hampered the ability to generalize findings, including those influenced by particular user characteristics, beyond that particular setting. It may therefore be preferable to undertake more multi-centre studies, which are more common in large Phase III RCTs of drugs.7
Accounting for the wider organizational implications of introducing devices
An additional way in which devices differ from drugs is that the implementation of a new device can often have wider organizational implications.7 For instance, there may be a need for additional training of physicians or other health professions, or the introduction of a given device may require a hospital to reorganize services to accommodate the new technology or procedure. Larsen et al.28 highlighted that, in their evaluation of ICDs, they did not capture service changes resulting from use of the device. Namely, routine discharge was no longer employed and this modification reduced length of stay and overall costs.
In the case of some cardiac devices, such as ICDs and CRT, new implantation techniques must be learned and accommodated. Additionally, these devices are often implanted in catheter labs, where other procedures (e.g. percutaneous coronary intervention, ablations) are concurrently performed, posing potential capacity issues and service reorganization demands. Part of the demands for reconfiguring current services alongside the introduction of a new device is that they often need to be regularly checked or verified to ensure optimal therapy.29 This can be associated with additional human and financial commitments.
However, organizational impacts are rarely examined in economic evaluations and are arguably more relevant in the context of devices than drugs. Part of the reason for this is that organizational effects typically occur in the medium- to long term, and so they are not being captured or measured in trials of devices, which are, as discussed earlier, generally short term. In a recent article, Le Goff-Pronost and Sicotte30 advocated for consideration of organizational impacts in evaluating cardiac telemedicine applications.
‘Genericization’ and class effect
Another way in which devices are different than drugs is that equivalent clinical evidence may not be available for all products, making comparisons difficult. Those undertaking economic evaluations often ‘genericize’ recommendations in the absence of specific evidence to differentiate products, which may be a practice influenced by the experience of pharmaceuticals.7 Cowie et al.17 in an evaluation of single-chamber ICDs did, however, note that their findings could not be extrapolated to more sophisticated, new generation devices with probable better lead and circuitry technology, durability, and cost effectiveness. The appropriateness of extrapolating evidence from one device to another deserves meaningful consideration, as different devices, while having the same clinical indication or outcome, may have different properties or modes of action that cannot be ‘genericized’ without supporting evidence.7 For example, in their economic evaluation of TAVI, the Belgian Health Care Knowledge Centre (KCE) argued that reimbursement of the technology should wait until an ongoing RCT was completed.20 However, it was not clear as to whether the KCE would accept this as evidence of the effectiveness of TAVI in general, or just the particular device that was the subject of the RCT.
Price of devices and variations over time
In the case of devices, prices often change over time, due to market entry of new products, iterative developments, or to the ways in which devices are procured in different health systems. This differs from drugs, where prices do not often change until the product loses patent protection and there is typically less diversity in procurement mechanisms.7
These considerations can raise challenges for the economic evaluation of devices and ultimately influence decisions about pricing and adoption. Three studies in the review31–33 identified that transient device cost data, due to ongoing product developments, market dynamics, and reimbursement arrangements, limited the robustness of the cost-effectiveness analyses. Again, the shorter duration of most device RCTs likely would not reflect said price variations. If initial pricing or adoption decisions are based on cost-effectiveness evidence and a cost-effectiveness threshold, changing price dynamics could potentially lead to changes in the value for money assessment. For example, when NICE first considered drug-eluting stents, it concluded that they were cost effective as compared with bare-metal stents. However, under re-evaluation 4 years later, changes to the price differential between the two technologies impacted on the ICER. This was one reason why NICE concluded that drug-eluting stents were no longer cost effective unless the difference in acquisition costs between them and bare-metal stents was £300 or less.34
Other methodological challenges
While there are key differences between devices and drugs to consider in applying economic evaluation, as discussed herein, our review also highlighted additional methodological challenges that arguably face both types of technology. With respect to HTA in general, analysts often note difficulty in assessing the indirect and broader, non-medical socio-economic costs and benefits of health technologies, such as impacts on productivity, carer quality of life, and the economy. Either the evidence is not there to employ in evaluations or it is of poor quality. We saw evidence of this in our review and several studies commented on the need to address these issues. Of course, these outcomes are indeed technically challenging and resource intense to measure, but nevertheless important to consider in order to fully substantiate the potential value of new innovations.
In this article, we have outlined several key challenges to the evaluation of devices using select examples in cardiac device therapy and how they differ from drugs. These issues introduce greater uncertainty into the decision-making process for devices and may result in decisions that do not accurately reflect their true value to clinical practice, patients, and to society at large.
To address some of the existing hurdles to, and evidence gaps in, the economic evaluation of devices, several areas warrant further attention. In particular, attention needs to be paid to some of the challenges in conducting randomized clinical studies, greater consideration of observational studies, and strengthening current use of economic modelling. Use of observational studies, for instance, could allow for more effective capture and assessment of the impact of the learning curve and other medium- to long-term changes in device use. For example, two studies on drug-eluting stents noted that evidence gathered in actual practice would provide better estimates of restenosis rates and selection of a repeat revascularization procedure.32,33 Such approaches are of growing interest, with studies being increasingly sponsored by governments, industry, scientific societies, and other stakeholders. However, their rigour is not always sufficient and therefore resulting evidence sometimes lacks credibility in the view of decision makers.
Some of the challenges around the ongoing product modifications experienced by devices suggest that there is unlikely to be a ‘steady-state’ period where a device could be evaluated in an RCT. Therefore, as Drummond et al.7 suggest, it might be better to consider the evaluation of devices as an iterative process, updating clinical and economic estimates with emerging evidence in actual use. Again, this would require new or revised approaches, as it would be infeasible to conduct multiple trials to re-evaluate a device under changed circumstances. Strengthening the current use of modelling for devices could potentially provide a better way to capture iterative changes and accommodate ongoing device evaluation. Part of the challenge here is in devising new ways to incorporate emerging evidence into analyses and to update findings to decision makers to inform policy. While there are indeed technical hurdles to this process, procedural barriers would also need to be addressed. Namely, greater collaboration between relevant stakeholders (e.g. industry, decision makers, assessment bodies) and enhanced transparency in the use of models is warranted.
Finally, given the important potential organizational impacts of devices in hospitals or other health care settings, such considerations should be included in comparative economic evaluations. Indeed, the local organizational context may be important for harnessing the improved cost effectiveness of a device. Better longer-term observational studies may offer an opportunity to better assess organizational aspects. Furthermore, additional research is necessary to understand what organizational issues managers and health professionals encounter when introducing a new device, as well as how these considerations can best be incorporated into HTAs.
Conflict of interest: none declared.
This research was undertaken as part of an unrestricted grant to the London School of Economics (LSE) by Eucomed, the European Medical Devices Association. The LSE is a partner in the European Health Technology Institute (EHTI), an organization supported by Eucomed to further research into the economic aspects of medical devices.