Developing a decision support system to link health technology assessment (HTA) reports to the health system policies in Iran

Abstract

The recent increase of ‘Health Technology Assessment’ (HTA)-related activities in Iran has necessitated the clarification of policy-making process based on the HTA reports. This study aimed to develop a Decision Support System (DSS) in order to adopt evidence-informed policies regarding health technologies in Iran. The study can be classified as Health Policy and Systems Research. A core panel of seven experts conducted two separate reviews of relevant literature for: 1- Determining the potential technology-related policies. 2- Listing the criteria influencing those policy decisions. The policies and criteria were separately discussed and subsequently rated for appropriateness and necessity during two expert meetings in 2013. In the next step, The ‘Discrete Choice Experiment’ (DCE) method was employed to develop the DSS for the final technology-related policies. Accordingly, the core panel members independently rated the appropriateness of each policy for 30 virtual technologies based on the random values assigned to all the criteria for each technology. The obtained data for each policy were separately analysed using stepwise regression model, resulting in a minimal set of independent and statistically significant criteria contributing in the experts’ judgments about the appropriateness of that policy. The obtained regression coefficients were used as the relative weights of the different levels of the final criteria of any policy statement, shaping the decision support scoring tool for each policy. The study has outlined 64 policy decisions under 7 macro policy areas concerning a health technology. Also, 34 criteria used for making those policy decisions have been organized within a portfolio. DCE, using stepwise regression, resulted in 64 scoring tools shaping the DSS for all HTA-related policies. Both the results and methodology of the study may serve as a guide for policy makers (researchers), particularly in low and middle income countries, in developing decision aids for their own context-specific HTA-related policies.

Health policy and systems research, health technology assessment, decision support system, Iran, discrete choice experiment

Key Messages

• Considering the various aspects raised by a health technology, different policies with varying sets of criteria must be adopted in different areas. This reveals the complexity of a comprehensive technology-related policy-making process and the necessity of developing a decision support system.

• The study introduces a method strategy of developing a decision support system aiming to link the information provided through an HTA report to a holistic set of relevant policies.

Introduction

From the beginning of the 21^st century, particularly following the World Health Report 2000, a great attention at national and international levels has been given to strengthen the health systems as inherent components of sustainable development in each society (Haines et al. 2012). Essentially, a high performance health system needs to adopt deliberate policies at its different levels.

Among the wide range of issues concerning the health systems performance is the crucial and challenging issue of confronting a [new] health technology (in its wider meaning; including the related services) particularly in publicly funded health systems where there must be greater accountability for spending from public revenues. The rapid growth of medical expenditures has necessitated the need for wise decisions regarding the coverage of new health technologies -as a significant potential contributor to such growth- from one hand and, disinvestment from currently covered technologies with, for example, uncertain efficiency according to newly emerged evidence, from the other hand. The field of Health Technology Assessment (HTA) has basically been developed as a response to this need (Abelson et al. 2007).

HTA aims to inform health systems policy makers about the probable short- and long-term effects of employing a health technology (Giovagnoni et al. 2009). It especially evaluates the potential direct and indirect benefits and harms as well as uncertain outcomes of applying a novel technology or an evolved version versus its alternatives (Garrido et al. 2010). In other words, HTA should provide decision makers with a systematic analysis of different policy choices in terms of economic, social, ethical, legal and environmental aspects of recruiting a particular technology (Lehoux and Williams-Jones 2007).

Over the last three decades, HTA has gained a special place as a kind of policy-relevant studies particularly in developed health systems (Battista and Hodge 1999; Hutton et al. 2008; O'Donnell et al. 2009). Nevertheless, in many cases, HTA reports have only described the different aspects of a particular technology and failed to provide clear practical recommendations. Some experiences have shown that, HTA reports constitute only one of the inputs for the development of technology-related policies, and very often, they have not been considered as the most important input for policy making process (Gerhardus and Dintsios 2005). However, well conducted HTAs are potentially able to influence a wide range of health system crucial decisions.

Some health systems apply special arrangements to attain the maximum potential benefits of HTAs (O'Donnell et al. 2009). Typically, a set of explicit/implicit criteria are employed in order to derive different policy choices from HTA documents during a decision-making process (Devlin and Sussex 2011; Cromwell et al. 2015). Different countries’ approaches are contrasted by differences in policy choices, the criteria used, and the method of consensus development.

The increase in HTA-related activities in recent years in Iran (Doaee et al. 2013) has necessitated the clarification of the process of adopting distinct policies based on HTA documents.

The context: Iranian health and health technology system

There appears to be no effective mechanism established in order to regulate the health technologies pathway (from the entry phase up to the financing, service provision, and so on) resulting in a disorganized market in this field in Iran. Therefore, it seems difficult to have a precise and comprehensive understanding of the main characteristics of this market.

This may be somewhat due to rather short history of organized efforts in this field in Iran. The HTA-related activities in Iran officially started as a secretariat in 2007. Initially the HTA projects were conducted by trained faculty members in medical universities on the request of the Secretariat. In 2010, following a structural change in the Ministry of Health and Medical Education (MOHME), the HTA Office was embedded within the ‘Health Technology Assessment, Standardization and Tariff Department’ under the Deputy of Curative Affairs in MOHME. More details on HTA history, current structure and activities in Iran have been provided elsewhere (Doaee et al. 2013).

Healthcare system (financing and service provision) in Iran has a hybrid model structure. Iran has an integrated primary health care network of about 2,000 currently active rural and urban health centers, budgeted by the government and administered through medical universities under the supervision of the MOHME. Also, social insurance is a main feature of Iranian healthcare system offering a basic benefit package of health services to, currently estimated, about 95% of the population. In addition, the role of commercial insurance in healthcare financing has been increased especially in the recent decade (Jadidfard et al. 2013). The latter mainly seems attributable to the inefficiency of the two other mentioned structures. In Iran, the health system has an integrated and centralized nature with the responsibility for policy-making, planning and supervising the related activities in all areas and in both public and private sectors assigned to the MOHME. In 2014, 6.9% of Iranian GDP was spent on health, of which only 41.2% was funded from public resources (World-Bank-2016).

Considering the various and complex aspects raised regarding a health technology, if a system tends to appropriately face the issue in a comprehensive way, different policies must be adopted in different areas. Potentially, each policy decision entails a different set of criteria (with different weights) that should be rationally considered in its adoption. Numerous policies with varying sets of criteria reveal the complexity of a holistic and consistent technology-related policy-making process and the necessity of developing a decision aid tool in this field. This study aimed to develop a Decision Support System (DSS) in order to adopt evidence-informed policies based on HTA reports. We presumed that such national model can significantly help enhance the connection between HTA and policy making in Iran.

Methods

This study can be classified under the rather emerging concept of Health Policy and Systems Research (HPSR). We had a multidisciplinary and multi-affiliation core panel of seven experts in the fields of health policy and health economics with varied backgrounds and familiar with the theoretical and practical aspects of HTA. The panel members’ affiliations included the knowledge management units in the main Iranian medical universities, scientific committee of HTA office and relevant policy positions in the MOHME.

Conceptual framework

In the beginning of the study, a conceptual framework was developed for the process of HTA-related policy-making. The resulted product is depicted in Figure 1. Accordingly, development of technology-related policies occurs through a three-step process. First and basically, in order to provide the evidence input (both local and global) there is a need to systematically synthesize the huge, raw and often fragmented data and information about the different aspects of the technology under consideration. This task is met by the knowledge management units in medical universities. This ‘assessment’ step results in the development of HTA report. In the next step (appraisal), based on the evidence provided, members of the scientific committee of the HTA Office (MOHME) separately make judgments about the decision criteria (for all technology-related policies) within the health technology portfolios (expert input). Subsequently, an initial set of policies with potentially different recommendation levels is generated by the aid of the Decision Support System (DSS). Eventually, the final decisions are made by a relevant policy council that may add considerations (system input; including politics, legislations, mandates, higher policy directives, and ideology) to the draft of policies in order to develop the final technology policy profile.

Figure 1.

The Conceptual framework developed for linking HTA reports to health system policies.

This study aimed to generate a DSS pursuing and supporting evidence-informed policy-making and expediting transparent policy development (step 3) based on health technology portfolios (step 2). The DSS is to aid rational policy making with respect to a holistic policy profile regarding the different aspects of a health technology based on the information provided through an HTA report.

Accordingly, the study was conducted in 3 main steps:

1. Development of an extensive list of technology-related policy choices: firstly, in the core panel and with the consideration of the Iran’s MOHME structure, the general areas of technology related policies were determined and classified into the following seven areas:

   • Policies to determine the technology status in the country’s health market;

   • Policies for provision/prescription of technology [-related services];

   • Policies of financing and reimbursement for technology related services;

   • Policies for regulating, tariff setting and pricing of technology related services;

   • Research policies concerning the technology;

   • Education policies concerning the technology;

   • Innovation policies concerning the technology.

Subsequently, the policies under each of these macro policy areas were determined via a brainstorming session led by the first author with substantial studies and experiences in the fields of HTA and HPSR, and the counselling assistance of the other members of the core panel, as well as a complementary review of relevant literature. For the review purpose, all types of literature including textbooks, reports of original research, reviews, discussion papers, workshop and institutional reports, and commentaries were potentially included in the review. The internet searches, using relevant keywords, were conducted through Google Scholar in September 2013. Grey literature, including upstream national policy documents, national development plans and governmental reports were also considered. The gathered materials were then converted into policy statements under any of the macro policy areas.

A meeting was then held in November 2013 with a purposive sample of 13 individuals (other than the core panel members) including the members of the HTA Office and individuals with relevant policy positions in MOHME and some with substantial managerial and policy experiences, or extensive education and research background in the field of HTA. The listed potential policies, already sent to them, were discussed during the meeting and finally the participants were asked to rate the appropriateness of each policy statement on a scale of 1-9. The policies with median scores of 7 and above were included within the final health technology-related policy profile package.

• 2. Listing the criteria influencing HTA-related policy decisions:in this step, the potential decision criteria for each of the technology-related policies were determined through a similar process, i.e. the brainstorming of the members of the core panel as well as a complementary literature review. Again, a meeting was held with the same participants as the previous step in December 2013. The criteria list was discussed for each policy and the participants were then asked to rate the necessity of each criterion on a scale of 1-9. Similarly, only the criteria with median scores of 7 and above remained within the list of criteria for each technology-related policy. A single list of all the criteria influencing all policies was then prepared and organized within a taxonomy. The levels of each criterion were determined. Accordingly, The HTA portfolio was formed as the basis for decision making about all health technology-related policies.

• 3. Decision modeling:in this step, ‘Discrete Choice Experiment’ method was used for decision modeling (judgment analysis of the study panel members) and subsequent development of a Decision Support System (DSS). Actually in this step, we were seeking for the final set of the criteria as well as their weights in influencing the appropriateness rate of each particular policy for any technology. For this purpose, firstly, 30 virtually completed technology portfolios were created. In order to generate a virtual portfolio, one level of each criterion in the HTA portfolio was selected randomly. Afterwards, all seven members of the core panel independently rated the appropriateness of each policy statement on a scale of 0 to 100 for each of the virtual portfolios (hypothetical technologies) after discussing it. Accordingly, 210 data sets (7 raters for 30 technology portfolios) were produced for each policy. In order to obtain statistically significant criteria and the weights of their different levels [in appropriateness ratings made by the panel members], these data were separately analysed for each of the policies using stepwise regression model and the following formula:

Policy_k Score = Σα_jC_j, where

C_j = j_th criterion

α_j = Regression Coefficient (weight) of j_th criterion

For any particular policy, only the criteria identified in step 2 as being potentially relevant were included in the analytical model (above formula). The analysis resulted in a [minimal] set of independent criteria contributing in the experts’ judgments about the appropriateness of each policy. For each policy statement produced in the first step of the study, the obtained regression coefficients were used as the relative weights of the statistically significant criteria (the highest attainable value of any significant criterion level on an approximate scale of (±) 0-100), shaping the decision support tool for that particular policy. For this purpose, in order to produce a user-friendly decision rule, the regression coefficients of significant criteria levels for each policy were scaled and rounded up (using the formula shown in the footnote of Table 3) to achieve a maximum score of about 100 for all relevant criteria of a policy.

Results

69 health technology-related policies were initially drafted by the core panel and organized as statements under topics within seven major policy areas. Table 1 provides the taxonomy of the policies as well as their median appropriateness rating scores (MASs) by the expert panel (first meeting). 64 policies with MASs≥ 7 were included in the final package of policies (the output of the first phase of the study).

Table 1.

Taxonomy of health technology-related policies and their inter-dependencies, as well as the median appropriateness scores (MAS) given by the experts in the first meeting. (Note: five policies with MAS < 7 do not have classification code and are excluded from the final policy profile)

Taxonomy of HTA-related Policies		Dependency	MAS
1. Policies to determine the status of T.* in the country’s health market
1.1. Technology entry into the research field
1.1.1	T. X† is permitted to enter the country for research activities.	–	7
1.2. Technology entry into the healthcare market
1.2.1	T. X is Not permitted to enter the country’s healthcare market.	–	9
1.2.2	T. X is permitted to enter the country’s healthcare market.	–	9
1.2.3	T. X is allowed to enter the country’s healthcare market, provided that trained manpower is available.	–	8
1.2.4	T. X is allowed to enter the country’s healthcare market, provided that the appropriate regulatory (supervisory) mechanisms are established.	–	8
1.3. The way T. is supplied to the country’s health market
1.3.1	T. X should enter the country’s health market through the imports of its final products.	1.2.2	8
1.3.2	T. X should enter the country’s health market through domestic production.	1.2.2	8
1.4. Technology suppliers to the healthcare market
1.4.1	The authorized and preferred importing companies of T. X. should be determined.	1.3.1	7
1.4.2	The authorized and preferred manufacturing companies of T. X. should be determined.	1.3.2	7
1.5. Technology brand choice in the healthcare market
1.5.1	The acceptable and preferred brand(s) of T. should be determined.	1.2.2	7
1.6. Technology (T. manufacturer) quality assessment before market entrance
1.6.1	In order to obtain permission to enter the country’s healthcare market, T. X does not require quality approval.	1.2.2	8
1.6.2	T. X requires quality approval to obtain permission to enter the country’s healthcare market.	1.2.2	8
1.7. Policies related to license revocation of T.
1.7.1	T. X imports should be permanently stopped.	–	7
1.7.2	T. X imports should be temporally discontinued until its safety is assured.	–	8
1.7.3	T. X imports should be temporally discontinued until its efficacy is assured.	–	8
1.7.4	The domestic production of T. X should be permanently stopped.	–	7
1.7.5	The domestic production of T. X should be temporally discontinued until the safety is assured.	–	8
1.7.6	The domestic production of T. X should be temporally discontinued until the efficacy is assured.	–	8
1.7.7	T. X should be removed from the healthcare market.	–	7
1.8. Technology quantification
1.8.1	T. utilization is subject to rationing policies (The percentage of optimal coverage and the methods for rationing of T. [related services] should be determined).	1.2.2	8
–	T. utilization is subject to disinvestment policy.	1.2.2	5
1.9. Organizing stratification and geographical distribution of T.
1.9.1	Spatial (geographical) policy should be adopted for T.	1.2.2	8
1.9.2	A localization policy should be adopted for T.	1.9.1	8
1.9.3	An active logistic (distribution) policy should be adopted for T.	1.9.1	8
1.9.4	T. related service(s) should be included within the basic benefit package of health services.	1.2.2	9
2. Policies regarding the prescription or provision of T. (-related services).
2.1. Determining the prescribers/providers of T. (-related services)
2.1.1	The authorized and preferred providers (specialty fields) as well as the authorized and preferred prescribers (specialty fields) of T. (-related services) should be determined.	1.2.2	9
–	The level of provision (primary/secondary/tertiary) should be determined for T. related service(s).	1.2.2	6
2.2. Determining the quantity cap for prescription/provision of T. related services.
2.2.1	A quantity cap (maximum amount of prescription/provision) should be determined for T. related services (The supervisory body for utilization rate of T. related services at the provider level should be determined subsequently).	1.2.2	7
2.3. Provision sector (public (governmental) vs. private)
2.3.1	The related services should only be provided by the government.	1.2.2	7
2.3.2	Part of the services should be provided by the government.	1.2.2	7
2.3.3	The provision of services should be assigned to the private sector.	1.2.2	7
3. Financing policies
3.1. The essential investment to set up T.
3.1.1	T. Installation should be supported by public subsidies. (The way (grant/loan/investment partnership) and proportion of the government subsidies should be determined subsequently).	1.2.2	8
3.1.2	Running costs of T. should be subsidized by government budget (the proportion of running costs covered by the government should be determined subsequently).	1.2.2	8
3.2. Purchasing T. -related services
3.2.1	Purchasing of T. -related services should be subsidized through government budget (the proportion of costs paid from the government budget should be determined).	1.2.2	8
3.2.2	The dominant payment method to T. providers should be fee-for-service (fee-per-item-of-service).	1.2.2	7
–	The dominant payment method to T. providers should be salary based.	1.2.2	5
3.2.3	The dominant payment method to T. providers should be capitation based.	1.2.2	8
3.2.4	T. (-related services) should be covered by the public insurance.	1.2.2	9
3.2.5	Public insurance coverage for T. (-related services) should not require user fees (copayments).	3.2.5	9
3.2.6	T. (-related services) costs should be directly paid by the consumers or through private insurance.	1.2.2	8
3.2.7	T. (-related services) costs are better to be covered by the charities or donors (philanthropists).	1.2.2	7
4. policies related to regulation, tariff-setting and pricing for T. (-related services)
4.1. Technology utilization monitoring and recording
4.1.1	T. (-related services) requires utilization recording mechanism at the provider level.	1.2.2	7
4.1.2	Utilization of T. (-related services) requires centralized monitoring system.	1.2.2	8
4.2. Technology-related clinical practice guideline
4.2.1	T. requires local practice guideline.	1.2.2	9
–	T. requires setting standards and specific clinical policies.	1.2.2	6
4.3. Tariffs and pricing
4.3.1	T. (-related services) tariffs should be set by the Ministry of Health.	1.2.2	8
4.3.2	T. (-related services) prices should be determined by the market equilibrium.	1.2.2	7
4.4. Marketing and advertisement
4.4.1	T. Advertising in specialized medical journals for professional audience is allowed.	1.2.2	7
4.4.2	T. advertisements through public press are allowed.	1.2.2	7
4.4.3	Use of T. should be increased through advertising in mass media.	1.2.2	8
4.4.4	T. advertising is not permitted.	–	7
5. Research policies related to T.
5.1. Research studies associated with T. X.
5.1.1	Efficacy study is required for T. X.	–	8
5.1.2	Effectiveness study is required for T. X.	–	8
5.1.3	Cost study is required for T. X.	–	8
5.1.4	Economic evaluation is required for T. X.	–	9
5.1.5	Social impact study is required for T. X.	–	7
5.1.6	Provisional demand analysis study is required for T. X.	–	8
5.1.7	Health impacts of T. X need to be studied.	–	8
5.1.8	Budgetary impacts of T. X need to be studied.	–	8
6. Education policies related to T.
6.1. Educational interventions related to T.
6.1.1	The disciplines responsible for T. education through their [approved] curriculum should be determined.	1.2.2	9
6.1.2	The discipline(s) responsible for T. training through their vocational pre-service and on-service training programs should be determined.	1.2.2	8
6.1.3	T. utilization should be promoted through public education and awareness.	1.2.2	8
6.1.4	Public education and awareness is required to prevent supplier induced demand for T. X [-related services].	1.2.2	8
6.1.5	Education of patients who are the potential users of T. -related services should be considered through the use of specific manuals/pamphlets.	1.2.2	8
7. Innovation policies related to T.
7.1. Innovation support and T. transfer
7.1.1	T. X needs the support of Research and Development (R&D) in the industrial sector.	–	8
7.1.2	T. X needs the support of Research and Development (R&D) in the academy.	–	8
7.1.3	Specific venture capital investment is required for T. transfer and localization.	–	7
7.1.4	Venture nurturing (non-financial support) is required for T. transfer and localization.	–	7
–	‘Technology-specific innovation system’ should be developed for T.	–	5

*

T.: ‘the technology’.

†

X: Title of the technology of interest.

Of the 37 initial decision criteria for all policies, 34 were considered necessary by the expert panel (median necessity rates of ≥7 in the second meeting) (Table 2). The criteria were arranged under three general headings, namely: general characteristics of the technology (4 criteria), the importance of the technology (10 criteria) and market status of the technology (20 criteria). The levels of each criterion were determined and the HTA portfolio was formed accordingly (the output of the second phase of the study).

Table 2.

Decision criteria for all technology-related policies, and their median necessity rates (MNR) given by the experts in the second meeting, as well as the levels of the criteria with MNR ≥ 7

Decision criteria for all technology-related policies																											MNR*
General characteristics of the technology (T.)
Type of T.	Pharmaceutical □ Surgical □ Diagnostic method □ Diet and Nutrition □ Instrument □ Implant □ Procedure □ Biotechnology □ Nanotechnology □ Information T. □ Service □ Program □ Behavior Treatment □ Complementary- medicine □ Service innovation □ others: …																										8
R&D Intensity of T.	Low □			Moderate □													High □										7
Life cycle phase of T.	Entry □	Development □				Domination □									Maturation □								Downturn □				7
Patent condition	Active □			Close to expiration □													Expired □										7
Importance of the technology
Frequency of Health Problems Addressed by T.			very high □				high □				moderate□					low □				very low □						unclear □	9
Severity of related Health Problems			very high □				high □				moderate□					low □				very low □						unclear □	9
T. efficacy(authenticated by research)			very high □				high □				moderate□					low □				very low □						unclear □	8
Strength of supporting evidence regarding T. efficacy			very high □				high □				moderate □					low □				very low □						unclear □	8
T. effectiveness (in reducing the burden of related problems in the society)			very high □				high □				moderate □					low □				very low □						unclear □	9
T. efficiency			very high □				high □				moderate □					low □				very low □						unclear □	9
T. safety			very high □				high □				moderate □					low □				very low □						unclear □	9
Public Interest Status of T.			Private service□							Service with externality□											Public service□						7
Strategic importance			In public health □		In economy & industry □								In science & technology □								No Strategic importance□						8
Exclusiveness of T.			There are suitable alternatives□								There is no suitable alternative □										There is no alternative □						8
Market status of the technology
T. entrance status in national market			Not entered □						Case basis □							Limited □						Widely □					7
Market source of T.			Importation □													Domestic production □											8
Economic efficiency of T.																											6
Quality variation of T. in the market			Good & similar qualities □								Quality varies partly □									Quality varies greatly □							8
Installation (investment) cost of T.			very high □				high □				moderate □					low □				very low □						unclear □	8
Running (maintenance) costs			very high □				high □				moderate □					low □				very low □						unclear □	8
Unit cost of using T.			very high □				high □				moderate □					low □				very low □						unclear □	8
Budgetary impact of T.			very high □				high □				moderate □					low □				very low □						unclear □	8
Organizational impact of T.																											5
Demand (willingness to pay) for T.			very high □				high □				moderate □					low □				very low □						unclear □	8
Supply (willingness to provide)			very high □				high □				moderate □					low □				very low □						unclear □	8
Price elasticity of demand			Elastic (-1 → -∞) □								Inelastic (0 → -1) □									Unclear □							8
Price elasticity of supply			Elastic (1 → +∞) □								Inelastic (0 → 1) □									Unclear □							7
Income elasticity of demand(IED)			Luxury good (IED>1)□								Normal good (0<IED<1)□									Inferior good (IED<0)□							9
Socioeconomic status of the T. beneficiaries			Often low □								Undifferentiated □									Often high □							9
Stakeholder groups pressuring for access to T.			Professional organizations □								Patients groups □							Political entities □						Media □			7
Geographical distribution of T. related health problems			Dispersed throughout the country (even) □											Aggregate or geographical clusters (uneven/clustered)□													7
Information asymmetry (between service provider and recipient about T.)			very high □					high □				moderate □							low □						very low □		7
Potential for off-label use of T.			very high □					high □				moderate □							low □						very low □		7
Potential for T. Misuse.			very high □					high □				moderate □							low □						very low □		7
Potential for social iatrogenesis																											5
Availability of trained manpower			Available in adequate numbers □									Available but not adequate□										Unavailable □					9
Availability of regulatory capacity			Available and sufficient □									Available but not sufficient □										Unavailable □					8

*

If a criterion has been rated for more than one policy, the highest MNR has been presented here.

Finally, the third phase of the study (DCE using stepwise regression model for data analysis) resulted in 64 stand-alone scoring tools shaping our DSS for all HTA-related policies. Each of these 64 tools is differentiated by a different set of criteria in connection with the corresponding policy decision and different weights of these criteria, shown by the allocated values (scores) of the different levels of each relevant criterion. For a sample policy (#1.2.2), the regression coefficients of the criteria and the corresponding values allocated to them in the final decision-making scoring tool has been presented in Table 3. The resulted decision support scoring tool for this particular policy is depicted in Figure 2. The other 63 scoring tools for the rest of the policies can be available on request.

Figure 2.

Sample of a decision aid-scoring tool from the final DSS. This sample acts specifically as the decision support tool for the policy ‘1.2.2’ (‘the technology X is permitted to enter the country’s healthcare market’). The highlighted figures point to the relative values (weights) of the relevant criteria levels for this particular policy (more details are provided in the text).

Table 3.

Sample of the results of the third step of the study: regression coefficients of the criteria within the initial list of the criteria (obtained in the second step) related to the policy 1-2-2, as well as the relative weights (values) allocated to the statistically significant criteria levels in the corresponding decision support scoring tool (see Figure 2)

Criteria	Levels	Regression coefficients	Allocated relative weights in the final scoring tool**
Severity of the related Health Problems	very high	1.28	–
	high	1.02	–
	Moderate	0.07	–
	low	−0.39	–
	very low	−1.03	–
	unclear	−0.88	–
Technology efficacy (authenticated by research)	very high	1.61*	30
	high	1.57*	30
	Moderate	0.22	–
	low	−0.59*	−10
	very low	−1.71*	−30
	unclear	−1.68*	−30
Strength of supporting evidence regarding the technology efficacy	very high	2.67*	50
	high	2.11*	40
	Moderate	0.08	–
	low	−1.65*	−30
	very low	−2.23*	−40
	unclear	−2.09*	−40
Technology safety	very high	1.67*	30
	high	1.33*	25
	Moderate	0.01	–
	low	−2.89*	−50
	very low	−3.44*	−60
	unclear	−2.87*	−50
Exclusiveness of technology	There are suitable alternatives	0.26	–
	There is no suitable alternative	0.47*	10
	There is no alternative	1.06*	20
Availability of trained human resource	Available in adequate numbers	0.11	–
	Available but not adequate	−0.65*	−10
	Unavailable	−1.81*	−30
Availability of regulatory capacity	Available in adequate numbers	0.21	–
	Available but not adequate	−0.63*	−10
	Unavailable	−1.73*	−30

*

Statistically significant (P value < 0.05).

**

The relative weight values were calculated using following formula: RW = α × 100/(∑α_max), where:.

RW: relative weight of any significant level, (the resulted values have been rounded up to the nearest multiples of 5; e.g. for the first significant criterion level: RW = 28.09 and the allocated value is 30).

α: regression coefficient of any statistically significant criterion level,.

α_max: maximum significant regression coefficient of any criterion.

How to use the DSS?

For a specific technology [-related service], the portfolio is filled separately by the technical expert panel (i.e. the members of the HTA scientific committee) based on the information available, preferably via an adapted HTA report. Afterwards, for any of the listed policies, a reference is made to the corresponding scoring tool from the DSS. For each relevant criterion, a mean score is calculated by averaging the values of its levels according to the portfolios judgments (see the example below). The final scores of all relevant criteria are then summed, resulting in that particular policy’s score that is attributable to a predetermined recommendation level.

Recommendation levels can be considered as follows: scores >75 for a policy indicate that that policy is strongly (highly) recommended with regard to the technology under assessment. Scores between 50 and 75 mean that the policy should be considered for that particular technology. Low and very low recommendation levels relate to the policy scores of 25 to 50, and <25, respectively.

Acting similarly for all 64 policies results in a universal package of policies in different areas with different recommendation levels for the technology under assessment (a simple software can be constructed to handle the process of listing the policies with their final scores according to the HTA portfolios ratings made by the experts). As an example, for the decision scoring tool, depicted in Figure 2 concerning the policy ‘1.2.2’ (with 6 relevant criteria), supposing that eight members of the expert committee separately fill the portfolio for a particular technology, of whom five evaluate the technology efficacy (the first relevant criterion for this policy) as ‘high’, two rate it as moderate and one rates low, we get a mean score of 17.5 for this criterion ((5*30) + (2*0) + (1*(-10)=140, divided by 8 (number of raters)). Acting similarly for the remaining criteria (supporting evidence, safety, exclusiveness, availability of trained human resource and availability of regulatory capacity), mean scores of, e.g. +35, +20, +15, -15, -12.5 are obtained, respectively. A total sum score of 60 is acquired for the policy 1.2.2, indicating that this policy (‘the technology is permitted to enter the country’s healthcare market’) ‘should be considered’ (though, Not ‘highly recommended’).

Discussion

The study has outlined potential policy decisions about a health technology, as well as a list of the criteria for making these decisions. More importantly, based on these materials, the study has introduced a methodology to develop a decision support system (DSS) aiming to link the information provided through an HTA report to those policies. We addressed the issue that different policies demand potentially different sets of criteria with different weights. As expected, many of the identified criteria were in common in different policies. Therefore, a specific criterion may be in common for a number of policies but with potentially different relative weights. Besides, in our reviews, we did not find any study aiming to identify the potential policies in different areas regarding a health technology.

About the results

A review of HPSR literature shows that the majority of the works have been descriptive and suffered from weak conceptualization and do not provide the audience with the process of how the policies have been developed (Gilson and Raphaely 2008).

As far as our reviews showed, most of the work done in the HTA field have focused on the appropriate design, harmonization and adaptation of HTA reports (Hailey 2003; Kristensen et al. 2009; Turner et al. 2009; Garrido et al. 2010; Nicod and Kanavos 2012; Kriza et al. 2014) or addressing the criteria [and their relative weights] used by decision-makers in different contexts, mainly about coverage/priority-setting/reimbursement decisions for new technologies (with little attention paid to the other broad aspects raised by a health technology) (Nobre et al. 1999; Oortwijn et al. 2002; Baltussen et al. 2010; Golan et al. 2011; Defechereux et al. 2012, Goetghebeur et al. 2012, Tromp and Baltussen 2012). The latter (by 2013) has been well addressed in separate reviews conducted by Guindo LA (Guindo et al. 2012), Fischer (Fischer 2012) and Cromwell I et al (Cromwell et al. 2015).

Though we did not find any study with similar approach in accessible literature, we should mention the remarkable work of Goetghebeur MM et al, (Goetghebeur et al. 2008). They organized the 15 identified components (criteria) of healthcare decision-making into a multi-criteria decision analysis value matrix. In addition, they developed a methodology to synthesize the evidence needed for each criterion. Afterwards, the instrument has been tested in several subsequent studies for validation and further development (Goetghebeur et al. 2010; Tony et al. 2011; Miot et al. 2012).

The DSS developed in this study also needs to be validated through testing in real world decision-making situations, before formal introduction and dissemination. Furthermore, as a new concept, the way DSS is introduced to the Iranian health system is of special importance. As pointed out in the methods section, in our core panel as well as expert meetings we had individuals from different levels of the HTA-related policy-making process -as illustrated in our conceptual framework in Figure 1; i.e. the faculty members in the knowledge management units in the main Iranian medical universities, members from the scientific committee of the HTA Office and relevant policy positions in the MOHME. We supposed that this panel composition with engagement of individuals from different relevant affiliations (authorities) would ease the process of introduction and dissemination of the DSS after development through e.g. planned communication.

As well, if the DSS is to be effectively used for policy making in Iran, it will have implications for all levels of technology-related policy-making including HTA reports format. In this regard, the national standard frameworks for preparation of HTA reports have already been introduced by the first author prior to the DSS presentation. In these frameworks, which will be discussed later in more details, each of the 34 criteria of the HTA portfolio in this study is addressed by a number of questions that should be answered by any HTA team.

A relevant concern may be about the seven contrasted macro-policy areas addressed in the DSS. It seems that this would complicate the integration of the DSS in decision-making, particularly if these areas are to be addressed and adopted by different authorities. As mentioned in the introduction section, the Iranian health system has a centralized nature with the responsibility of policy-making, planning and supervising health related activities in all areas -including education, research, innovation, financing, regulation and service provision etc.- and in both public and private sectors, integrated and assigned to the MOHME. So the main reference body for the DSS is the MOHME. This structural feature will potentially pave the way for the adoption of the DSS in all policy areas to a large extent. Obviously, in unintegrated systems, particular measures must be taken into consideration in order to proper and coordinated application of such multi-domain DSS by the involved institutions.

Moreover, we should have in mind the additional barriers and challenges of applying HTA in low and middle income countries in terms of required infrastructures, inadequate trained manpower, etc. Olyaeemanesh et al. (2014) have identified 22 challenges that may potentially hinder the development of the application of HTA reports in health policy-making in Iran. Though not all of these challenges will be dispelled by solely applying the DSS, some can be addressed at least to some extent by appropriate use of DSS. Among them are the ‘Lack of integration with the country's macroeconomic policy’, ‘Lack of a local model for the arrangement of related organizations’, ‘Insufficient application of a comprehensive framework in the HTA system’, ‘Lack of coordination with the HTA process at the policy-making level’ and ‘Being passive to new technologies’ (Olyaeemanesh et al. 2014). To address other challenges, appropriate actions should be undertaken. Some have been initiated. The introduction of the national standard frameworks for HTA reports is an eminent instance that in addition to harmonizing HTAs in different fields will help to address several of those challenges, e.g. common understanding among stakeholders and policy makers and planning for academic training of experts. The frameworks have been developed in three versions (full, rapid and snapshot for different decision-making situations) separately for diagnostic and therapeutic technologies. They are available through the webpage of the Iranian HTA office (in Persian) (MOHME-2016). Organizing knowledge management units (KMUs) in medical universities is another action that has been conceptualized and initiated (Manavi et al. 2013) and can potentially provide excellent opportunities for allocation of specialized units to HTA activities as well as training local experts with high capacity for HTA-related tasks, e.g. analysis and synthesis of the wide range of local and global evidence, as the core components of the HTA studies.

Finally, it is worth mentioning that dynamicity and openness to change are necessary components of such works. For example, changing circumstances may demand for a new policy decision entailing a new criterion that has not been addressed within the current portfolio. Also, the DSS, as the final product of the project, is to act as an aid and is not to replace the policy-making body. Therefore, it is recommended that, in each case, the comments of the relevant policy council should be taken into consideration. According to our recommendation levels, we suggest that policies with scores >75 should be offered only for final approval by such council; policies with scores <25 do not need to be offered; policies with scores between 50 and 75 should be considered for further discussion; and policies with scores between 25 and 50 may be reviewed before exclusion.

About the methodology

Health policy and systems research (HPSR) is often accused of being ambiguous or unclear in scope and nature, lacking robust methodology and having a limited possibility for generalization of its findings so that policy makers in settings other than a HPSR original context may not put high value on the findings and, consequently, not apply them in their own circumstances (Gilson 2012; Mills 2012). HPSR is basically a problem-based study area and therefore, it can employ a wide range of very often mixed research designs within a multidisciplinary approach to address the comprehensiveness and complexity of the issues of interest (Gilson 2012).

The method strategy used for this study has been specifically designed to best serve the study goal. Accordingly, we found the ‘discrete choice experiment’ (DCE) an appropriate method for the purpose of a multi criteria decision analysis approach in the third step. DCE has primarily been emerged as a quantitative method in business in order to elicit the potential consumers’ preferences among virtual alternatives of a product (goods or services) with different combinations of utilities prior to assembling/rendering. Basically, DCE seeks for the relative weights of different attributes (decision criteria in our work) contributing in individuals’ preferences particularly in the absence of real data (virtual portfolios in our work) (Mangham et al. 2009). Its use in health areas, particularly in health economics and, more recently, for policy and planning purposes, has been increased in recent years. The latter has especially been the case with deriving the criteria (and their relative weights) used by different stakeholders for making, e.g. coverage or priority-setting decisions in different contexts (Koopmanschap et al. 2010; Golan et al. 2011; Erdem and Thompson 2014).

Despite its advantages, DCE has some limitations, especially when there are too many attributes. In our work, to avoid needing larger sample sizes, only the criteria identified in step 2 as being potentially relevant were included in the analytical model for any particular policy. Nevertheless, it seems that for policy decisions with >5-6 relevant criteria, our sample size of 210 data sets is not ideally sufficient for the purpose of data analysis using stepwise regression. Actually, the hard work of discussing 30 virtual portfolios by seven members of the core panel and subsequent rating of the appropriateness of each of 64 policy statements for each portfolio within the study timeframe impeded us from having a larger sample size that the maximum number of criteria for a policy in step 2 required.

Conclusion

The main characteristic of this study may be its attempt to have a comprehensive as well as pragmatic approach to the crucial issue of facing/encountering a health technology within a health system through an evidence-informed policy-making process. As the global debates have strong influences over national issues, the local experiences and solutions adopted for particular problems can have lessons/implications for the international community. Though the primary focus of the study has been the Iranian health system, we think that both findings/products of this study (the Decision Support System for 64 health technology-related policies) and the methodology used for this purpose may serve as a guide for policy makers and policy researchers in other settings, particularly low and middle income countries, in developing decision aid tools for their own context-specific HTA-related policies, based on likely different sets of criteria with different weights that better reflect the realities of their particular circumstances and conditions. Further investigation is highly recommended in order to more clarify the probable aspects of such complex issue that might not have been well addressed in our study (particularly with both the sets of policy statements and corresponding criteria) due to rather limited body of evidence regarding the issue (compared to other health research areas) and the broad perspectives and changing expectations and preferences of different stakeholders.

Funding

This research was funded through the grant no#1393-1-172-1290 from the Iranian ‘Ministry of Health and Medical Education’ (MOHME) at the ‘Shahid-Beheshti university of Medical Sciences’. The views and opinions stated in the article are those of the authors and do not necessarily reflect the views of Iranian MOHME or Government.

Acknowledgements

The authors wish to thank Dr Alireza Olyaeemanesh (General Director of Health Technology Assessment, Standardization and Tariff Department, Deputy of curative affairs, MOHME, Tehran, Iran), Shila Doaee (HTA Office, MOHME, Tehran, Iran) and the other members of our expert panel (Dr. Hadi Beihaghi, Dr. Ali Ardem, Mohammadreza Mobinizadeh, Mina Nedjati, Parisa Aboee) and discussion group panels for their valuable contributions.

Conflict of interest statement. None declared.