Abstract

This paper reports a qualitative investigation of factors contributing to success in 10 collaborative international forestry research projects funded by the Australian Centre for International Agricultural Research (ACIAR) in Vietnam. Success factors were identified, and the relative success of projects was evaluated in terms of research achievements and impacts, through analysis of ACIAR's project records and interviews with key project participants. This process identified 22 factors considered to either enhance or diminish project success, with the most frequently identified being: collaborative scoping and design; skills mix and time allocations; funding and equipment; scientists’ commitment and collaboration; and capacity building. Three projects, representing different categories of assessed research achievement and impact, were examined for evidence of relationships between these success factors and the relative success of the projects. This assessment suggested that most of the identified success factors were evident in the project with high research achievements and high impacts; and, conversely, that there was evidence of factors that diminish project success in a project that had low achievements and low impacts. The results reported here can help improve the design and implementation of future collaborative forestry research projects.

Introduction

International collaborative research in agricultural and natural resource management is often funded through Official Development Assistance (ODA) programs, and evaluations have shown such investments can generate significant benefits to farmers and rural communities (Raitzer, 2003; Lindner et al., 2013). The conduct of international agricultural research is a complex activity, producing a wide variety of outputs, which are influenced by factors such as the capacity of the collaborating partners and the stage of activities in the research-for-development continuum (Bantilan et al., 2004). In addition, the pathways from research to impact in agriculture, forestry, fisheries and natural resources research are complex and non-linear (Millstone et al., 2010; Mayne and Stern, 2013; Joly et al., 2015), and definitions of ‘success’ can be contested and controversial (McLeod et al., 2012). ODA interventions interact with other factors and rarely lead to development outcomes on their own; consequently, there are various challenges in establishing relationships between an intervention and its impact (Stern et al., 2012). Similarly, even well-designed evaluations of research investments may not lead to organizational learning for research project leaders, team members or funders (Forss et al., 1994; Horton and Mackay, 2003). For example, findings from economic impact assessments may not identify why changes occurred, or how to improve future research programs (Horton and Mackay, 2003).

In this context, this article seeks to identify factors that affect the success of international collaborative forestry research projects, and explore whether there is an apparent relationship between these factors and the evaluated level of success of a project. We investigated these questions through a comparative qualitative analysis of 10 collaborative forestry research projects between Australia and Vietnam. We evaluated the relative success of each project from project records, using a previously developed methodology (Bartlett, 2016a), and surveyed the views of key project participants, and then sought evidence of how the factors were manifested in projects with different levels of success. We distil lessons that are able to be influenced, enhanced or facilitated by those who design and fund ODA-research projects and those with responsibility for implementing these projects. Our approach was informed by that of McLeod et al. (2012), who advocated a qualitative approach focused on ‘understanding how the various project stakeholders subjectively perceived project outcomes and the evaluation criteria they drew on in doing so’.

There is limited published literature that documents the generic factors that affect the success of ODA-funded forestry research projects. As Blamey and Mackenzie (2007) have noted, context can be the key to uncovering the circumstances in which, and the reasons why, a particular intervention works. Because each project inevitably faces its own unique set of opportunities and constraints, it is often difficult to define which factors are unique and context dependent, and which are more widely applicable. There are many external factors that can play a role in determining the ultimate impact (or lack of impact) for any given project. Some examples from the literature include the availability of the technologies, such as improved germplasm (Franzel et al., 2004); dissemination of knowledge in a form appropriate to the users (Thangata and Alavalapati, 2003); their capacity to take risks (Mercer, 2004); market incentives (Pattanayak et al., 2003); security of land tenure (Suyanto et al., 2005) and their access to ancillary resources such as skills and finance (Farrington et al., 1997). Forestry research typically involves complex systems involving biophysical and social elements and which, compared with agricultural systems, require much longer time frames to produce the desired products (Henderson, 2000). For forestry research projects undertaken in developing countries, achieving positive impacts is likely to depend on multiple factors, which can be interdependent (Byron, 2001).

The Australian Centre for International Agricultural Research

The Australian Centre for International Agricultural Research (ACIAR) is a federally funded agency that commissions collaborative agriculture, fisheries and forestry research projects in developing countries. ACIAR projects seek to generate knowledge, technologies and capacity to achieve better decision-making, changed agricultural practices and policies that, in turn, generate positive scientific, economic, social or environmental impacts (ACIAR, 2014). In ACIAR terminology, projects generate outputs which, if adopted, lead to outcomes and impacts. Outputs are defined as the products of the research, including technologies, knowledge, capacity and policy inputs, that can be adopted or used by the ‘next users’ as inputs for further research; outcomes are changes in practice, products or policies consequent on the adoption of outputs and impacts are changes in markets, the state of common resources and to individuals or communities that can be attributed to the adoption of the research outputs by the ‘end users’ of the research (Davis et al., 2008).

In accordance with its governing legislation (Commonwealth of Australia, 1982), ACIAR funds research projects conducted by Australian or international scientists with scientists in partner countries, with capacity building of research partners supported in parallel with research activities. Over a 30-year period, ACIAR has invested over AUD 100 million to fund 150 forestry projects and activities in 29 countries; most projects have been implemented in Indonesia, Vietnam and Papua New Guinea (Bartlett, 2016b). ACIAR has a commitment to evaluating the effectiveness and benefits of its projects (ACIAR, 2014), with all large projects having externally conducted end-of-project reviews, some projects having adoption studies conducted by former project leaders and ~10 per cent of projects subject to externally conducted impact assessments. However, it does not have a standard approach for comparing project achievements or for identifying the factors that contribute to the relative success of projects (Bartlett, 2016a).

Defining project success

In this paper, success is defined following the interpretation used in other ACIAR studies as having two primary dimensions: the first is the extent to which planned research outputs are achieved and adopted by ‘next users’, such as the participating scientists, farmers, processors and policy makers, termed achievements; the second is the extent of the impacts resulting from wider adoption of the research outputs by ‘end users’, typically stakeholders outside the project and often beyond its life, termed impacts (Pearce, 2010). In both dimensions, this study focuses on those factors that could be influenced by those responsible for research design, implementation and support, rather than external factors that are beyond the reach of the project leaders or managers to influence. Carden (2004) presents a complementary approach that focuses on factors beyond the reach of a research project, such as its influence on policy formulation.

Factors believed to influence a research project's success

There are few studies that report project-level factors contributing to success of agricultural research projects. An ACIAR impact assessment study (Pearce, 2010) surveyed 30 people, who were Australian project leaders or ACIAR-employed research program managers and country managers and identified 14 factors that contributed to successful project outcomes, with the following six factors most often identified by respondents:

  • Clearly defined objectives and research questions based on a clear stakeholder needs and with a project plan that assigns clear responsibilities to participants.

  • Strong communication leading to good collaboration, including formal and informal communication arrangements and compatible language skills.

  • Trust, complementarity and alignment of interests, including effective interpersonal relationships and mutual empathy and respect.

  • Good project leadership and management support, including the capacity to empower the research team, co-ordinate diverse groups and engender institutional support.

  • Strong and capable research team, including having the right technical abilities and the time commitment to undertake the required research; and

  • Institutional support both for the Australian and in-country partner.

This list provides a useful benchmark for this research, which seeks to confirm their applicability for forestry research projects from Vietnam and explore whether or not scientists from the partner country have the same view as Australians on the relevance of these factors.

Forestry development and ACIAR's forestry research investments in Vietnam

Vietnam is a country of almost 90 million people in South-East Asia. Over the 60 years up to 1995, forest extent declined to ~9.8 million hectares or 29.6 per cent of Vietnam's land area (Government of Vietnam, 2007), but has since increased to 14.7 million hectares or 44.4 per cent of land area (FAO, 2015). Planted forests have played a very significant role in achieving this restoration of forest cover, with a total of 3.66 million hectares or 25 per cent of Vietnam's forest area being classified as planted (FAO, 2015). Since 1988, the Government of Vietnam has allocated forest land to communities on renewable 50 year leases and much of this has been planted with fast-growing short rotation species such as Eucalyptus and Acacia (Amat et al., 2010). An estimated 250 000 smallholder farmers are growing acacia plantations on rotations of 5–10 years (Nambiar et al., 2014), primarily for the production of pulpwood.

Following the Doi Moi economic reform policies of the mid-1980s, the Government of Vietnam introduced a range of measures, including land tenure reforms and forestry policies, such as the 1998 Five Million Hectare Reforestation Program, to encourage smallholder farmers to plant commercial trees. The Vietnam Forestry Development Strategy 2006–2020 aspires to 16.24 million hectares of forest by 2020, including 4.15 million hectares of plantations, and recognizes the contribution that science and technology transfer has made to the quality and efficiency of its afforestation programs (Government of Vietnam, 2007). Both the achievements and concerns about aspects of Vietnam's reforestation program have been discussed in the literature. For example, increasingly substantial economic benefits for smallholders and regional economies are being generated from acacia plantations (Byron, 2014): but these gains followed an initial phase of poor growth associated with use of inferior germplasm or incorrect species-site matching (Nguyen and Gilmour, 1999); and future growth of this sector depends on avoiding environmental degradation (Amat et al., 2010), and improving and sustaining productivity from these plantings (Nambiar et al., 2014). Concerns have been expressed about loss of higher quality agricultural land (de Jong et al., 2006), disruption of existing land use systems (Clement and Amezaga, 2009), and loss of access for collection of non-timber forest products, and inequitable allocation to poor households (McElwee, 2009).

Vietnam has a large and expanding timber processing industry, with the annual value of export timber products growing at a rate of 40 per cent between 2000 and 2010 (Phuc and Canby, 2011); by 2005, wood products had become the nation's fifth largest export commodity. Vietnam is now one of the world's largest exporters of secondary wood products, principally furniture, with wood products’ export earnings reaching $3.4 billion in 2010 (Phuc and Canby, 2011). However, there may be impediments that prevent smallholders from fully capitalizing on the markets associated with domestic wood processing industries (Putzel et al., 2012).

ACIAR's forestry research investments in Vietnam began in 1993 and, until 2011, all projects were undertaken only with the Forest Science Institute of Vietnam, the predecessor of the Vietnam Academy of Forest Sciences. From 1992 to December 2014, ACIAR completed 20 forestry research projects in Vietnam; the majority of these operated in multiple countries, with the activities in Vietnam being part of a larger research project. The projects cover 5 of the 10 research themes from the ACIAR forestry program (Bartlett, 2016b):

  • Theme 1: Domestication and improvement of Australian trees.

  • Theme 2: Silviculture for Australian trees.

  • Theme 3: Domestication and silviculture of non-Australian trees.

  • Theme 4: Forest health and biosecurity.

  • Theme 5: Value added processing and treatment of wood.

The domestication and improvement of Australian tree species, which could be grown on short rotations, contributed greatly to the expansion of the planted forests in Vietnam. Various species of Eucalyptus, Melaleuca and Acacia were first introduced to Vietnam in the 1950s and 1960s. ACIAR's projects on the domestication and management of Eucalyptus and Acacia have facilitated significant improvement in the productivity of these Australian trees in Vietnam (Fisher and Gordon, 2007), with 50–100 per cent gains in wood production demonstrated in trials (Harwood et al., 2015). By 2013, the estimated area of Acacia plantations was 1.1 million hectares and there was a further 200 000 hectares of Eucalyptus plantations (Harwood and Nambiar, 2014).

Methods

The methodology for this case study involved a preparatory phase to identify suitable research projects for the study followed by three phases of research: identification of success factors; evaluation of relative success of projects and identification of relationships between the success factors and the relative success of different projects. This process is illustrated in Figure 1.

Figure 1

Research methods flow diagram.

Figure 1

Research methods flow diagram.

Phase 0: Identification of projects for the case study

In the preparatory phase, 10 of the 20 projects ACIAR implemented in Vietnam between 1994 and 2012 (Table 1) were selected for the case study, taking into account the following factors:

  • Focusing on medium to large research projects, rather than small research activities.

  • Ensuring representation of projects from each research theme.

  • Inclusion of projects across the 20-year period, including some projects that were part of a linked program over at least 10 years.

  • Inclusion of some projects conducted entirely in Vietnam and some that were regional projects, with smaller components conducted in Vietnam.

  • Having adequate project records available, including project document, annual final report and external end-of-project review report.

Table 1

Summary information for ACIAR's completed Vietnam forestry projects

ACIAR project code Focus of research Duration Funding (AUD M) Countries Theme 
FST/1992/027 Domestication of Australian Acacias 1994–1998 1.040 China, Vietnam T1 
FST/1993/010* Physiology and genetics of Acacia auriculiformis 1994–1998 0.785 Thailand, Vietnam T1 
FST/1993/112 Double diffusion treatment of eucalypt poles 1994–1995 0.061 Vietnam T5 
FST/1993/118* Seeds of Australian trees 1993–1999 3.844 Vietnam +6 others T1 
FST/1994/019 Genetic diversity and propagation of mangroves 1999–2005 0.867 Thailand, Vietnam T3 
FST/1994/033 Leucaenas for Asian, Pacific and Australian agriculture 1995–2000 1.279 Vietnam +2 others T3 
FST/1994/041 Minimizing disease impacts on eucalypts in south-east Asia 1996–2000 0.788 Thailand, Vietnam T4 
FST/1995/124 Insect threats to Acacia and Eucalyptus plantations in Asia 1997–1998 0.138 Vietnam +3 others T4 
FST/1996/005 Domestication strategies for important species of Meliaceae 1999–2003 0.629 Vietnam +3 others T3 
FST/1997/024* Resistance and control of Hypsipyla shoot borer 1999–2003 1.145 Vietnam +4 others T4 
FST/1998/085 The taxonomy of Hypsipyla robusta and allied species 1999–2001 0.153 Vietnam + 9 others T4 
FST/1998/096* Domestication of Australian trees 2000–2004 2.209 Vietnam +7 others T1 
FST/1999/095* Improving eucalypt sawn wood: genetics and silviculture 2005–2009 0.683 China, Vietnam T2 
FST/2000/003* Mixed species plantations of high value trees 2002–2006 0.940 Vietnam T3 
FST/2001/021* Improving eucalypt sawn wood: sawing and drying 2005–2009 0.520 China, Vietnam T5 
FST/2002/112* Domestication of Meliaceae and management of Hypsipyla 2005–2009 0.386 Vietnam +2 others T4 
FST/2003/002* Development of triploids and polyploid breeding for Acacias 2004–2009 0.506 Vietnam, Sth Africa T1 
FST/2006/087* Sawlog silviculture for Acacias 2008–2012 0.928 Vietnam T2 
FST/2005/047 Eucalyptus biosecurity workshop 2004 0.043 Vietnam T4 
FST/2007/025 Socio-economic study of Acacia 2004–2009 0.029 Vietnam T5 
ACIAR project code Focus of research Duration Funding (AUD M) Countries Theme 
FST/1992/027 Domestication of Australian Acacias 1994–1998 1.040 China, Vietnam T1 
FST/1993/010* Physiology and genetics of Acacia auriculiformis 1994–1998 0.785 Thailand, Vietnam T1 
FST/1993/112 Double diffusion treatment of eucalypt poles 1994–1995 0.061 Vietnam T5 
FST/1993/118* Seeds of Australian trees 1993–1999 3.844 Vietnam +6 others T1 
FST/1994/019 Genetic diversity and propagation of mangroves 1999–2005 0.867 Thailand, Vietnam T3 
FST/1994/033 Leucaenas for Asian, Pacific and Australian agriculture 1995–2000 1.279 Vietnam +2 others T3 
FST/1994/041 Minimizing disease impacts on eucalypts in south-east Asia 1996–2000 0.788 Thailand, Vietnam T4 
FST/1995/124 Insect threats to Acacia and Eucalyptus plantations in Asia 1997–1998 0.138 Vietnam +3 others T4 
FST/1996/005 Domestication strategies for important species of Meliaceae 1999–2003 0.629 Vietnam +3 others T3 
FST/1997/024* Resistance and control of Hypsipyla shoot borer 1999–2003 1.145 Vietnam +4 others T4 
FST/1998/085 The taxonomy of Hypsipyla robusta and allied species 1999–2001 0.153 Vietnam + 9 others T4 
FST/1998/096* Domestication of Australian trees 2000–2004 2.209 Vietnam +7 others T1 
FST/1999/095* Improving eucalypt sawn wood: genetics and silviculture 2005–2009 0.683 China, Vietnam T2 
FST/2000/003* Mixed species plantations of high value trees 2002–2006 0.940 Vietnam T3 
FST/2001/021* Improving eucalypt sawn wood: sawing and drying 2005–2009 0.520 China, Vietnam T5 
FST/2002/112* Domestication of Meliaceae and management of Hypsipyla 2005–2009 0.386 Vietnam +2 others T4 
FST/2003/002* Development of triploids and polyploid breeding for Acacias 2004–2009 0.506 Vietnam, Sth Africa T1 
FST/2006/087* Sawlog silviculture for Acacias 2008–2012 0.928 Vietnam T2 
FST/2005/047 Eucalyptus biosecurity workshop 2004 0.043 Vietnam T4 
FST/2007/025 Socio-economic study of Acacia 2004–2009 0.029 Vietnam T5 

* Projects analysed in the case study.

Phase 1: Identification of project success factors

We used qualitative data, derived from interviews with former research project participants, to identify the factors considered to be most influential in achieving or hindering project success. For each project, the Australian project leaders, Vietnamese project coordinators and other scientists who had been involved in each project were interviewed. A total of 24 scientists, comprising 11 from Australia and 13 from Vietnam, were identified from project records and interviewed individually by the primary author using a standard set of questions (available as Supplementary online material). Interviewees were asked to explain what they thought constituted success for an ACIAR project, and then to nominate five factors that can enhance project success, and five factors that can diminish project success. Other questions sought their views about aspects of the project's design, implementation and other contextual factors. The research protocol was approved by the Australian National University Human Ethics Committee (protocol no. 2014/051).

HyperRESEARCH (Researchware, Inc. – http://www.researchware.com/ accessed 13 June 2014) qualitative data analysis software was used to analyse interview data to establish perspectives on the definition of project success and to facilitate aggregation of thematic aspects of the responses into two lists of factors that contribute to either enhancing or diminishing project success. Individuals’ responses to questions about each project's design and implementation were analysed as well as their responses on factors affecting project success. When respondents covered aspects of multiple factors in a single response, each aspect was identified, allocated to the most relevant factor and counted. When the respondents identified aspects related to the same factor in two or more responses, the aspect was counted only once, against the most relevant factor.

The primary author compared the two lists to identify complementary expressions of the same factor, and prepared concisely worded statements of the factors that can enhance or diminish the success of research projects. The data were further analysed to identify the frequency of identification of each success factor, to give an indication of which success factors are considered most important, and whether there were any notable differences in the factors identified by Vietnamese or Australian respondents.

Phase 2: Evaluation of relative success of the case study projects

We used qualitative data drawn from internal ACIAR project records to evaluate the relative success (the evaluation questions and guidance on evidence sought are available as Supplementary online material) of each of the 10 projects. The records included project documents; annual reports; mid-term reviews; final reports; external end-of-project reviews; adoption studies and external impact assessments; project-related publications and written correspondence between ACIAR and project staff. These data provided perspectives from project participants, research program managers and external reviewers of projects.

To evaluate relative success, the author used a score-card matrix methodology (Bartlett, 2016a) for each project, and assigned scores for four criteria related to research achievements: project design; results achieved; collaboration and publications; and four criteria related to research impacts: capacity building outcomes; scientific outcomes; economic outcomes; and social and policy outcomes. Under this methodology, scores totalling 10 were assigned for each of research achievements and research impacts, with both research achievements and scientific outcomes criteria assigned scores of up to 4 and all other criteria assigned scores of up to 2. The resulting scores for each of research achievements and research impacts were summed and then graphed. Scores of 0.0–5.0 were considered to be low achievements or low impacts; scores of 5.1–10.0 were considered to be high achievements or high impacts. This approach facilitated the identification of projects that represent one of four project success categories based on the assessed levels of research achievements and impacts: high achievements–high impacts; high achievements–low impacts; low achievements–low impacts and low achievements–high impacts.

Phase 3: Identification of relationships between success factors and the level of relative success achieved by different projects

To explore possible relationships between the identified success factors and the evaluated relative success of a project, three projects, representing different project success categories, were selected for a more detailed analysis. The nature of the selected projects is shown in Table 2; with further information on the type of research conducted in each project and the way in which various success factors influenced its level of success provided in Appendix 1.

Table 2

Details of projects analysed to explore relationships between success factors and project success categories

Project success category Project number Theme Title of project 
High achievements-High impacts FST/1998/096 T1 Domestication of Australian trees for reforestation and agroforestry 
High achievements-Low impacts FST/2006/087 T2 Optimizing silvicultural management and productivity of Acacia plantations for sawlogs 
Low achievements-Low impacts FST/2001/021 T5 Improving the value chain for eucalypt sawn wood: sawing and drying 
Project success category Project number Theme Title of project 
High achievements-High impacts FST/1998/096 T1 Domestication of Australian trees for reforestation and agroforestry 
High achievements-Low impacts FST/2006/087 T2 Optimizing silvicultural management and productivity of Acacia plantations for sawlogs 
Low achievements-Low impacts FST/2001/021 T5 Improving the value chain for eucalypt sawn wood: sawing and drying 

As this task was exploratory in nature, two methods were used. Firstly, interview responses (IR) from the Australian and Vietnamese respondents who had held leadership positions in the selected projects were further analysed using HyperRESEARCH to identify any references to the way each of the success factors identified through the Phase 1 methods had enhanced or diminished success. Secondly, relevant project records (PR) for the three projects were reviewed by the primary author to identify any evidence about the way the various success factors may have influenced the project's success. Using these two sources of information, subjective ratings were assigned by the primary author for the apparent influence of each of these success factors on the project's success. The following five category rating system was used:

  • Strongly enhances – presence of factor appears to have strongly enhanced success.

  • Enhances – presence of factor appears to have enhanced success.

  • Neutral – no evidence that the factor enhanced or diminished success.

  • Diminishes – absence of factor appears to have diminished success.

  • Strongly diminishes – absence of factor appears to have strongly diminished success.

Results

Interpreting success and identifying success factors

Views from project participants on what constitutes project success varied considerably, with some finding it difficult to articulate what success meant to them. The HyperRESEARCH analysis enabled the sentiments from the participants’ responses to be combined into a definition of success. A successful ACIAR forestry research project can be considered to be one which, in the context of the time and resources available, involves good scientific methods, achieves what it set out to do, enhances capacity, facilitates ongoing scientific relationships and generates knowledge or technologies that can improve the system under investigation and result in benefits for the next or end users.

The HyperRESEARCH analysis of participants’ responses on the factors that can enhance or diminish project success identified 20 factors that they considered to enhance project success and 19 factors they considered to diminish project success (Table 3). When considered as a whole, there were 22 different factors identified that influence project success (Table 3), with most responses on factors which diminish success being the converse of those nominated for enhancing success. However, among the responses, there were three factors identified that diminish success, and two factors that enhance success, for which there was no converse factor nominated.

Table 3

Success factors, ordered by their frequency of identification, showing participants’ views on aspects that enhance or diminish project success

Factor No. Success factors Participants’ views on factors that can enhance success (ES) or diminish success (DS) 
Collaborative scoping and design ES: Collaboration and quality of project scoping and design 
DS: Topic poorly understood, poor design, too complicated or ambitious, inflexible 
Skills mix and time allocations ES: Appropriate researcher skills mix and time allocations 
DS: Researcher skills or time allocation 
Funding, facilities and equipment ES: Adequate funding and other resources, including donor and partner contributions 
DS: Inadequate funds or restrictions on funding flow or provision of research facilities 
Scientists commitment, collaboration and focus ES: Good commitment, implementation focus and collaboration of partner scientists 
DS: Scientists not committed or collaborating well, inflexible or not completing tasks 
Team and technical capacity building ES: Supporting capacity building, informal and formal study 
DS: Poor focus on capacity or team building 
Mutual benefit of research topic ES: Selection of research issue with mutual benefits 
DS: Research not linked to both partners’ interests 
Selection and commitment of partner institutions ES: Appropriate selection and commitment of partner institutions 
DS: Inappropriate selection and lack of commitment from partner institution 
Site selection and scientific rigour of trials ES: Selection of sites and scientific rigour of trials 
DS: Inappropriate locations for research trials or lacking secure tenure 
Leadership and management ES: Effective project leadership and management 
DS: Poor leadership or management of project 
10 Strong, culturally appropriate team relationships ES: Strong, respectful and culturally appropriate team relationships 
DS: Collaborating scientists not understanding or respecting local culture and partners 
11 Time spent on in-country collaboration ES: Sufficient time spent in country 
DS: Inadequate time in country 
12 Effective communications and research networks ES: Within-project communications, use of knowledge, researcher networks 
DS: Poor communications or misunderstandings due to language barriers 
13 Links to impact pathway and user benefits ES: Impact pathway links during and after project generating benefits for end users 
DS: Research not connected to impact pathway or outputs is inappropriate for end users 
14 Implementation flexibility, monitoring and review ES: Implementation flexibility with systems for managing, monitoring and adapting activities 
DS: No flexibility to adapt or no review during implementation 
15 Continuity of partner institutions and team ES: Continuity of partner institutions and team 
DS: Collaborators not motivated or leaving project 
16 Duration of project ES: Not identified 
DS: Project duration too short 
17 Donor influence on design ES: Donor input to quality of project design 
DS: Donor imposition on project design or inadequate discussion with country partner 
18 Long-term research collaborations ES: Long-term relationships via follow on projects 
DS: Not identified 
19 Continuation of research post project ES: Scientists actively continuing research theme after project using enhanced capacity 
DS: Scientists not do not utilize new capacity or continue research post project 
20 Alignment with national development objectives ES: Alignment with government objectives 
DS: Not contributing to government objectives 
21 Experience of project leader in country ES: Not identified 
DS: Project leader lacking country experience or cultural sensitivity 
22 Trust within team ES: Trust within team 
DS: Not identified 
Factor No. Success factors Participants’ views on factors that can enhance success (ES) or diminish success (DS) 
Collaborative scoping and design ES: Collaboration and quality of project scoping and design 
DS: Topic poorly understood, poor design, too complicated or ambitious, inflexible 
Skills mix and time allocations ES: Appropriate researcher skills mix and time allocations 
DS: Researcher skills or time allocation 
Funding, facilities and equipment ES: Adequate funding and other resources, including donor and partner contributions 
DS: Inadequate funds or restrictions on funding flow or provision of research facilities 
Scientists commitment, collaboration and focus ES: Good commitment, implementation focus and collaboration of partner scientists 
DS: Scientists not committed or collaborating well, inflexible or not completing tasks 
Team and technical capacity building ES: Supporting capacity building, informal and formal study 
DS: Poor focus on capacity or team building 
Mutual benefit of research topic ES: Selection of research issue with mutual benefits 
DS: Research not linked to both partners’ interests 
Selection and commitment of partner institutions ES: Appropriate selection and commitment of partner institutions 
DS: Inappropriate selection and lack of commitment from partner institution 
Site selection and scientific rigour of trials ES: Selection of sites and scientific rigour of trials 
DS: Inappropriate locations for research trials or lacking secure tenure 
Leadership and management ES: Effective project leadership and management 
DS: Poor leadership or management of project 
10 Strong, culturally appropriate team relationships ES: Strong, respectful and culturally appropriate team relationships 
DS: Collaborating scientists not understanding or respecting local culture and partners 
11 Time spent on in-country collaboration ES: Sufficient time spent in country 
DS: Inadequate time in country 
12 Effective communications and research networks ES: Within-project communications, use of knowledge, researcher networks 
DS: Poor communications or misunderstandings due to language barriers 
13 Links to impact pathway and user benefits ES: Impact pathway links during and after project generating benefits for end users 
DS: Research not connected to impact pathway or outputs is inappropriate for end users 
14 Implementation flexibility, monitoring and review ES: Implementation flexibility with systems for managing, monitoring and adapting activities 
DS: No flexibility to adapt or no review during implementation 
15 Continuity of partner institutions and team ES: Continuity of partner institutions and team 
DS: Collaborators not motivated or leaving project 
16 Duration of project ES: Not identified 
DS: Project duration too short 
17 Donor influence on design ES: Donor input to quality of project design 
DS: Donor imposition on project design or inadequate discussion with country partner 
18 Long-term research collaborations ES: Long-term relationships via follow on projects 
DS: Not identified 
19 Continuation of research post project ES: Scientists actively continuing research theme after project using enhanced capacity 
DS: Scientists not do not utilize new capacity or continue research post project 
20 Alignment with national development objectives ES: Alignment with government objectives 
DS: Not contributing to government objectives 
21 Experience of project leader in country ES: Not identified 
DS: Project leader lacking country experience or cultural sensitivity 
22 Trust within team ES: Trust within team 
DS: Not identified 

The interview data comprised 299 participant responses related to individual success factors. The frequency of identification of the 22 success factors by the 11 Australian and 13 Vietnamese respondents, for responses related to both enhancing and diminishing project success, is shown in Figure 2. The two most frequently identified factors, which together represented 20 per cent of the responses, were collaborative scoping and design; and skills mix and time allocations. Twelve of the success factors (Nos. 1–12 from Table 3) together represented 80 per cent of the responses, and so were considered as the most important factors affecting project success in this study.

Figure 2

Frequency of identification of the 22 project success factors by Australian and Vietnamese respondents.

Figure 2

Frequency of identification of the 22 project success factors by Australian and Vietnamese respondents.

Most of the success factors were identified consistently by Australian and Vietnamese respondents, but there were some differences apparent. Vietnamese respondents more frequently identified success factors such as skills mix and time allocation; mutual benefit of research topic; strong, culturally appropriate relationships; leadership and management; and duration of project. Australian respondents more frequently identified success factors such as: time spent on in-country collaboration; effective communications and research networks; implementation flexibility, monitoring and review; continuity of partner institutions and team; and donor influence on design.

Evaluation of the relative success of the forestry projects

The results of this analysis (shown in Figure 3) demonstrate that the apparent success of a project can be quite different depending on whether the evaluation focuses on its achievements, its impacts or both its achievements and its impacts. In the evaluation based on research achievements, eight projects (80 per cent) received scores of six or more, whereas in the evaluation based on research impacts, seven projects (70 per cent) received scores of only four or less. If success requires both high achievements and high impacts, then only three projects (30 per cent) could be considered successful.

Figure 3

Evaluation of relative success of the 10 Vietnam forestry projects.

Figure 3

Evaluation of relative success of the 10 Vietnam forestry projects.

Considering the evaluation scores for both the research achievements and the research impacts, it is apparent that the case study projects represent three categories of project success (see Figure 4): projects with low achievements and low impacts; projects with high achievements but low impacts and projects with high achievements and high impacts. In this case study, there were no examples of projects that had the unlikely combination of low achievements yet high impacts.

Figure 4

Project success categories based on combinations of research achievement and impact evaluation scores, and location of case study projects within those categories.

Figure 4

Project success categories based on combinations of research achievement and impact evaluation scores, and location of case study projects within those categories.

Evidence of success factors in selected projects

The primary author's assessment for the apparent influence of each success factor on project success, derived from the IR and evidence from PR, is shown in Table 4.

Table 4

Expression of success factors within three projects with different evaluated levels of success, with the 12 most frequently identified factors shown in *bold italics

graphic 
graphic 

This analysis showed that for the project that had high achievements and high impacts on the evaluation scores, there was good evidence that the presence of most of the success factors strongly enhanced the project's success. Conversely, the evidence from the analysis showed that, for the project that had low achievements and low impacts, nearly half of the success factors were absent. The project that had high achievements but low impacts showed the presence of some success factors and the absence of others, particularly the absence of links to the impact pathway. These relationships were more evident in information from interview records than in project records. This may be because the interview questions were designed to identify this type of information, whereas project records are variable in content and may not contain information specific to the success factors.

The analysis also showed that there is a reasonably clear relationship pattern between those success factors which can be influenced during project design (Nos. 1, 2, 3, 6, 7, 16, 17, 20 and 21) and the evaluated level of research achievement and research impact. The high achievement–high impact project showed evidence of almost all of these factors strongly enhancing or enhancing the project's success. This demonstrates the importance of careful consideration of these success factors during the design of a forestry research project.

Patterns of relationship were less clear for the 10 success factors, which can be influenced during project implementation. There was evidence that the presence of most of these factors had enhanced the level of success, which suggests that regardless of the quality of the project design, a project team that is well led and focused is more likely achieve the planned project outputs. Similarly, the absence of the success factor related to links to the impact pathway and user benefits appears to have strongly diminished the success of both the high achievement–low impact and the low achievement–low impact projects.

Discussion

Factors that influence a research project's success

Many forestry production systems involve a complex diversity of components, have relatively long production cycles compared with most agricultural crops and involve products that require an efficient value chain and well-developed markets to realize their economic value. This means that forestry research generally requires long-term commitments and multi-faceted programs to generate substantial impacts (Henderson, 2000). Various authors have examined the factors that influence the success of forestry development initiatives which research projects seek to support. For example, preconditions for success of smallholder plantation forestry have been identified as secure land tenure, viable production technologies, the ability to protect trees to maturity and demand and access to profitable markets (Byron, 2001). Factors that influence the success of community forestry programs have been shown to include addressing social, economic and gender inequalities, secure property rights, intra-community governance, government support for community forestry and material benefits to community members (Baynes et al., 2015). While the impact of forestry research projects may be influenced by these factors, there are also other factors that can affect the success of a research project.

Almost all of the success factors identified in this study have relevance for project design and/or project implementation, with only three factors (Nos. 15, 18 and 19) being beyond the control of those who design and implement research projects and one other factor (No. 13) being only partially under their control. The approach used in this study indicates that project participants can identify a wide range of factors that influence success. It also found that it is possible to demonstrate that there is some relationship between the expression of these success factors in a project and its evaluated level of success. However, the findings on success factors should not be regarded as blueprint for successful projects. Rather, they should be considered carefully during project design and implementation and the relevant factors applied where appropriate.

Many of the 22 success factors identified by this study, that can enhance or diminish success of forestry research projects implemented in developing countries, are broadly consistent with those identified in previous studies of research projects (Miles, 1998; Pearce, 2010) and of development projects (Miles, 1998).

However, some are additional to those reported previously, and others highlight the importance of particular aspects of previously identified factors. The additional success factors were

  • provision of adequate funding and facilities to conduct the planned research – this was the third most frequently identified factor and includes having mechanisms to ensure funds flow to researchers in a timely manner.

  • team and technical capacity building – this was the fifth most frequently identified factor and considered a particularly important contributor to greater success. It was previously identified only in the study of construction projects (Miles, 1998). It includes on the job training and mentoring, postgraduate study, study tours and work placements with the Australian partner.

  • site selection and scientific rigour of trails – for those projects for which this factor is relevant, these included elements such as long-term tenure security, appropriateness for species being planted, support of the local community and research being designed and implemented in a way that will produce scientifically valid results.

  • implementation flexibility with processes for monitoring and reviewing activities – this was more frequently identified by Australian respondents, reflecting the importance of having flexibility within the design, systems for monitoring project activities and donor support to review and adapt project activities including through a mid-term review.

  • donor influence on project design – this was considered a positive contributing factor when the donor influenced the quality of the science, but a negative factor when donor driven aspects were imposed or unilateral decisions were made.

  • existence of long-term research collaborations – this was identified as a factor contributing to greater success, and reflects the contrasting situations of projects that follow a previous project with those that are one-off.

  • continuation of the research post project – this was identified by some Australian and Vietnamese respondents, and reflects their view that the willingness of the receiving institution and the scientists to use the new research skills and knowledge to continue related research after the project ends is important in judging a project's success; and

  • project leader's experience in the partner country – this was identified only as a contributor to lesser success and reflects the importance of the project leader having a good understanding of the culture and operating environment in the partner country.

Of the factors that had been previously identified, and for which this research identified particular aspects, the most significant were

  • collaborative scoping and design – including a strong emphasis on the importance of genuine collaboration between the partners in formulating the project design, and the potentially negative impact when Australian scientists insist on aspects of the design, as well as reiterating the importance of properly understanding the topic and situation and then having clear objectives and activities that are not overly ambitious.

  • skills mix and time allocations – this included recognition of the importance of having the right skills in the team to conduct the research as well as having adequate time allocations for each scientist working on the project.

  • institutional support – selecting partner institutions that are genuinely interested and willing to provide institutional support during project implementation.

  • good leadership and management – this was considered relevant to both the international and partner sides of the collaboration and includes ensuring partner scientists understand what tasks need to be undertaken and by when.

  • time in country – funding sufficient travel to enable adequate time to be spent in country working with the partner scientists.

  • effective communications and research networks – while the importance of having good communication within the team has previously been identified, the respondents also emphasized the value of researchers developing and using research networks beyond the team.

  • trust and interpersonal relationships – fostering an environment where partner scientists respect and trust each other, with international scientists displaying cultural sensitivity.

  • project duration – having sufficient time to achieve the planned research outputs; and

  • links to impact pathway and user benefits – previously the importance of having explicit adoption mechanisms had been identified, but this research highlighted the broader issue of embedding the research within the context of the impact pathway and ensuring that the research outputs are relevant to the needs of the end users.

Two factors that had previously been identified (Pearce, 2010) as factors that contributed to the success of ACIAR research projects were not identified by the participants in this research. They were having in-country collaborators with good linkages to other relevant agencies; and the involvement of industry and commercial partners. This may be because the original study included participants from a broader range of agricultural and fisheries projects.

Relationships between success factors and a project's assessed level of success

Previous work by the primary author (Bartlett, 2016a) to develop and test a method for evaluating the relative success of multiple research projects has been extended in this study, by exploring whether relationships exist between a project's assessed level of success and the series of factors thought by project participants to enhance or diminish success. Understanding how the success factors are expressed in projects with different combinations of research achievements and impact could facilitate improvement in the design and implementation of future research projects. Over time, the results of such evaluations and analysis may help to improve the effectiveness of both individual projects and a program of research.

The study has shown evidence that these success factors are manifested in different ways in projects with different levels of evaluated success (see Table 4). It is clear that a project that has high research achievements and high impacts is likely to exhibit evidence that most of the identified success factors have contributed to the enhanced success, as illustrated by the domestication of Australian trees project (FST/1998/096). Conversely, a project that has low achievements and low impacts is likely to exhibit evidence of the expression of these factors that diminish project success. In the project on sawing and drying of eucalypt timber (FST/2001/021), factors such as scoping and design, funding, donor influence on project design, selection of trial sites, and leadership and management all contributed to the lower level of success. These relationships with relative project success appear to be strongly evident for the 12 success factors most frequently identified by project participants.

Conclusions

There is a strong emphasis on aid effectiveness in the delivery of ODA-funded research programs (OECD, 2005). In the case of agricultural (and related) research, it is important to have an understanding of the ways in which desirable impacts can be enhanced and adverse impacts diminished (Millstone et al., 2010). Better understanding of the factors that can enhance or diminish the success of different research projects in different circumstances is an important element of this more general understanding. This case study of 10 ACIAR forestry research projects implemented in Vietnam has identified 22 success factors, 12 of which represent 80 per cent of participants’ responses, indicating that these factors are likely to have a strong influence on the perceived level of success achieved by a project.

The findings from this research on factors that contribute to project success correspond well with those previously identified (Miles, 1998; Pearce, 2010), but also suggest some additional factors and clarified particular aspects of some previously identified factors. Most of the success factors in this study had particular relevance to project design and project implementation. This finding is helpful for research program managers and project leaders, as they have the ability to influence these factors and thereby the ultimate effectiveness of the research project.

This study demonstrated that it is informative to consider both research achievements and impacts when evaluating the success of a research project, and that the success factors identified do relate to levels of project success. Paying attention to success factors related to project design, particularly the degree of collaboration with partners, the experience of the project leader in the country where the project will be implemented and the time allocations for the collaborating scientists, is likely to enhance prospects of the project's success. Success is also influenced by some aspects of project implementation, including the commitment and collaboration of the partners, the degree of capacity building undertaken, the selection of locations for conducting field research, how much time the collaborating scientists are able to spend in country working with their partners, and – where relevant – the quality and design of experimental sites. There are also factors outside the control of a project that can affect its success, including the longevity of the research collaboration, the continuity of partners involved in a project and the mechanisms that enable research outputs to be widely disseminated to end users. Overall, the results reported here suggest that the qualitative approach applied in this research can help understand why some research projects are more or less successful than others, and that the identification of factors that contribute to the level of project success provides useful guidance for those managing and implementing collaborative forestry research programs and projects.

Supplementary data

Supplementary data are available at Forestry online.

Acknowledgements

This research was conducted as part of postgraduate studies while the primary author was employed as the Forestry Research Program Manager by the Australian Centre for International Agricultural Research. The support and encouragement of ACIAR's Chief Executive Officer, Dr Nick Austin, to undertake postgraduate study and to allow access to all the research project records is gratefully acknowledged. We acknowledge the 11 Australian and 13 Vietnamese scientists who were interviewed and their willingness to provide detailed comments on the research projects used in the case study. We thank the two external reviewers for their constructive comments and also Clive Hilliker, from the Fenner School of Environment and Society, for improving some of the figures and tables.

Conflict of interest statement

The primary author is an employee of the Australian Centre for International Agricultural Research.

Funding

This work was not supported by any funding grant, but conducted as part of a part-time postgraduate research degree being undertaken at the Australian National University.

References

ACIAR
.
2014
ACIAR Strategic Plan 2014–18
 .
Australian Centre for International Agricultural Research
.
Amat
,
J.P.
,
Tùu
,
B.P.
,
Robert
,
A.
and
Hùu
,
N.T.
2010
Can fast growing species form high-quality forests in Vietnam, examples in Thù’a Thiên-Huê province
.
Bois et Forêts des Tropiques
 
No. 305
(
3
),
67
76
.
Bantilan
,
M.C.S.
,
Chandra
,
S.
,
Mehta
,
P.K.
and
Keatinge
,
J.D.H.
2004
Dealing with diversity in scientific outputs: implications for international research evaluation
.
Res. Eval.
 
13
(
2
),
87
93
.
Bartlett
,
A.G.
2016
a
Evaluating relative success of donor-funded collaborative research projects
.
Res. Eval
 . doi:10.1093/reseval/rvw009.
Bartlett
,
A.G.
2016
b
The evolution and impacts of ACIAR's forestry research program over three decades
.
Aust. For.
 
79
(
3
),
171
188
.
Baynes
,
J.
,
Herbohn
,
J.
,
Smith
,
C.
,
Fisher
,
R.
and
Bray
,
D.
2015
Key factors which influence the success of community forestry in developing countries
.
Glob. Environ. Chang.
 
35
,
226
238
.
Blamey
,
A.
and
Mackenzie
,
M.
2007
Theories of change and realistic evaluation: peas in a pod or apples and oranges
.
Evaluation
 
13
(
4
),
439
455
.
Byron
,
N.
2001
Keys to smallholder forestry
.
For. Trees Livelihoods
 
11
(
4
),
279
294
.
Byron
,
N.
2014
The acacia economy of Viet Nam
.
Unpublished report for ACIAR
 .
Australian Centre for International Agricultural Research
, p.
76
Carden
,
F.
2004
Issues in assessing the policy influence of research
.
Int. Soc. Sci. J.
 
56
(
179
),
135
151
.
Clement
,
F.
and
Amezaga
,
J.M.
2009
Afforestation and forestry land allocation in northern Vietnam: Analysing the gap between policy intentions and outcomes
.
Land Use Policy
 
26
(
2
),
458
470
.
Commonwealth of Australia
.
1982
Australian Centre for International Agricultural Research Act, 1982
 .
Commonwealth of Australia
.
Davis
,
J.
,
Gordon
,
J.
,
Pearce
,
D.
and
Templeton
,
D.
2008
Guidelines for assessing the impacts of ACIAR's research activitie
.
ACIAR Impact Assessment Series Report No. 58
 .
Australian Centre for International Agricultural Research
. p.
120
.
de Jong
,
W.
,
Sam
,
D.D.
and
Hung
,
T.V.
2006
Forest Rehabilitation in Vietnam: Histories, Realities and Future
 .
Center for International Forestry Research (CIFOR)
, p.
6
.
FAO
.
2015
Global Forest Resources Assessment 2015: Desk Reference
 .
Food and Agriculture Organization of the United Nations
. p.
253
.
Farrington
,
J.
,
Thirtle
,
C.
and
Henderson
,
S.
1997
Methodologies for monitoring and evaluating agricultural and natural resources research
.
Agric. Syst.
 
55
(
2
),
273
300
.
Fisher
,
H.
and
Gordon
,
J.
2007
Improved Australian tree species for Vietnam
.
ACIAR Impact Assessment Series Report No. 47
 .
Australian Centre for International Agricultural Research
. p.
33
.
Forss
,
K.
,
Cracknell
,
B.
and
Samset
,
K.
1994
Can evaluation help an organization to learn
.
Eval. Rev.
 
18
(
5
),
574
591
.
Franzel
,
S.
,
Denning
,
G.L.
,
Lillesø
,
J.P.B.
and
Mercado
,
A.R.
Jr
2004
Scaling up the impact of agroforestry: lessons from three sites in Africa and Asia
.
Agrofor. Syst.
 
61–62
(
1–3
),
329
344
.
Government of Vietnam
.
2007
Viet Nam Forestry Development Strategy 2006–2020. Hanoi, Office of Prime Minister, 54 pp.
Harwood
,
C.E.
,
Ha
,
H.T.
,
Tran
,
H.Q.
,
Butcher
,
P.A.
and
Williams
,
E.R.
2004
The effect of inbreeding on early growth of Acacia mangium in Vietnam
.
Silvae Genet.
 
53
,
65
69
.
Harwood
,
C.E.
,
Hardiyanto
,
E.B.
and
Yong
,
W.C.
2015
Genetic improvement of tropical acacias: achievements and challenges
.
South. For.
 
77
(
1
),
11
18
.
Harwood
,
C.E.
and
Nambiar
,
E.K.S.
2014
Sustainable plantation forestry in South-East Asia
.
ACIAR Technical Reports No. 84
 .
Australian Centre for International Agricultural Research
.
Henderson
,
J.S.
2000
Evaluating the impact of forestry research for development
.
Int. For. Rev.
 
2
(
3
),
191
199
.
Horton
,
D.
and
Mackay
,
R.
2003
Using evaluation to enhance institutional learning and change: recent experiences with agricultural research and development
.
Agric. Syst.
 
78
(
2
),
127
142
.
Joly
,
P.-B.
,
Gaunand
,
A.
,
Colinet
,
L.
,
Larédo
,
P.
,
Lemarié
,
S.
and
Matt
,
M.
2015
ASIRPA: a comprehensive theory-based approach to assessing the societal impacts of a research organization
.
Res. Eval.
 
24
(
4
),
440
453
.
Lindner
,
R.
,
McLeod
,
P.
and
Mullen
,
J.
2013
Returns to ACIAR's investment in bilateral agricultural research
.
ACIAR Impact Assessment Series No. 86
 .
Australian Centre for International Agricultural Research
. p.
48
.
Mayne
,
J.
and
Stern
,
E.
2013
Impact evaluation of natural resource management research programs: a broader view
.
ACIAR Impact Assessment Series Report No. 84
 .
Australian Centre for International Agricultural Research
. p.
79
.
McElwee
,
P.
2009
Reforesting “Bare Hills” in Vietnam: social and environmental consequences of the 5 Million Hectare Reforestation Program
.
AMBIO
 
38
(
6
),
325
333
.
McLeod
,
L.
,
Doolin
,
B.
and
MacDonell
,
S.G.
2012
A perspective-based understanding of project success
.
Project Manag. J.
 
43
(
5
),
68
86
.
Mercer
,
D.E.
2004
Adoption of agroforestry innovations in the tropics: a review
.
Agrofor. Syst.
 
61-62
(
1–3
),
311
328
.
Miles
,
D.W.J.
1998
Evaluating international development projects: lessons from the construction sector
.
Evaluation
 
4
(
4
),
469
482
.
Millstone
,
E.
,
Van Zwanenberg
,
P.
and
Marshall
,
F.
2010
Monitoring and evaluating agricultural science and technology projects: theories, practices and problems
.
IDS Bull.
 
41
(
6
),
75
87
.
Nambiar
,
E.K.S.
,
Harwood
,
C.E.
and
Kien
,
N.D.
2014
Acacia plantations in Vietnam: research and knowledge application to secure a sustainable future
.
South. For.
 
77
(
1
),
1
10
.
Nguyen
,
V.S.
and
Gilmour
,
D.
1999
Forest rehabilitation policy and practice in Vietnam
.
Proceedings of a National Workshop November 1999
 .
IUCN
.
OECD
.
2005
Paris Declaration on Aid Effectiveness. Paris, Organisation for Economic Cooperation and Development, 15 pp.
Pattanayak
,
S.K.
,
Mercer
,
D.E.
,
Sills
,
E.
and
Yang
,
J.-C.
2003
Taking stock of agroforestry adoption studies
.
Agrofor. Syst.
 
57
(
3
),
173
186
.
Pearce
,
D.
2010
Lessons learned from past ACIAR impact assessments, adoption studies and experience
.
ACIAR Impact Assessment Series Report No. 69
 .
Australian Centre for International Agricultural Research
.
Pearce
,
D.
,
Jilani
,
A.
and
Templeton
,
D.
2013
Adoption of ACIAR project outputs 201
.
ACIAR Adoption Studies Reports No. AS010
 .
Australian Centre for International Agricultural Research
. p.
87
.
Phi
,
H.H.
,
Harwood
,
C.
,
Beadle
,
C.
,
Nambiar
,
S.
,
Pinyopusarek
,
K.
,
Vu
,
D.H.
et al
.
2009
Sustainable and profitable development of acacia plantations for sawlog production in Vietnam. Unpublished report from AusAID's Collaboration for Agriculture and Rural Development (CARD) program - Project Number: 032/05 VIE, p.
10
.
Phuc
,
X.T.
and
Canby
,
K.
2011
Vietnam: Overview of Forest Governance and Trade
 .
European Forest Institute
. p.
51
.
Putzel
,
L.
,
Dermawan
,
A.
,
Moeliono
,
M.
and
Trung
,
L.Q.
2012
Improving opportunities for smallholder timber planters in vietnam to benefit from domestic wood processing
.
International Forestry Review
 
14
(
2
),
227
237
.
Raitzer
,
D.A.
2003
Benefit-Cost Meta-Analysis of Investment in Interational Agricultural Research Centres of the CGIAR
 .
Food and Agricultural Organisation of the United Nations
. p.
45
.
Stern
,
E.
,
Stame
,
N.
,
Mayne
,
J.
,
Forss
,
K.
,
Davies
,
R.
and
Befani
,
B.
2012
Broadening the Range of Designs and Methods for Impact Evaluations
.
Working Paper 38
 .
Department of International Development London
, p.
127
.
Suyanto
,
S.
,
Pandu Permana
,
R.
,
Khususiyah
,
N.
and
Joshi
,
L.
2005
Land tenure, agroforestry adoption, and reduction of fire hazard in a forest zone: a case study from Lampung, Sumatra, Indonesia
.
Agrofor. Syst.
 
65
(
1
),
1
11
.
Thangata
,
P.H.
and
Alavalapati
,
J.R.R.
2003
Agroforestry adoption in southern Malawi: the case of mixed intercropping of Gliricidia sepium and maize
.
Agric. Syst.
 
78
(
1
),
57
71
.

Additional information on the three projects studied to explore the expression of success factors in projects assessed as having different success categories

FST/1998/096 ‘Domestication of Australian Trees’

This 5-year domestication and tree breeding project increased the capacity of Vietnamese tree breeding scientists and established seedling seed orchards, seed production areas and genetics gains trials for key Acacia and Eucalyptus species. Various scientific and technical publications were produced by project partners including an international journal article (Harwood et al., 2004). This project followed more than 10 years of previous collaboration related to supply and testing of Australian tree germplasm.

The project has been assessed by the author and the respondents as the most successful of the 10 projects studied. Almost all of the identified success factors apparently have contributed to enhancing its success with about half of the factors being considered by the respondents to have strongly enhanced its success. The project design that had been influenced by the donor was relatively simple, but effective. Only one of the four objectives had a strong research focus while the others focussed on related development activities. The project had good leadership on both the Australian and Vietnamese sides, both partners were strongly committed to the project and saw mutual benefits from the project. The Australian scientists had strong scientific skills in tree breeding and domestication of Australian trees and had existing working relationships with the Vietnamese partners, with the main scientists spending 60–80 per cent of their time on the project. The Vietnamese scientists were well led and highly motivated and their institution contributed additional financial resources to expand the number of trials established.

An ACIAR impact assessment found that this project and its predecessor had generated very substantial economic impacts from the widespread planting of improved tree germplasm and the research investment had generated benefit cost ratio of 79:1 (Fisher and Gordon, 2007). This project was very well connected to the impact pathway during its implementation and after it concluded. The Government of Vietnam established mechanisms through the Ministry of Agriculture and Rural Development to disseminate the certified tree germplasm to farmers throughout the country under the 5 Million Hectare Reforestation Policy. The project outputs have had an enduring legacy, with the seed orchards still being used today. One Vietnamese respondent indicated that seed from the seed orchards is marketed globally and the profits are used to fund additional tree breeding research.

FST/2006/087 ‘Optimizing silvicultural management and productivity of high-quality acacia plantations, especially for sawlogs’

This 4-year project focused on developing silvicultural practices to enable production of sawlogs from smallholder plantations, in support of Vietnam's goal to increase the supply of domestically produced timber for its wood industries (Government of Vietnam, 2007). When acacias are grown for pulpwood rotations of 5–6 years are common, whereas rotations of 10–12 years are needed for one quarter of the logs to achieve sawlog specifications (Byron, 2014). The project followed a 3-year development project (AusAID's Collaboration for Agriculture and Rural Development (CARD) Project Number: 032/05 VIE), involving pruning and thinning trials in acacia plantations in north-central Vietnam, which had showed some promising prospects for sawlog production – although some of the trials were impacted by a typhoon in 2008 (Phi et al., 2009). This ACIAR project established new trials involving fertilization, thinning and pruning at seven sites located in southern, central and northern Vietnam and monitored these trials for 3 years.

The project was assessed by the author as having high research achievement but low impact. It would always be difficult for a 4-year project on a forestry system that takes 10–12 years to reach rotation age to achieve substantial impacts for end users. The project design included activities to disseminate information to smallholders but these were not implemented during the life of the project. The analysis shows that most of the success factors related to the project implementation phase contributed positively to the success of the project, though there were problems related to poor collaboration between partners in the different regions of Vietnam where the various trials were located.

The weaknesses in this project appear to relate predominantly to various success factors related to the project design. The duration of the project meant that, while the project produced good information on the system's productivity up to age three, it could not present conclusive results on the sawlog system's financial returns, which is necessary to convince growers to change their practices and delay income receipt for several years. It was also apparent from the respondents that lack of effective collaboration with Vietnamese partners on the project design and ACIAR's influence on the selection of partners and locations for the research trials diminished the project's success.

FST/2001/021 ‘Improving the value chain for plantation-grown eucalypt sawn wood in China, Vietnam and Australia: sawing and drying’

This 4-year project was designed to conduct research related to improving the production of sawn timber from small diameter eucalypt logs, with research conducted in China, Vietnam and Australia. Apart from building research capacity, the project conducted a sawing trial involving 10-year-old Eucalyptus urophylla logs processed in a small sawmill in Vietnam. This analysis focused on the activities conducted in Vietnam but it is apparent that there were greater achievements in China (Pearce et al., 2013).

The Vietnamese component of the project was assessed by the author as having low achievements and low impacts. The analysis suggests that the project was poorly designed, with many of the success factors related to project design contributing to diminished project success. The analysis indicated that respondents considered about half of the success factors related to project implementation, particularly the capacity building factor, had contributed to enhanced success. Inadequate attention to the others resulted in diminished success. At the completion of the project, the scientific reports from the Australian sawing trials were not translated into a manual that could be easily understood by Vietnamese partners. The project had no mid-term review, which precluded a discussion on how the research might have been refocussed to generate outputs more aligned to end user needs.

There were four design-related issues that also diminished success. Firstly, there was inadequate scoping and collaboration with Vietnamese partners in the project design. ACIAR and the Australian researchers assumed that research was necessary on the production of sawn timber, rather than on other products, such as veneer, and that there were sufficient suitable eucalypt resources existing in Vietnam to sustain a sawlog industry. Secondly, it assumed that appropriate and committed Vietnamese wood processors could be found to participate in the research and then adopt the recommended practices. However, only one small sawmill participated and it did not have the technology available to properly dry or recondition the sawn timber. Thirdly, inadequate funding was provided for the planned activities, with ACIAR reducing the project's funding by 46 per cent in the final stages of design without adjusting the magnitude of the planned research activities. Fourthly, the project leader had not previously worked in Vietnam and only became involved in the final stages of the project's design, following the retirement of the planned leader.

Author notes

3
Present address: Forestry Research Program Manager, Australian Centre for International Agricultural Research, GPO Box 1571, Canberra, ACT 2601, Australia.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.