Appendix A4 Mid-term Evaluation Report (February 2003) Introduction

Quality and Utility of Specific Products

1. Biodiversity Assessments

Many international and national experts associated with this project have conducted a number of field biodiversity surveys at the four sites and produced many technical reports. However, the quality of the reports is variable, from good professional work to mere collections of out-of-date data drawn from early work of the local research institutes and universities. Some reports actually contain misleading information; for example, the Honghe NR list referred to many dry-land and even montane species that are not, in fact, found within the reserve boundaries. It is highly questionable whether international consultants should have undertaken so much basic data collection and data checking in the course of such short visits. Where domestic experts have been hired, they have often been used as interpreters and facilitators.¹⁴ The division of work responsibilities between international and national experts has not proven to be an efficient use of resources. National experts who know the language, customs and species in the areas would have done the basic survey and inventory work much more efficiently. International experts, on the other hand, would be better employed drawing on their broad experience to provide general guidance, introduce new methodologies and find new ways of collecting and analysing biodiversity data from their broad experience.

2. Socio-economic Surveys¹⁵

1. 
   an international social economic advisor (ISEA) to be recruited for five person months and spending one month a year in the CPMU and local PMUs, the role being to provide advice and support to the project and to backstop sub-contracts 2 and 4 ; and
   

2. 
   a national socio-economic advisor (NSEA) to be recruited for 41 person months (the equivalent of one full-time person for 3.4 years), the role being to act as a CPMU-based technical support person.
   

Between preparation of the project document and implementation, the TOR for the ISEA changed to the following at each project site:

• 
  to assess actively social and economic conditions;
  
• 
  to determine specific needs for social and economic monitoring;
  
• 
  to advise on a framework for baseline socio-economic surveys to be carried out at the community and household levels; and
  
• 
  to propose a participatory mechanism for community-project conflict resolution.
  

As a result of this change in role, the five one-month inputs became one two-month inputs and three one-month inputs, delivered “at appropriate times during the remaining period of the contract.” The role of the NSEA was also changed from technical support to become essentially an assistant and language interpreter in support of the ISEA.

• 
  How will the more detailed and focused social-economic baseline studies – where required – be initiated. How will they be paid for? (Note that the ADB-funded Sanjiang Plain project will include technical assistance for the socio-economic assessment and that the WWF Yangtze programme includes various focused social and economic surveys and studies.)
  
• 
  How will the socio-economic monitoring – where required – be designed, initiated and implemented, and who will carry out the site-level data gathering? (Note that the WWF Yangtze programme also includes monitoring work.)
  
• 
  How will these activities be guided and supported, given the apparent exhaustion of contracted time for the ISEA and the “absence” of a NSEA?
  

3. Training

Training is the most important benefit of the project to date. Almost everyone the MTR met identified training as the key benefit. Some of the training is already having an impact: nature reserve staff reported that they had used Participatory Rural Appraisal (PRA) approaches to talk to local people and found them helpful. The effect of other training is more intangible, peoples’ general knowledge of wetland issues, co-management and other approaches have increased but so far this has not translated into changes in how they perform their jobs. The study tours have also had an influence; one provincial official observed after visiting New Zealand that its not important to build large buildings on nature reserves. Another provincial official returned from visiting wetland sites in Europe with a new understanding and commitment to preserve wetlands in China. The MTR team agreed that training is the most successful aspect of the project. We also feel that the training could be made more effective, with proper targeting, planning, selection of participants and better tailoring of the training to needs.

Expenditures on training – On the project input side, $4,129,00 USD are explicitly earmarked for training (see Table 3.2). In addition, training is a component of a number of other activities, although the cost for the training is not indicated separately. (See Table 3.3.)

The Training Needs Assessments – Four different consultants were hired (one for each site) to complete the training needs analysis at the beginning of the project. These generated separate reports, which were complemented by an overview or synthesis report prepared by one of the TNA specialists. The TNAs generally followed a standard and accepted formula at each site and for the CPMU and came up with a standard package of training activities involving six types of training:

• international and in-country fellowships;

• international and in-country study tours;

• international-standard short courses delivered in-country; and

• national short courses or workshops.

There are two points of interest here: (i) the kinds of training suggested by the TNA specialist closely resembled the list included in the original project design document (page 67); and (ii) the training workshops and short courses proposed for the provinces and nature reserves were general in character, rather than tailored to the specific sites.

However the CPMU described this overall plan, with listed courses and personnel priorities, as “unworkable” due to, among other things, the consultant’s “poor knowledge of the actual situation in China.” The CPMU then recruited its own training manager/consultant to produce, a “new practical training program that met the actual needs of the project sites.” Despite not having a workable training plan between July 2000 and April 2001, a number of training courses were delivered, including site-level short courses in basic computer and the Internet, financial management, reserve management planning and wetland survey, monitoring and evaluation.

Table 3.2

** Does not appear on project description list of specialists to be recruited (see consultant’s final report, Appendix A).

It is difficult to discern any substantial difference between what was proposed by the international TNA specialist in terms of topics, content and trainers’ content for the national training workshops and what was actually implemented as workable training by the CPMU. The programme of short courses adopted by the CMPU closely resembled that laid down in the project document. The short courses proposed were still mostly generic and not tailored to the local needs (tailoring of training was an improvement suggested to the MTR by nature reserve and provincial officials). The only differences that we noted were the expansion of international study tours (the budget was revised to accommodate this) and expansion of the number of key officials from SFA and CPMU participating in international and national study tours and fellowships.
Content and targeting of training – It is difficult to discern whether the national short courses were successful or not, as English-language versions of the national training specialists’ reports were not available to the MTR team. However, reports on the courses delivered by international trainers were available. A particular concern repeatedly raised by these trainers (for example, by the PRA and co-management trainers) was that a number of participants in their courses had no subjective or objective need for the particular training. Three kinds of mismatch were noted: (i) too senior people attending field-level courses; (ii) field level staff attending courses targeted to senior management or decision-makers; and, (iii) people attending course in areas not relevant to their jobs.
A review of the available materials dealing with course content indicated that the training actually provided by the international specialists was of a reasonably high quality. However, much of its value may not have been realized because it was delivered through a translator/interpreter who lacked sufficient knowledge of or experience with the subject matter. More effort could have been made by UNOPs and other parties to locate and provide trainers with the necessary language skills and technical expertise from within the international or Asian regional scene.
CPMU training needs – According to the CMPU’s report to the MTR, central officials (CMPU and senior project management) received a high proportion of the “senior level” training (as shown below), compared to senior staff of the four PPMUs and 11 nature reserves:

• 
  1 of the 3 international fellowships;
  
• 
  10 of the 24 positions on international study tours (shared among six officers);
  
• 
  9 of the 44 positions on national study tours;
  
• 
  8 attendances at overseas seminars or workshops; and
  
• 
  5 attendances at international seminars or workshops delivered in China; and
  
• 
  2 of the 5 national fellowships for PhD degrees.
  

The project design did not envisage extensive training and capacity-building of CPMU or senior project management staff. However, there has clearly been extensive training, and, in our view, there are several plausible reasons for this:

• 
  the project document overestimated the capacity of SFA personnel and should have provided capacity building for project staff
  
• 
  the persons appointed to the particular positions did not reach the required professional standards outlined in the project document and thus required training,
  
• 
  there was a lack of understanding of the nature of the project and of the intended training focus on individuals from outside the project office (i.e. nature reserve, provincial and national officials), or
  
• 
  staff treated the training as a perquisite or bonus.
  

4. Equipment

• 
  Unlike the specifications in the project document, equipment has been supplied to 11 Nature Reserves, not the 6 identified in the project document. As a result, there has been a dilution of the equipment delivered to individual nature reserves from what was planned. (eg. 10 vehicles have been distributed across 11 nature reserves [1 per reserve except West Dongting], instead of 6 reserves).
  
• 
  Although most of the equipment for NRs has already been procured, the reserves still have very little equipment for biodiversity monitoring and patrolling. Without such equipment, the NR staff is unable to perform these functions properly.
  
• 
  Central procurement of equipment following the NEX guidelines has caused delays and increases costs. Also, because equipment is not procured through local suppliers, no training or servicing of the equipment is available for the equipment once it is delivered.
  
• 
  Whereas it was felt necessary to base the training programme on a training needs assessment it was not felt necessary to base equipment delivery on any clear analysis of real needs. Instead, a standard packet of the same equipment was delivered to each province and project site. In no case was equipment sufficient and it was sometimes inappropriate. To maximize the use of equipment as well as to suit the local condition, a clear analysis of the real needs of each site, including the need for training on the use and maintenance of the equipment should have been done before any equipment was procured¹⁶.
  
• 
  Equipment has been provided without consideration of how it would be maintained. Before purchasing equipment there should be a commitment from provincial co-funding agencies that they were able or willing to pick up the consequent maintenance and operation costs.
  
• 
  A packet of good quality GIS hardware and software (ArcGis) has been delivered to PPMU at each site and, however, at the time of the MTR, no training on the equipment had been provided.
  
• 
  The distribution of equipment between CMPU, PPMU and NR needs to be reviewed. CPMU appears to have more equipment than planned (eg. 14 desktop computers and 6 laptops vs 8 and 2 in the plan), including technical and field equipment that was not designated for CPMU (eg. GPS systems, telescopes, binoculars and tents). The nature reserves appear to have received less (eg. 11 telescopes sent to NR reserves vs 25 in the plan, 0 cellular phones vs 20 in the plan). Generally PPMUs appear to have received what was planned, except for vehicles, where 9 instead of 5 have been provided to PPMUs.
  

4. Protected Area (PA) Management Plans

At the time of the MTR mission, management plans had been developed for seven nature reserves (the 6 identified in the project document plus one in Ruoergai). Unfortunately, all of the management plans developed by international GEF consultants are regarded by Nature Reserve staff as impractical, as they are too costly to implement Although none of the plans can be implemented as they stand, the exercise has not been a complete waste: some NR managers view the plans as ideal and expect to use them as guidance while developing practical plans, other managers intend to implement some (affordable) parts of the plans. Another positive contribution to PA management has been through the delivery of training in PA management and Participatory Rural Assessments. Capacity to identify key species in PAs has been enhanced but only for birds. NR staff remains very poor at recognizing plants, fish and invertebrates and unaware of threats from invasive species.

• 
  There is no budget for implementation. GEF funds paid for development of plans but not for financing their implementation. The project document is not explicit about this but appears to have assumed that implementation would be paid from provincial or national budgets as part of the co-funding. Whether or not provincial and national governments knew of this assumption, they have not made any budgetary provisions to implement the plans.
  
• 
  The TORs for consultants hired to develop management plans required a comprehensive plan, did not require the plans to be doable (i.e., within the financial means of NRs) and did not allow much field/consultation time for consultants to discuss the plans with local authorities.
  
• 
  International consultants prepared these management plans without adequate consideration of the budgets of the agencies that would implement them.
  
• 
  Basic data needed for the plans was either lacking, not translated or provided very late in the planning process. The lack of data has affected the quality and accuracy of management plans. The most critical data gaps were biodiversity and socio-economic information and good quality maps. Other data sometimes missing were the master plans of NRs.
  
• 
  Too much emphasis was placed on producing PA management plans, rather than supporting local NR staff to develop the plans (a TORs problem). By undertaking much of the planning themselves, project experts have undermined the role of local planners and managers rather than help build up their capacity.
  
• 
  Improper use of international and national experts. As with the socio-economic and biodiversity assessments, international experts were expected to collect the basic background information themselves. If NR staff or national experts had done this it would have been much more efficient. Qualified national experts, although available in China, were often not recruited to support the international experts, and separate translators were usually not hired (see Appendix C). As a result, instead of being technical, the national experts’ role often was reduced to translation.
  
• 
  Dilution of effort. Management plans for all four nature reserve in Ruoergai have been developed. Using expensive international consultants to do the same job four times is a waste of resources and doesn’t allow any learning and improvement of the planning approach from one plan to the next.
  
• 
  Another way to make the plans practical would be to monitor and revise them as they are implemented. However, there is no budget to do this. The budgets for the international experts are all used up.
  

4. Subcontract 1: Basemaps and Biodiversity Overlays

The time estimates in the project document for delivery of basemaps are unrealistic. Due to the time required to complete the bidding process, sub-contract 1 started much later than anticipated. This lead to frustrations by consultants who had to complete field surveys, data collection and management plans without the benefit of the base maps or GIS overlays that should have formed a planning tool.

• 
  Is it sensible or appropriate to have this work done through an external sub-contractor rather than completing the work in SFA’s own GIS division, and building SFA’s capacity in the process? This point was raised by CPMU and PPMUs all of whom feel SFA could have undertaken this work.
  
• 
  Who is going to develop the methodology for collecting the biodiversity data and how will this data be analyzed and integrated into useful maps/models for management purposes? The institute doing sub-contract 1 is technically excellent at GIS and map production but does not have the biodiversity data collection, analysis and modeling skills need to produce practical biodiversity overlays. This issue is discussed farther in Chapter 5.
  

4. Subcontract 3: Restoration of Honghe Hydrology

This sub-contract is slated to finish sometime in the spring of 2003. At the time of the MTR, the sub-contractor had produced a plan costing roughly $8 million USD to construct a rubber dams to retain water in Honghe NNR. The cost of this solution is far in excess of the $1,087,500 in counterpart funding earmarked in the project document for water compensation engineering works in Sanjiang (page 31 of project document). Unless a cheaper solution is found, it is unlikely that the water level will be restored in Honghe NNR. Moreover, the plan may not be flexible enough to cope with further developments and water diversions, as it has been developed in isolation of other plans (such as the proposed expansion of Honghe NNR to more than three times its size). Unless the work becomes more practical and linked with future development and water restoration plans, the investment will have been wasted. A final consideration is that the flora of the site may be already compromised by the lowered water levels.

1. 
   Project Management capacity development (training) – Staff who have received training at the CPMU will be available to support ongoing work on other projects. In particular, a group of SFA staff has been learning project management skills. Also, English-language and computers skills have increased in CPMU and, to a degree, in the PPMUs/provincial offices.
   

2. 
   National consultant development – Consultants who work with the project will take away from it a much better understanding of such projects and the problems related to them. Their capacity to design more practical programs will be significantly enhanced. In a more general way, national experts have come into contact with new ideas and new ways of thinking through working with international experts.
   

3. 
   GEF and UNDP credibility – The credibility of GEF and UNDP has been undermined as a result of the flawed bidding process for sub-contracts and the misuse of international consultants. Problems with the sub-contracts and with management of international consultants have caused financial losses for some contractors (for example, Wetlands International) and have reduced their goodwill towards the project and its partners.
   

4. 
   Institutional change – The high profile of this project and perceived problems with performance may galvanize interest in and a desire for change among major partners. For example, GEF is discussing policy changes related to monitoring of the Wetlands Project. The CPMU/SFA have expressed interest in a sharper focus on quality and results, instead of merely delivering activities, and they are supportive of higher levels of provincial involvement.