Formative Evaluation of UNICEF’s Water Supply Programme in DPR of Korea

, , KP

1. Background

The consultancy is linked to Annual Work Plan (AWP) 2016 with specific reference to Output 4.1 – By 2016, National and Provincial Government capacity to review and update WASH sector policies, plans and strategies to support, evidence based and high-impact intervention increased.

There was significant Government investment in water supply through electrical pumping all over the country during 1960s and 1970s. However, due to the insufficiency of electrical power, erratic power supply, underinvestment in maintenance and replacement of the electrical and water network grids, the water supply systems in most of locations are not functioning well, compromising quality and quantity of water delivered to beneficiaries. In addition, it puts burden on communities, including energy needed for construction, repair and maintenance. Ultimately, inadequate water supply systems threaten the health and well-being of children and their families, and comprise efforts to improve the lives of children through interventions in health, nutrition and other sectors.

Realizing the above factors, government partners and Ministry of City Management (MoCM) appreciate Gravity Flow Water Supply System (GFS) as a sustainable solution. Now, UNICEF has prioritized GFS together with solar pumping options for new water supply projects.

UNICEF piloted the first GFS project in 2001 and has so far implemented 30 (Up) County Town level and 11 (Ri) Village level GFS projects. UNICEF conducted WASH Programme reviews in 2006 and 2008 to learn from the past and improve the programme in the coming years. One of the specific recommendations in 2008 emphasizes supporting piped water system with GFS technology rather than pump based system to ensure sustainability. In DPR of Korea, WASH programmes are also implemented by other international organizations namely: WHO, IFRC, SDC and five NGO’s which operate under the EU umbrella.

In 2016 the MoCM and UNICEF have continued to collaborate on water supply projects addressing the needs of county towns. Additionally, MoCM and UNICEF are building on the mission to develop a shared learning agenda, with specific attention to sustainability of the water supply projects, specifically the GFS.

The UNICEF 5-Year WASH Programme has the following broad target: “By 2021, women and children have equitable access to sustainable, clean water and sanitation services, and practice improved hygiene and sanitation behaviours including in humanitarian situation”. To help realize this the WASH Programme is targeting scaled-up water supply interventions in the 9 Convergence Counties in which UNICEF is supporting  Health, Nutrition and WASH interventions within an integrated approach for improving maternal, neonatal and child health. In partnership with MoCM, UNICEF will not only scale up the programmes in the 9 convergence counties but will also support interventions to improve the sustainability of water supply systems implemented in the past. 

2. Overall Purpose and Objective of the Formative Evaluation

The purpose of this formative evaluation is to review the UNICEF DPRK Country Office’s WASH Programme over the last 10 years, focusing on GFS systems. In addition the evaluation will provide guidance to ongoing programme in the 9 Convergence Counties under new County Programme for 2017-2021.

This formative evaluation is aimed at identifying and documenting: 

  1. What has worked well, what are the good practices and strengths;
  2. What did not work well and what are the weaknesses and challenges;
  • How can we improve strategies and programme designs. 

The formative evaluation is expected to identify good practices and lessons learnt in the implementation of the programme as well as barriers and bottlenecks, and use these findings to formulate recommendations to improve the current and future WASH programme. 

This formative evaluation findings and recommendations will guide WASH Programme to: 

  1. Identify both management and operational gaps on previous programmes and ways to bridge these
  2. Improve the management of existing convergent programming, design of new activities and work plans with government, and strengthening of programme monitoring and partnerships.

The formative evaluation recommendations will be informed by regional and global UNICEF WASH strategies and norms, and should purpose concrete ways on how to align GFS WASH Programme to the new Global WASH Strategy for 2016-2030, taking into account the particular circumstances of the programming environment in DPR of Korea. This formative evaluation will also be guided by UNICEF evaluation findings and reviews in other similar countries, especially with respect to measures and methodologies for assuring the sustainability of water supply systems.

The formative evaluation will cover GFS projects implemented over the last decade and some of the current ongoing GFS projects with the following objectives: 

  • Evaluate the progress made in design, construction and completion of GFS;
  • Assess the range of challenges faced by local authorities to operate and maintain GFS systems (along with any solutions to challenges that have been employed locally) and use the assessment to formulate recommendations for improvement;
  • Determine whether approaches and lessons learnt have been and can be further replicated and scaled-up to wider areas and/or to the national level;
  • Assess whether the GFS programme has improved over the years and if good practices are integrated into the newer programmes;
  • Review available evidence on health and living standards changes for children and women due to the implementation of GFS projects.

 This formative evaluation will also consider:

  • Equity – to what extent the programme has reached hard-to-reach, disadvantaged and vulnerable communities, and the degree to which the convergent approach has incorporated an equity approach and contributed to a reduction in national and sub-national disparities for children based on geography, gender and disability.
  • Relationship and coordination with national and sub-national stakeholders, and identify opportunities for improved collaboration and coordination.   

3. Formative Evaluation Process and Methods

This formative evaluation will be conducted as per the ToR, which will require both quantitative and qualitative analysis and therefore appropriate methods and tools shall be designed.

The Formative Evaluation focuses on the five criteria defined by OECD/DAC, in addition to the criteria of equity (as described above): 

  • Relevance and estimated influence/likely impact of the programme;
  • Efficiency – correlation between time, effort, knowledge, money invested (cost benefit analysis) and results achieved;
  • Effectiveness – results and outcomes, main enabling and hindering factors in achieving the targets and goals and impact of the programme interventions;
  • Sustainability of the programme – ownership by local communities, education/health institutions; main enablers and bottlenecks that can enhance/inhibit sustainability of achieved changes for UNICEF;
  • Impact – whether or not the programme meets or exceeds the stated outcomes/expectations. 

To serve the interest of UNICEF, these criteria should be applied in the framework of UNICEF’s Equity based Approach[1], Human Rights Based Approach as well as Results Based Management. 

Suggested Methodology

The evaluator is expected to use mixed methods to capture both qualitative and quantitative data, including desk review, semi structured interview and observations by conducting field visits and hold consultations with project partners and to some extent target beneficiaries if permitted. The evaluator is expected to liaise closely with UNICEF and MoCM. 

4. Sources of Information 

  • Previous project review reports, assessments and evaluations;
  • Assessment Report on Focus County Approach 2015;
  • DPR Korea Country Programme Development 2017-2021;
  • DPR Korea MICS survey, 2009
  • DPR Korea Socio-economic, Demographic and Health Survey (SDHS), 2014
  • UNICEF DPRK Country Office Annual Reports
  • Detailed explanation of the programme objectives, outline and activities by the implementing partner;
  • Manuals, reports from capacity building interventions;
  • Data collected through survey questionnaires, interviews with key stakeholders, focus group discussions.
  • Feasibility Reports
  • Donor Reports
  • UNEG Norms and Standards for Evaluation
  • UNICEF-Adapted UNEF Evaluation Reports Standards
  • UNICEF WASH global Annual Reports and EAP WASH Annual Reports
  • UNICEF WASH Strategy, 2016-2030 

5. Work plan 

The table below shows a preliminary formative evaluation schedule that may be subject to change during the process in agreement with UNICEF. The total period of this formative evaluation (distance and in-country) with respect to the level of effort from the evaluator is expected to be 8 weeks conducted over a period of approximately 12 weeks starting from 29th September and ending on 19th December 2016, including 4 weeks within DPR of Korea (in Pyongyang and the provinces). 

Description

Responsible

Timeline (Prelim.)

Evaluation – inception phase

 

 

Desk review of the existing documents with focus on GFS projects

Evaluator

 

3 days (starting 1 week from the date of signing the contract) (Distance)

Development of the evaluation methodology, formative research tools and protocols; instruments and work plan (including the questionnaire)

Evaluator with help of UNICEF, MoCM, CBS

3 days (Distance)

Field testing and adopting of the tools

Evaluator with help of UNICEF, MoCM, CBS

3 days (In-country)

Inception report[2] (including evaluation work plan and timeline, presentation of methodological approach, instruments to be used, annotated outline of final report), to be presented and approved by UNICEF and MoCM

Evaluator

5 days (In-Country)

Evaluation – implementation

 

 

–        Data gathering (field visits, meetings, interviews, focus group discussions)

–        Coding and data entry 

–        Data processing and analysis

Evaluator

 

15 days (In-Country)

Evaluation – report preparation

 

 

Interim evaluation report  and presentation (draft findings with comprehensive quantitative and qualitative analysis, conclusions and recommendations from all data sources used in the evaluation) 

Evaluator

 

3 weeks  (Distance)

Feedback and comments from UNICEF and MoCM

UNICEF, MoCM

implementing partner

2 weeks

Incorporation of recommendations from UNICEF and

MoCM after review

Evaluator

 

3 days (Distance)

Final evaluation report with presentation (including summary), subject of approval by UNICEF

Evaluator

 

3 days (Distance)

Use of evaluation findings:

 

 

Dissemination of the final report to all partners and stakeholders and discussions on further improvement of the programme and next stages

UNICEF & MoCM

Onwards 

 Required outputs and deliverables

 The evaluator is expected to deliver the following:

  • Development and presentation of the formative evaluation methodology, formative research tools and protocols; instruments and work plan (including the questionnaire)
  • Inception report
  • A debriefing to present the findings, conclusions and recommendations to UNICEF.
  • Presentation and submission of draft evaluation report including all the findings, conclusions and recommendations.
  • Presentation and submission of final report incorporating the comments and suggestion on the draft report. 

Final report (approximately: 30-40 pages) consists (but not limited to) of the following chapters and in line with the UNICEF Evaluation Report Standards[3] and the Global Evaluation Reports Oversight System[4]: •

  • Executive summary
  • Country context and Background
  • Description of the evaluated projects
  • Description of formative evaluation methodology (including a discussion of its limitations)
  • Description of main findings organized by formative evaluation criteria including analysis, conclusions, lessons learnt and recommendations: short, medium and long-term accountabilities
  • Body of the text that covers all the components of the programme, including the institutionalization of the programme, sustainability and key constraints;
  • List of activities that are implemented
  • Annexes: case studies, formative evaluation instruments (questionnaires, focus group reports), list of project locations, institutions/organizations and individuals that participated in the process, bibliography and any quantitative database developed as part of the formative evaluation;
  • Complete database/data sets (filled out questionnaires)
  • Dissemination materials (two pager containing main findings in English language, power point presentation) 

6. Accountabilities, Reporting 

The evaluator will lead the formative evaluation process at all stages and coordinate with UNICEF, CBS, MoCM and other stakeholders. The evaluator is responsible for provision of deliverables listed above in time and with good quality. The evaluator will be managed by Murat Sahin, Deputy Representative, UNICEF, DPRK. 

UNICEF staff will review and approve the deliverables listed and provide relevant documents. 

7. Qualification Requirements (Selection Criteria) 

The evaluator shall be an international WASH expert with background in carrying out evaluation in UNICEF WASH programmes with experience in WASH programme design and strategy formulation. The evaluator will be responsible for all components of the formative evaluation including development of the evaluation methodology (subject to UNICEF approval); selection, training and overseeing field work of the enumerators (for both qualitative and quantitative components) and quality assurance of the process of data analysis and report writing. The evaluator shall fulfill following crieteria: 

  • High quality project appraisal as per the requirements of the ToR, including methodological aspects. (compliance with the ToR)
  • Proven expertise and experience of the evaluator in carrying out evaluations and/or assessment of WASH programmes/projects (10 years);
  • Fifteen (15) years of experience in evaluation specifically in the area of programme and strategy design and assessment, including demonstrable specific knowledge of UNICEF WASH programming;
  • Excellent knowledge of monitoring and evaluation methodologies (demonstrated by previous evaluations carried out by the evaluator; a sample report to be enclosed);
  • Evaluator’s qualifications: Advanced university degree in evaluation/assessment, Civil Engineering, Sanitary/Environmental Engineering or in WASH related field;
  • Excellent analytical report writing skills (demonstrated through a sample report provided);
  • Excellent written and spoken English required (demonstrated through sample reports provided);
  • Good communication and presentation skills
  • Experience in project appraisal in towns and rural communities and a good understanding of context is an asset;
  • Familiarity with UNICEF corporate systems and programming processes is an asset;
  • Knowledge of the country context and WASH programmes in DPRK/East Asian region is an asset; 

8. Duty station and Official Travel Involved  

The evaluator is to include in the budget costs for travel to/from the DPR of Korea in relation to this formative evaluation including per-diems which will cover accommodation and food. 

9. Duration 

September 29th, 2016 – December 19th, 2016 (8 weeks level of effort within this 12 week time period) 

10. Performance Indicators 

Criteria for performance are quality of work, timeliness (ability to keep to strict deadlines), accuracy, initiative, responsibility, competence and communication.   

11. Estimated cost  

The estimated cost should be proposed by the evaluator. Budget should be derived based on the deliverables mentioned above and the required tasks for the formative evaluation. The evaluator will be selected based on the “best value for money” approach. The evaluator will be paid upon successful completion of assignments and submission of the deliverables in accordance with the following payment schedule: 

Stages

Percentage of payment 

Upon approval of the formative evaluation methodology

(inception report)

30%

Submission of the final reports, endorsed by UNICEF

70%

[1] Please see guidelines at http://mymande.org/content/how-design-and-manage-equity-focused-evaluations

[2] The inception phase will clarify the methodology and approach to be taken for this formative evaluation; depending on this there might be some changes to the contract initiated with the company.

[3] http://www.unicef.org/evaluation/files/UNICEF_Evaluation_Report_Standards.pdf 

[4] http://www.unicef.org/evaluation/files/UNICEF_Global_Evaluation___Report_Oversight_System_aFinal.pdf 


Opening Date 5 Sep 2016 Korea Standard Time
Closing Date 13 Sep 2016 Korea Standard Time


POSITION TYPE

ORGANIZATION TYPE

EXPERIENCE-LEVEL

DEGREE REQUIRED