End of Term Evaluation of the Environment and Natural Resources Management Project

United Nations Development Programme (UNDP)

, , MW

Background

UNDP has continued to support the Malawi (GoM) through UNDAF outcomes focusing on environment and energy for sustainable economic development.  The Malawi Growth and Development Strategy (MGDS) II 2011/12-2015/16 is the overarching operational medium-term strategy for Malawi. Its main objective remains to reduce poverty and achieve the Millennium Development Goals (MDGs). It is organized into six thematic areas: i) Sustainable Economic Growth; ii) Social Development iii) Social Support and Disaster Risk Management (DRM); iv) Infrastructure Development; v) Improved Governance and; vi) Cross-cutting issues of gender and capacity development.  Under MGDS I, Malawi has achieved macro-economic stability, economic growth, unprecedented poverty reduction, national food security and a 50 % reduction in HIV prevalence rates. 

Links to the National Development Plan.

During the 2012 -2016 cycle, under the MGDS theme 1 on sustainable economic empowerment, UNDP’s support focused on improved coordination, investment planning, mainstreaming and knowledge management at the national and district levels to ensure a low emission and climate-resilient development.   These objectives will be achieved by strengthening the policy environment, improving data and information management, and enhancing capacities for resource mobilization, coordination and monitoring of institutions responsible for climate change mitigation and adaptation, environment and natural resources management, disaster risk management and energy planning.   UNDP support to government under this theme was through four main projects, namely: National Climate Change (NCCP), Disaster Risk Management (DRM), Environment and Natural Resources Management; Sustainable Energy Management (SEM) and Poverty and Environment (PEI); but in this evaluation only the Environment and Natural Resources Management (ENRM) project will be evaluated. 

Environment and Natural Resources Management (ENRM).

ENRM: The Environment and Natural Resources Management Programme Support Document   highlights current and emerging ENRM issues reflected in MGDS II, UNDAF 2012-16 and UNDP CPD 2012-2016 and outlines a strategy to deal with them.  The programme prioritized activities under four broad themes, namely: Mainstreaming, Information & Knowledge Management, Coordination and Capacity Development at national and district levels.  In general, mainstreaming has been enhanced by support for the expeditious review, harmonization and enactment of appropriate ENRM legislation and policies and the development of institutional mechanisms at national and district levels.  Support for the development of strategic environmental and district land-use plans were expected to lead to an integration of conservation and development plans and devolution to districts to support implementation of these plans at District level.  To bolster Information and Knowledge Management, direct investments were expected to be made to improve data collection and management, research, periodic environmental data collection and the development of specific indicators and survey techniques. Further, development of best management guidelines and support to information centers to assist in the dissemination of data and information.  Better donor and government coordination, through the support of an ENRM sector working group and support to the setting up of a Climate Change & Environment & Natural Resources Management Sector Wide Approach (SWAp) will lead to more effective and efficient decision-making.  Efforts to enhance human capacity though the provision of training at tertiary institutions. 

Programmes and Projects Review.

Between May and August, 2016 the County Office embarked on a process to explore options for the configuration of the CO programme portfolio into 2017.   The exercise was a response to three main developments: 1) most programmes and projects in the current portfolio were designed to close in 2016, the final year of the 2012-2016 programming cycle; 2) new demands for projects to be aligned with UNDP Strategic Plan, core strengths of UNDP and reflective of programmatic approaches that avoid fragmentation and are innovative; 3) the realization that the current programme portfolio cannot be carried into 2016 due to limited core resources.

In the final analysis, among other conclusions, consideration will be given to: 

  • Prepare a redesigned DRM project which ensures strengthening preparedness, early recovery and resilience as well as supporting the young people. (Noting UNDP’s normative mandate and DRR’s special role in building resilience addressing preparedness)
  • Developing a larger access to energy programme as it has potential to enhance communities’ livelihoods by incorporating key elements of and lessons from the Sustainable Energy Management and Decentralized Energy Services projects;
  • Integrating ENRM and PEI and
  • Developing an overarching program that encompasses adaptation and mitigation.

The review also came up with three main lessons, namely: 1) to ensure appropriate institutional arrangements and thus avoid multiple layers of institutions and clarity on what would be the collaboration arrangements between national and local levels; 2) to build clear exit strategies and 3) the importance of appropriate level of leadership in steering committees to ensure availability of members at meetings.

 
Duties and Responsibilities

Evaluation Scope and Objectives.  

Scope:

  • Time period: January 2012- October, 2016
  • Geographical coverage: national.
  • Thematic coverage:   environment, climate change, disaster risk management and energy.

The evaluation will assess the performance of the project using the OECD evaluation criteria: relevance, effectiveness, efficiency and sustainability.

Based on the design consideration and implementation experience, the evaluation will identify priority interventions to be considered during the 2017-18 period and how they should be packaged.

Objectives:

  • Assess whether, and to what extent, the project’s outcome and outputs have been achieved;
  • Determine the impact, both positive and negative, as well as intended and non-intended from contribution of the project to the achievement of the outcome;
  • Examine and analyze factors which have positively and negatively impacted on achievement of project outputs and outcome;
  • Assess the effectiveness and appropriateness of institutional arrangements and partnership strategies of each project;  
  • Assess the role of the Ministry of Natural Resources Energy and Mines   in implementing the ENRM project; 
  • Assess the extent to which the UNDP-supported project outputs and non-project assistance contributed to the respective UNDAF and Country Programme outcome;
  • Examine the extent to which gender equality and women empowerment and human rights targets as cross-cutting issues were integrated and achieved;
  • Document lessons learnt and best practices during the course of implementation to inform future decisions in project design, implementation and management of similar interventions;.
  • Provide a framework for a large and coherent programme encompassing priority interventions in the areas of disaster risk management, climate change, energy and environment.

UNDAF Outcome:

All three projects fall under the UNDAF 1.2, namely:  Improved management of environment, natural resources and climate change for sustainable development at national and district level by 2016.  The UNDAF outcome was adopted verbatim as the UNDP Country Programme Outcome MWI-27.

Expected UNDAF Outputs:

  • Environment, natural resources, climate change, and disaster risk management mainstreamed in policies, development plans and programmes at national level and implemented in 14 disaster-prone districts;
  • Data and knowledge on the impact of climate change, environmental and natural resources degradation and natural disaster collected and made accessible to decision makers in Government, Private Sector and Civil Society;
  • Targeted population in selected districts benefit from effective management of environment, natural resources, climate change and disaster risk by 2016;
  • Output 1.3.4: Innovative renewable and energy saving technologies piloted in targeted locations in rural and peri-urban areas enabling the development of a national programme. (subsequently dropped).

Evaluation Purpose.
The purposes of the end of term evaluation are to:

  • Determine the extent to which the outcome and outputs of the three projects have been achieved;
  • Assess UNDP’s contribution to the outcome;
  • Document the achievements, best practices and lessons learnt during the course of implementation of the three projects to inform future decisions in design, implementation and management of similar projects.
  • Provide recommendations for future programming based on the results from the three projects while taking into account the aspirations of the Country Office to rationalize its portfolio to have few, large and more coherent projects during the period 2017 -2018.

The independent evaluation is to be commissioned from 1st December, 2016.

The main users of the evaluation results include:

  • The Programme Steering Committees for NCCP, Environment & Natural Resources Management, Poverty & Environment Initiative, Energy and Early Warning and Disaster Risk Management.
  • Ministry of Natural Resources, Energy and Mining;
  • Office of the President and Cabinet;
  • Environmental Affairs Department;
  • Department of Energy Affairs;
  • Department of Climate Changes Services and Meteorological Services;
  • Department of Water Services;
  • UNDP

Evaluation Criteria:

  • The evaluation will use standard evaluation criteria to assess its performance, which includes relevance, effectiveness, efficiency, impact and sustainability.

Evaluation questions:

  • In order to meet the objectives and purpose of the evaluation, the evaluators will among other tasks answer the following questions:

Design and Relevance:

  • Whether the problem the projects addressed is clearly identified and the approach soundly conceived;
  • Whether the target beneficiaries of the projects are clearly identified;
  • Whether the outcome and outputs of the projects were stated explicitly and precisely in verifiable terms with SMART indicators;
  • Whether the relationship between outcome, outputs, activities and inputs of the projects are logically articulated;
  • Whether the projects are relevant to the development priorities of the country;
  • Did the design of the projects take scale and scaling up into consideration;
  • Given the capacity building objectives of the projects, how effective were the projects’ capacity building interventions?

Implementation:

  • Whether the management arrangements of the projects were appropriate;
  • How effective was the delivery of inputs specified in the project documents, including selection of sub-grantees, institutional arrangements, identification of beneficiaries, scheduling of activities and actual implementation;
  • The fulfillment of the success criteria as outlined in the project document;
  • The responsiveness of the project management to significant changes in the environment in which the project functions (both facilitating or impeding project implementation);
  • Determine whether or not lessons learnt from other relevant programmes/projects were incorporated into the project.
  • The monitoring and backstopping of the projects as expected by the Government and UNDP;
  • The projects’ collaboration with industry, associations, private sector and civil society, if relevant.
  • The role of UNDP CO and its impact (positive and negative) on project delivery.

Efficiency:

  • Whether the projects resources (financial, physical and manpower) were adequate in terms of both quantity and quality;
  • Whether the projects resources are used effectively to produce planned results (Are the disbursements and project expenditures in line with expected budgetary plans)?
  • Whether the projects are cost-effective compared to similar interventions;
  • Whether the technologies selected (any innovations adopted, if any) were suitable;
  • Whether there is evidence to support accountability of the projects (to be used by UNDP in fulfilling its accountability obligations to its development partners); and
  • The delivery of Government counterpart inputs in terms of personnel, premises and equipment.

Effectiveness:

  • What are the major achievements of the project vis-à-vis its objectives, performance indicators and targets?  Please explain in detail in terms of impact, sustainability of results and contribution to capacity development.
  • Have there been any unplanned effects/results? 
  • Whether there is evidence of UNDP contribution to the outcomes of the projects.
  • What major factors affected project delivery and offer what appropriate interventions might have strengthened or addressed them.

Sustainability:

  • Assess whether or not the projects’ achievements are sustainable?
  • Is there an exit strategy for any of the elements of the programme?
  • What should be done to strengthen sustainability of project outcomes?
  • Assess whether or not the UNDP resource mobilization strategy for the project was appropriate and effective.

Evaluation Methods.

The evaluator should provide details in respect of:

  • Review of projects documentation. Review of key project documents such as approved project documents, recent studies, reviews, projects monitoring documents, disbursement reports, progress reports and other information available with implementing partners.
  • Construct a theory of change, identify detailed evaluation questions, methods (mixed methods) and instruments, stakeholder mapping, etc.
  • Data collection: (i) visits to selected stakeholders to carry out in depth interviews, inspection, and analysis of project activities; (ii) phone interviews and performance data surveys of institutions not visited in person; (iii) interviews with implementing partners.  For each of these interviews, the consultants should first develop and present their ideas for the content and format of the interview forms that will be applied to capture the information required, as well as the method to be used in administering them and tabulating the results.
  • Analysis: Data triangulation and analysis triangulation to validate evidence and arrive at findings.

The evaluator will be expected to develop and present detailed statement of evaluations methods/approaches in an inception report to show how each objective, evaluation question and criterion will be answered for all three projects.

Implementation Arrangements:

  • The Resilience and Sustainable Growth (RSG) Portfolio Manager will provide the overall oversight to the project evaluations and ensure timely delivery and satisfactory final products
  • A reference group will be established to assist in key aspects of the evaluation process including reviewing evaluation Terms of Reference, providing documents, providing detailed comments on the inception and draft evaluation reports and dissemination of evaluation findings, lessons learnt and recommendations.
  • The Programme Analysts responsible for Environment and Natural Resources Management, will support the evaluator on a daily basis with respect to providing background information and progress reports and other documentation, setting up stakeholder meetings and interviews, arrange field visits and coordinating with the IPs, grantees, beneficiaries and DPs.    The Programme Analysts will be supported by the UNDP M&E Specialist to ensure that the evaluation meets the expected UNDP standards. 
  • The evaluator will have the overall responsibility for the conduct of the evaluation exercise as well as quality and timely submission of reports (inception, draft, final etc).
  • The evaluator be expected to be fully self-sufficient in terms of office equipment and supplies, communication, accommodation and transport. Furthermore, the evaluators will be expected to familiarize themselves with the United Nations Evaluation Group’s standards and norms for conducting project evaluations.
  • The evaluator will provide the RSG Portfolio Manager with regular updates and feedback. 

Deliverables:

  • Inception reports – will be expected to be formulated within 5 days of the start of the assignment.  The reports will include a detailed approach and methodology, schedule, draft data collection protocols and an evaluation matrix.  The evaluator will propose a performance rating scale to be carried out for the four evaluation criteria: relevance, effectiveness, efficiency and sustainability.
  • Draft evaluation report – The Evaluator will present a Draft Report within 5 weeks after presentation of the inception report.
  • Lessons Learned reportthis will be a section within the evaluation report.
  • Final Evaluation Report. The evaluators will present a Final Evaluation Report 5 days after receiving feedback and comments on the draft report from key stake holders.

Time and Duration:

  • The evaluator will be hired for a maximum of 25 person/days.
  • Contract Start Date: 5 January, 2017.  Contract End Date: 28th February 2017.

Evaluation Ethics:

Responsibility of the CO to ensure credibility and independence of evaluation; responsibility of evaluator is to provide impartial, evidence-based, report adhering to international evaluation norms and standards, Code of Conduct etc.

 
Competencies

Evaluator’s competencies:

  • Organizational Development and Management
  • Strategic thinking
  • Team work skills; 
  • Result oriented;
  • Excellent communication skills.
 
Required Skills and Experience

Education Qualifications:

  • Minimum of a Master’s degree in economics, disaster risk management, energy, social sciences or environmental sciences. 

Experience:

  • Minimum of 7 years of professional experience in any of the areas of environment, energy, climate change and disaster risk management;
  • Of which minimum of 5 years’ experience in program development;
  • Experience in gender mainstreaming;
  • Experience in leading teams
  • Experience in conducting evaluations for UN agency, government or international aid agency projects in areas of energy, climate change, environment and disaster risk management.

Language:

  • Fluent in English.

Documents to be Included When Submitting the Proposals:

Interested individual consultants must submit the following documents/information to demonstrate their qualifications by 2nd December, 2016, before 1.30pm to the following email address: [email protected].

Technical Proposal specifying the two outputs of the Outcome to be assessed.

  • Explaining why they are the most suitable for the work.
  • Provide a brief methodology on how they will approach and conduct the work

Financial Proposal.

Personal CV (P11 Form) including past experience in similar projects and at least 3 references.

Proposals must include all three documents. Proposals not meeting this requirement will be rejected.

Financial Proposal:
Contracts based on daily fee.

The financial proposal will specify the daily fee, travel expenses and per diems quoted in separate line items, and payments are made to the Individual Contractor based on the number of days worked upon satisfactory completion of the required deliverable.

Travel:

  • All envisaged travel costs must be included in the financial proposal. This includes all travel to join duty station in Lilongwe /repatriation travel. 

Evaluation of Proposals:

Individual consultants will be evaluated based on the following methodologies:

Cumulative analysis:
When using this weighted scoring method, the award of the contract should be made to the individual consultant whose offer has been evaluated and determined as:

  • Responsive/compliant/acceptable; and
  • Having received the highest score out of a pre-determined set of weighted technical and financial criteria specific to the solicitation.

Technical Criteria weight; [70]

Financial Criteria weight; [30]

Criterial for Technical:

  • Criteria A: Master’s degree in economics, public administration, law or equivalent and 8 years of professional experience in the areas of economic and private sector development;
  • OR Bachelor’s degree in economics, public administration, law or equivalent and minimum of 15 years of professional experience in the areas of economic and private sector development – 5 points;
  • Criteria B: 8 years of professional experience in the areas of economic and private sector development OR 15 years of professional experience in the areas of economic and private sector development, for Master’s degree and Bachelor’s degree holders, respectively – 15 points;
  • Criteria C: Proven experience in gender mainstreaming or promoting gender equality – 10 points;
  • Criteria D: Experience in conducting evaluations for UN agency, government or international aid agency projects on economic and private sector development – 30 points;
  • Criteria E: Brief methodology on how they will approach and conduct the work in not more than 2 pages – 10 points.

Financial – 30 points
Combined Total Score (Maximum) – 100%.

Only candidates obtaining a minimum of 70 points in the Technical Evaluation would be considered for the Financial Evaluation
The financial score for the financial proposal will be calculated in the following manner:
Sf = 100 x Fm/F, in which Sf is the financial score, Fm is the lowest price and F the price of the proposal under consideration.
(Total Financial Maximum points = 100 points)

Total Score:

The technical score attained at by each proposal will be used in determining the Total score as follows:

  • The weights given to the technical and financial proposals are: T= 0.7, F=0.3;
  • The Total score will be calculated by formula: TS = St x 0.7 + Sf x 0.3;
  • TS – Is the total score of the proposal under consideration?
  • St – is technical score of the proposal under consideration.
  • Sf – is financial score of the proposal under consideration.

POSITION TYPE

ORGANIZATION TYPE

EXPERIENCE-LEVEL

DEGREE REQUIRED