Consultant - Evaluation of the ECHO HIP Humanitarian Aid Operation at Save the Children Nigeria

Length of Assignment: 8 weeks
Dates of Assignment: February 1st - March 26th, 2018.
Reporting to: Head, Monitoring, Evaluation & Learning and Humanitarian MEAL Adviser
Key Working Contacts: Project Manager, ECHO, MEAL Coordinator, Humanitarian PDQ Manager
Project: ECHO HIP

Project Summary

• Save the Children provided food assistance to vulnerable households in Jere and Mafa LGAs (where ECHO 3 was implemented), as many households in these communities remain vulnerable to food insecurity and engaging in negative coping strategies which could put children at risk. This food assistance was provided for 6 months (during the lean season) in Mafa and Jere LGAs during the lean season. Unconditional transfers were provided as vouchers which were exchanged for food items at participating vendors. Beneficiaries identified within the selected communities based on the vulnerability criteria.
• In order to address critical levels of malnutrition in children, SCI will support 7 existing outpatient therapeutic feeding programmes (OTPs) and 4 mobile outreach units in Jere and Konduga, to meet the continuing need for treatment for severe acute malnutrition (SAM) for children between ages of 6-59 months.
• Alongside reducing the risks to children’s health and wellbeing posed by food and nutrition insecurity, Save the Children worked to deliver and strengthen child protection services through:
• Providing psychosocial support through Child and Adolescent Friendly Spaces (CFS)
• Strengthening community-based protection mechanisms including Child Protection Committees (CPCs) and engaging and training a cadre of Community-based Case Workers
• Improving access to and quality of child protection services through identifying and providing case management services for unaccompanied and separated children (UASC) and other children at risk.
• The project is implemented through partnership with local CSOs (EYN and GEPaDC). Additionally, Save the Children worked closely with the State Ministry of Women Affairs and Social Development (MWASD) to strengthen their ability to fulfill their mandate to provide child protection services.

Purpose and Scope of the Evaluation

• Save the Children Nigeria Country Office will commission this end of project evaluation. Purpose of this evaluation is to assess both the processes and the outcomes of project and collect evidence and lessons learnt to present to the Project and Country Programme team to feed into further programming.
• The primary objective of this evaluation is to assess the overall achievements of the project in line with the agreed project results and objectives.

Specific Objectives
More specifically, implementation of this evaluation will aim for the following key specific objectives:

Specifically, the evaluation will look into:

• The extent to which SCI has been successful in meeting the project goal and objectives.
• To gather beneficiary feedback on project achievements.
• To synthesize the secondary data/reports produced by project and suggest recommendations to progress on core indicators in the target communities
• To produce SWOT analyses of the sectoral integration aspect of the project
• Review and compare achievements during ECHO 4 as compared to ECHO 3 funding?
• Document lessons learned and best practices
• The evaluation will focus primarily on all Results 1, 2 and 3 of the project log frame. The evaluation team is expected to conduct a participatory evaluation ensuring meaningful and ethical participation of beneficiaries and local partners/duty bearers.

Project Objectives

• Objective 1: Vulnerable HH receive food of sufficient quality and quantity to ensure immediate food needs are met
• Objective 2: Increase access to treatment of SAM for children under 5 and improved infant child feeding practices emergencies among PLW and care givers of children to prevent malnutrition
• Objective 3: Children and families have improved access to timely and quality child protection services in line with child protection minimum standard

Evaluation criteria and key questions
Evaluation study will explore the following key areas and questions based on the most relevant of the OECD-DAC criteria for evaluating humanitarian action and some additional areas which SCI would like to explore in depth:
Relevance/Appropriateness:

• To what extent the interventions address the needs of beneficiaries?
• Were the activities/output consistent with the needs of the affected community as agreed at baseline study?
• What was the progress on key achievements in line with the donor’s strategy and geographical focus areas for Nigerian humanitarian response in 2017? (Review of the progress as compared to the ECHO compliance standards)
• Were the interventions per sectoral planned and implemented in an integrated manner?
• Were communication and coordination between sectors regarding implementation secured?
• Are there overlaps of interventions in sectors (geographical or beneficiaries)? What was its impact?

Coverage, and non-discrimination:

• Did the project reach to the most vulnerable children affected by the crisis? Did the project reach out to children at risk of exclusion from service provision as identified in the baseline study? If so, how?
• How sensitive was the programme interventions to address the inequalities due to gender, race and age?

Effectiveness:

• Were the activities implemented as planned, achieving intended outcomes and objectives within the planned budget and timeline? Did the intervention meet international quality standards (Sphere, including Child Protection Minimum Standards (CPMS), HAP, as appropriate, Core Humanitarian Standards)?
• What were the key underlying factors influencing the achievement or non-achievement of specific objective and overall goal of the project?
• How effective were the project intervention to address key bottlenecks to ensure access to interventions to treat SAM cases.
• Document effective approaches which influenced community behavior to improve malnutrition status of children.
• How effective were the interventions to address the inequality in SAM treatment on the basis of gender, income and social status.
• How effective were Child Protection Committees (CPC) in identifying and appropriately referring child protection cases? What barriers to their effectiveness have been addressed and which ones still exist?
• How effective were Child Friendly Spaces (CFS) in providing age/ gender sensitive and culturally appropriate activities as well as psychosocial support and referral services?
• Did the intervention cause a change in the lives of the affected population vis-à-vis the project objectives?

More specifically focus on the following list of indicators where data source is endline evaluation:

• Percentage of the target population with acceptable Food Consumption Score (FCS)
• Percentage of target population that have a decreased Coping Strategy Index (CSI) score
• Percentage of PLW and/or caregivers of children under 2 who take part in mother to mother support groups who know at least 3 good practices in IYCF-E
• Percentage of CPC members reported to have actively identified, reported, and referred child protection cases
• Number of participants showing an increased knowledge on protection subject in focus

Child outcomes:

• Was there a sufficient child-focus in the project design and implementation?
• Were the intended outcomes of this project specifically for children met?
• Were any unintended child outcomes observed from cash transfer and other FSL/NFI/shelter activities under this project (e.g increase in school attendance; preventing children from being placed in institutions; reduction of the stress caregivers feel in meeting survival needs; reduction of the physical and verbal punishment of children; decreasing stress in children; improvements in children’s wellbeing)

Accountability:

• How were the communities engaged in the whole project cycle? Did the project encourage the genuine and active participation of children and communities?
• How accountable has the implementation been in terms of addressing the needs of children and their families?
• How effective is the accountability mechanisms set in place during implementation? Was the project resolved identified or perceived issues, conflicts, or complaints in the community through the mechanisms?

Efficiency:

• Assess the activities are cost-efficient within the programmes supported by project?

Impact:

• What are the key short term and long term changes produced by project, positive or negative and what are the key factors behind these changes?
• What are the key intermediate and lasting changes lead by the project interventions?
• Were there any unintended results of the intervention and how did it affect the outcomes?

Sustainability:

• To what extent did the benefits of the projects continue after the fund support is ceased?
• What will be the structures at community level and health facility level to identify, screen and treat malnutrition
• What will be the structures at community level to identify protection issues? Will the CPCs be continuing their activities?
• Has there been an exit strategy implemented

Lessons Learned:

• What were the good practices and challenges in the project? How did it able to affect implementation and project delivery?
• Were the recommendations on the ECHO 3 evaluation taken into account in the implementation on the current project?

Evaluation design and methodology
The evaluation would involve a combination of desk review and field based data collection. The detailed evaluation design is to be determined by the evaluator in agreement with the Monitoring, Evaluation, Accountability and Learning (MEAL) Manager. The following guidance should be taken into account:

• There should be meaningful and safe participation of children and young people, ensuring ethical standards for research with children are followed, including informed consent and assent, ensuring the best interest of the child, confidentiality and avoiding harm.
• Focus group discussions, key informant interviews, and lessons learned and validation workshops are encouraged.

The following data analysis is expected for the consultant to deliver:

• Baseline vs. Endline
• Evaluation Results vis-à-vis the ToR questions
• Evaluation Results vis-à-vis the logframe
• Gender and locale disaggregation

Profile and responsibilities of evaluator
The external evaluator/team must have experience and significant knowledge of humanitarian response mechanisms, specifically regarding the North east humanitarian response, and have previous experience in conducting external evaluations for conflict responses.

The evaluation team must be external to all stakeholders of the project. In addition, the following characteristics are highly desirable:

• Significant experience (5-8 years) in coordination, implementation, monitoring and evaluation of emergency response programmes AND expertise in at least 1 relevant sector (Food Security & Livelihoods, Nutrition, Child Protection)
• Demonstrated experience of evaluating humanitarian response projects is essential (experience of evaluating ECHO-funded projects is desirable)
• Knowledge and experience in child participatory approaches and child safeguarding
• Fluency in English and ability to write clear and concise reports (examples of previous work will be requested)
• Understanding of ECHO requirements as a humanitarian donors and specific requirements for this evaluation.
• Ability to manage the available time and resources and work to tight deadlines.
• The primary point of contact within the Humanitarian response will be the MEAL Adviser. Though the evaluator/team will be operating independently, field movements will always be in coordination with SCI Borno’s s safety and security team for access to implementation areas in scheduling of KII and FGDs with the community. SCI will make additional staff available to support the evaluation where required, for example for community entry support.

Deliverables
The evaluator/team is expected to provide a number of deliverables. These deliverables are to be submitted to SCI in both electronic and hard copy, as described below.

• Inception report according to the standard template, including detailed work plan (Please see the template proposed in the tools). Also, once TAs will review and sign off the inception report then consultant will start the data collection.
• Draft evaluation tools (interview guides) for the evaluation for approval by SCI.Draft and final evaluation reports.
• The final report, which should not exceed 25 pages excluding annexes, should be written in English and will be shared with ECHO as well as other stakeholders.
• Complete Logframe with the actual achievements disaggregated (by gender and location).

The suggested evaluation report format is:

• Cover Page
• Table of Contents
• List of Acronyms
• Executive Summary: must be a standalone summary, describing the Project, main findings of the evaluation, and conclusions and recommendations. This will be no more than 3 pages in length.
• Main Body: The main body of the report shall elaborate the points listed in the Executive Summary. It will include the following sub-sections:
  • Background Information
  • Methodology
  • Evaluation Challenges and Limitations
  • Main Findings & Analysis of Project Results
  • Identified Good Practices and Lessons Learned
  • Conclusions and Recommendations
  • Annexes (including list of stakeholders consulted for the evaluation, research tools, etc.)
• PowerPoint Presentation of key evaluation findings. Should the evaluator conduct any presentations as part of the evaluation, he/she should submit electronic copies to SC before the end of the evaluation, preferably at the time of the presentation.
• All documentation including raw data gathered during the evaluation should be handed to SCI before the evaluator/team’s work is certified complete.

Time-Frames:

• Time frame for the activities will be: February 1st – March 26th 2018

Plan for dissemination and learning:

• The evaluation report will be shared with ECHO as well as other external stakeholders.
• A validation workshop to provide feedback on the result of the evaluation should be conducted with stakeholders including beneficiaries (if feasible). A debrief session will also be scheduled with the SCI Borno project team.
• All data gathered during the evaluation and draft and final documents will remain the property of SCI. Should the evaluator wish to publish or disseminate all or part of the final evaluation, presentations conducted during the evaluation pertaining to the evaluation, communication materials, and/or other evaluation-related documents, request from SCI should be requested first.

Budget:

• Consultant should be able to take into account on their budget all human resources costs and transportation. Interested applicants must bid competitively for the works and include a detailed budget of all costs associated with the deliverable. Note that all laptops, software, and communications equipment should be provided by the evaluator/team. SCI will provide logistical arrangement during field visits. All other costs associated with the consultancy must be included in the bid.

Consultant’s fee and proposed mode of payment:

• 30% upon signature of contract and submission of inception report
• 40% upon submission of draft report
• 30 % upon submission of final evaluation report.

SCI reserves the right to withhold payment if deliverables do not meet the requirements outlined in this Terms of Reference.

Submission and evaluation of proposals
Proposals will be evaluated according to the following criteria:

• How well the requirements laid out in the ToR are met
• Skills and experience of the evaluator/team
• Cost effectiveness