External Independent Ex-Post Project Evaluator at Justice, Development and Peace Commission (JDPC)

Background

Caritas Germany (CG) as the lead, jointly with the Catholic Caritas Foundation of Nigeria (CCFN), the Justice Development and Peace Commission (JDPC) Maiduguri and the Catholic Agency for Overseas Development (CAFOD), operating as the Caritas Consortium have been implementing the multi-sectoral humanitarian action “Scaling-Up Assistance in Hard-to-Reach Areas” (SAHaRA) in Borno State. The project is funded by the European Union Civil Protection and Humanitarian Aid Operations (ECHO).

Since 2009, the Northeastern states of Nigeria have been affected by armed conflict and violence caused by insurgency activities of Boko Haram and counter military operations. Borno state has been the epicentre, leading to a high rate of displacement of 1.57 million people. Out of its total population of 5.6 million, 4.3 million are estimated to require humanitarian assistance. Around one million people in Borno are living in inaccessible areas and have no access to humanitarian assistance or basic services. The conflict has fuelled displacement, disruption of livelihoods, food insecurity, increasing rates of acute malnutrition and severely weakened basic services. A large number of women have become single heads of households as a result of separation or killings. The pervasive patriarchal norms continue to support unequal power relations. This context gives rise to high levels of GBV including intimate-partner violence.

The 10 villages targeted by this action are located in hard-to-reach areas in the Northeast of Borno in Kaga and Magumeri LGAs and consist on average of 180 households per village and in total a population of approximately 32,400 persons. An initial needs assessment highlighted significant needs in the areas of food security and WASH. Based on this, the primary project outcome was established, i.e. to provide direct access to integrated multi-sectoral, context-adapted assistance, which will cover the most urgent humanitarian needs of the most vulnerable people and communities in the locations in question.

Over the course of its implementation (May 2021 to February 2022), the intervention has reached over 1,300 households through unconditional cash transfers focusing on food security outcomes, as well as water, sanitation and hygiene (WASH) activities, including freshwater provision and hygiene promotion. A protection component focused on awareness-raising of sexual and gender-based violence (SGBV) and referral pathways for its victims complements the action.

As the ten-month project is coming to a close, the Caritas Consortium is commissioning an independent external evaluation to support donor accountability and organizational learning.

Evaluation Objective and Intended Audience

This evaluation aims to provide both learning as well as accountability through systematic assessment of the performance of the project in the sectoral areas it focuses on, capturing appropriateness, effectiveness, efficiency, impact and coverage of the intervention. The evaluation is supposed to present clear analytical insights, as well as concrete and actionable recommendations for future project phases to the consortium. In addition, it should provide insights for external stakeholders, i.e. the donor of the action, ECHO, as well as government institutions in Borno state.

Specific Objectives

Objective 1:   Evaluate to what extent SAHaRA Project has delivered effective, relevant and timely assistance to beneficiaries as set out in the proposal, with a particular focus on beneficiary satisfaction and accountability to affected populations. Highlight best practices and value added by the intervention and the consortium setup.

Objective 2:   Based on the findings, provide concrete and actionable recommendations on areas to improve on, on closing gaps in delivery and beneficiary satisfaction, and on improving implementation and consortium setup that can be incorporated into future program design.

Main Evaluation Criteria and Questions

The evaluation criteria and guiding questions are based on those established by the Active Learning Network for Accountability and Performance (ALNAP).[1] Out of ALNAP’s seven criteria, this evaluation will focus on six, taking into account local conditions in Borno state as well as the need for conciseness of the evaluation. The main evaluation questions are indicative and aim to provide the prospective evaluator with an overview of the evaluation’s supposed scope. More specific evaluation questions should be developed in the inception report, in collaboration with the project team.

1. Appropriateness: To what extent did the intervention objectives and design respond to beneficiaries’ humanitarian needs and priorities?

• To what extent did the assistance, particularly the cash component, meet immediate needs?
• To what extent did the project adapt and provide an appropriate response to context changes and emerging local needs, and the priorities of beneficiaries?
• Were proper accountability and risk management frameworks put in place to minimize risks on program implementation?
• How did the consortium setup support the appropriateness of the response?

 

1. Coverage: To what extent were the most vulnerable population groups reached by the intervention?

• To what extent have the cash transfers benefitted the most vulnerable households?
• How did the multiple sectoral components interact to target population groups with multiple vulnerabilities?
• To what extent were beneficiaries able to access the services and what barriers did they face to access assistance?

1. Effectiveness: To what extent has the intervention achieved its expected results, including any differential results across groups?

• To what extent did the action achieve its objectives, including the timely delivery of assistance?
• How have internal MEAL and quality assurance mechanisms been utilized to increase quality within the project?
• How has the consortium setup contributed to the achievement of project results?

1. Efficiency: To what extent has the intervention delivered results in an economic and timely way?

• How cost-efficient was the project, and its individual sectoral components?
• What opportunities exist within project to reach more beneficiaries with the available budget or to reduce costs while reaching at least the same number of beneficiaries without compromising quality?
• How quickly and effectively were protection issues addressed?
• Could any of the goods and services provided to the beneficiaries be more efficiently delivered through a cash modality?
• How has the consortium setup contributed to the cost and speed of implementation?

1. Impact: To what extent has the intervention generated significant positive or negative, intended or unintended, higher-level effects?

• Have the activities implemented improved beneficiaries’ lives?
• How did HHs receiving multiple sectoral interventions improve their food security status compared to those that received less support?
• To what extent has the program contributed to increasing market functionality and vendor activity in the intervention zone?
• To what extent did the project interventions contribute to build long-term community capacity?

1. Coordination: To what extent was the project communicated and coordinated with relevant stakeholders?

• To what degree have system-wide as well as sectoral standards (e.g. Core Humanitarian Sphere standards) been respected in the implementation of the action.
• To what extent did the consortium engage with other humanitarian or government stakeholders and how did this benefit the action?

cope of the Evaluation

The evaluation encompasses the entire project duration from May 2021 to February 2022 in all project locations in Kaga and Magumeri LGAs (through representative sampling). The evaluation should cover the project’s three sectoral results and their associated indicators as well as the key outcome indicators of the project. The project’s transfer modality (cash) should also be subject of the evaluation with a focus on how the modality was adopted as well as the modality’s advantages and disadvantages in the local context. All target groups should be involved in the evaluation. The evaluation should also include a comprehensive analysis of how the intervention context has changed over time and how this has affected project participants.

Methodology

The evaluator is required to apply an adequate evaluation design and mixed methodology approach including participatory methods[2] to produce verifiable information for answering the above-listed evaluation questions, make recommendations that are sufficiently valid and reliably based on dependable, triangulated data, analysis and interpretation. The collection of gender/age-disaggregated data should allow for an adequate presentation of how men/boys and women/girls are impacted by the project. Any interview activities should take into account local cultural and religious norms, e.g. utilizing female enumerators to conduct interviews with women and girls. The quantitative data collection should take place before carrying out FGDs and KIIs. This is so that the insights from the quantitative survey can inform qualitative data collection. A detailed design and methodology for the evaluation will be proposed by the evaluator in the inception report (see deliverables). The evaluator will report to the evaluation manager as well as the evaluation commissioners (i.e. Caritas Germany). The evaluator will execute his/her mission in complete independence and will receive only general instructions from Caritas Germany, justified by the necessities of the collaboration between the parties and the orderly execution of the confined tasks.

The choice of exact approach/methodology is the responsibility of the consultant and subject to approval by the evaluation manager and the commissioners. Suggested evaluation methods to be used may include, but are not limited to: 1) Review of project documentation (including baselines, endlines, monitoring reports, data stemming from the project feedback and complaints mechanism, as well as secondary data like statistical data, documents pertaining to similar projects led by other donor organizations, etc.). Existing project documents and reports will be shared with the evaluator. The evaluator is invited to request additional documents that may be needed for the completion of the evaluation. 2) Key Informant Interviews (KIIs). Including (semi-) structured interviews with beneficiaries is a must and should be documented through interview protocols and/or recordings and the list of respondents (e.g. direct project beneficiaries, project staff, senior management, etc.). The selection criteria should be based on gender, age, beneficiaries/non-beneficiaries and geographical spread in the project region, as well as representative of the project consortium. 3) Direct observation during field visits. The Caritas Consortium will organize field visits to different project sites. 4) Focus Group Discussions (FGDs) with beneficiaries, local leaders, and project staff. Similar to KIIs, these should be documented through interview protocols and/or recordings and a list of participants. 5) Standardized surveys and questionnaires to quantify results and conduct statistical analysis. The selection criteria should be based on gender, age, beneficiaries/non-beneficiaries and geographical spread in the project region. The exact sample size should be proposed by the evaluator and discussed with the evaluation commissioners. 6) Participatory Impact Assessment. It is required that the Evaluators apply participatory approaches in their data collection, to be found in the Feinstein International Center Practitioner’s Guide on Participatory Impact Assessment. [3]  Methods that allow impact measurement and reflection for illiterate people, e.g. ranking, scoring, rich picture, narratives, stories, timelines etc. must be considered.

Deliverables

The consultant is expected to provide the following deliverables within the agreed timeframe and budget:

• An inception report, which will serve as an agreement between parties on how the evaluation will be conducted. Items to address include:
  • Understanding of the issues and questions raised in the ToR
  • Data sources; how to assess the questions in the ToR
  • Research methodology, including suggested sample and size
  • Schedule of activities and traveling (timeline)
  • Detailed budget

• Appropriate draft data collection tools (e.g. methodological guidelines, FGD and KII interview questions). These tools will be validated by the evaluation manager and Caritas Germany and then finalized for field activity
• Raw data in any of the following statistical packages (Excel, STATA, SPSS, CSPro) and also transcribed qualitative scripts
• Powerpoint presentation of preliminary findings
• Organization of a learning session involving the project team
• A max. 20-page draft (MS Office and PDF for final) in English
• A max. 50-page final evaluation report (in MS Office and PDF for final), excluding annexes and methodology, in English. The final evaluation should contain appropriate graphs, visuals, tables and/or a dashboard with explanatory texts
• An Itemized price quotation for consultancy fee (exclusive of international and domestic travel expenses which will be provided separately)

The final evaluation report should consist of:

• Executive Summary (max. 3 pages)
•  Introduction
• Methodology (including sampling and limitations)
• Analysis and findings of the evaluation (structure to be agreed upon with evaluation manager and the evaluation commissioners)
• Address concerns, lessons learned and comments from the consortium team
• Conclusions for each evaluation criterion
•  Recommendations

8) Annexes:

• Stories of change and quotes from respondents
• Methodology, including sampling and limitations
• Relevant maps and photographs of the evaluation areas
• Bibliography of consulted secondary sources
• Finalized data collection tools
• Powerpoint presentation of preliminary findings
• Powerpoint presentation of executive summary of final evaluation report

Indicative Timescales

The data collection phase in the field is to be confirmed between the consultant and the evaluation manager and commissioners.

Phase	Deliverables	Payment	Working Days Indication	Tentative Dates (TBD)
Inception phase	Draft inception report		2	March 1 -2
Tools Development Phase	Deliverable 1: final inception report including budget, methodology, qualitative and quantitative research tools, approved by the evaluation manager and commissioners	30% of total budget	5	March 3 -7
Data Collection Phase	Desk review of project data		3	March 8 - 10
	Interviews (FGDs, KIIs) during field visits, collection of quantitative data		8	March 11 - 18
Data Analyses Phase	Data analysis and powerpoint presentation of initial findings to the program team	30% of total budget	3	March 19 - 21
Report Writing Phase	Draft evaluation report, for comment by evaluation manager, commissioners, and project team		7	March 22 - 28
	Deliverable 2: learning session with team		1	March 29
	Deliverable 3: final evaluation report approved by Caritas Germany	40% of total budget	2	March 30 – March 31
Total		100%	31 days

Location and official travel involved

The selected evaluator should be available to travel to field locations in Magumeri and Kaga LGAs of Borno State to conduct the field work; including training of enumerators and data collection. The service provider may work remotely for desk research, report writing and revisions following the project team’s review. Office space will be provided but the consultant is responsible for bringing his/her own IT equipment. Any required travel will need pre-approval by the Consortium’s security team and will be conducted in accordance with CG’s travel policy and applicable expense policy.

Professional Qualification and Experience

Required

• Minimum Master’s degree in Economics, Political Science, Business Management, Development, Humanitarian Affairs, or a related field
• At least 7 years’ experience in working with humanitarian multi-sectoral programs relating to Food Security, WASH and Protection
• Experience with quantitative and qualitative research, database management and statistical data analysis
• Experience evaluating FSL, WASH and Protection Programs
• Proven track record of communicating with beneficiaries
• Ability to assess and further develop a conceptual evaluation tool
• Ability to carry out field work in remote rural locations
• Commitment to ethical working practices

Preferred

• Experience of working in insecure environments in Northeast Nigeria
• Proven track record conducting Participative Impact Assessments

Guiding Principles and Values

The consultant is to conduct the evaluation in accordance with the principles outlined in the “Caritas Internationalis Management Standards,”[4] the “Caritas Code of Conduct”,[5]  as well as the “Guidelines on Combating Fraud and Corruption in the Project Work of Caritas Germany.” [6]

The evaluator(s) must take all required steps to ensure that the evaluation is designed and conducted to respect and protect the rights and welfare of the people and the communities of which they are members, as well as to ensure that the evaluation is technically accurate, reliable, and legitimate, and conducted in a transparent and impartial manner.

All documents and data acquired from documents as well as during interviews and meetings are confidential and to be used solely for the purpose of the evaluation. Interview partners will not be quoted in the report without their prior permission. The documentation as well as all material linked to the evaluation (produced by the consultants or the organisation itself) is confidential and remains at all times in the property of the contracting party.

Selection process

Expressions of interest will be accepted by individual consultants as well as from commercial companies, NGOs or academics. Interested candidates should submit their application material by 25/2/2022.

The application itself should include the following components:

• Curriculum Vitae (CV) including 3 professional references
• Cover letter transparently summarizing relevant experience
• Track record of conducted studies, research, publication and references
• A technical proposal for the evaluation, including the proposed evaluation methodology (including reference to the perceived feasibility of the ToR) and a work plan (max. 3 pages).
• Financial proposal, including proposed budget for the evaluation, a fee per working day (plus the respective VAT, if applicable) for a maximum of 31 working days and other applicable costs.
• A sample of recent writing (report or similar) relevant to the terms stated above
• Proof of professional registration and taxation
• If the evaluation is carried out by a team, CVs of each person to be involved in the assignment

CG will use its selection criteria, checklists and an interview process to select the successful consultant. All applications received will be assessed on the basis of cost-efficiency.