Country-level HPP Evaluation

About

This is the final evaluation of the Irish Aid three-year Humanitarian Programme Plan (HPP) in Lebanon (2019-2021), which was lately extended by one year, to 2022. This evaluation is part of the formal evaluation of the overall Irish Aid HPP being implemented in nine countries, one of which is Lebanon, by a team of external consultants working at a global level ¾ herein referred to as the global consultants. The findings and final report from this evaluation will ultimately be used by the global consultants to assess the overall Irish Aid HPP against the Organisation for Economic Co-operation and Development-Development Assistance Committee (OECD-DAC) criteria, as well as to produce an all-inclusive global report, bringing together key learnings from both the Irish Aid Programme Grant (IAPG) and HPP programmes to provide evidence-based recommendations for the future, for both Concern (as implementer) and Irish Aid (as donor). The final evaluation (HPP report) will also be used by the country team for learning purposes, feeding into the design of forthcoming programmes, Irish Aid’s in particular, from 2023 onwards.

Context

Lebanon has been enduring ¾ as of late 2019 ¾ an incomparable economic crisis that continues and now threatens the subsistence capacity of its people. Individuals and families have fallen deeper into poverty due to skyrocketing inflation rates, rising food prices, and loss of employment and income. The Covid-19 pandemic has exacerbated the situation even more, reducing (among others) people’s access to food, livelihood opportunities and other basic services. The pandemic has had a disproportionate impact on women and girls in particular. As per (GBVIMS, 2020), recurrent and prolonged lockdowns could eventually lead to increased risk of violence as a result of mounting tensions within households when families are confined to their homes, as well as the dire economic situation causing more stress and anxiety for many. The latter statement somewhat explains the recent-year rise in the number of recorded Gender-Based Violence (GBV)[1] and Intimate Partner Violence (IPV)[2] cases in Lebanon. As well, the country’s worsening economic situation, coupled with limited access to education, has lately ensued in an upsurge of reported cases of child labour and child marriage[3].

Status quo, domestic violence, sexual harassment and exploitation ¾ as well as child, early and forced marriage, remain the chief protection concerns for Syrian refugee and vulnerable Lebanese women and girls in Lebanon. When it comes to Syrian men and boys, they have been reportedly experiencing stress and feeling powerless due to their inability to fulfil their traditional roles as family providers and protectors. That said, the programme’s main aim was to support Lebanon’s Syrian refugees and vulnerable host communities in a serious attempt to reducing GBV and Child Protection (CP) risks to which they are exposed and to increasing their resilience vis-à-vis crises.

 

Subsequently, following the Beirut explosion on August 4, 2020 ¾ Concern’s HPP programme has been directly expanded to incorporate an additional emergency response allocation. In view of that, Concern has proceeded by providing financial support to Micro, Small and Medium Enterprises (MSME) in Karantina, Bourj Hammoud, Geitawi, and Port area (being the regions hit hardest by the blast). In doing so, Concern aimed at speeding MSME’s business recovery up, boosting livelihoods and creating short-term employment or Cash for Work (CfW) opportunities for the most vulnerable populations ¾ Syrian refugees and other migrants. Overall, Concern has provided 25 MSME with grant amounts ranging between USD1,500 and USD10,000.

 

Outcomes             

Nearly three years have passed since Concern (and Himaya, Concern’s local partner since 2020) has started implementing Protection, CP and GBV activities under HPP ¾ targeting the most vulnerable populations (refugee and host communities) ¾ with the aim to building safer and more resilient communities in Akkar and T5 (including Tripoli and five surrounding districts). Over the course of the programme, Concern and Himaya made sure to exchange relevant information and build on one another’s strengths, a crucial thing to ensuring the sustainability of the programme and the adequate implementation of its activities, explicitly: the provision of Engaging Men Programme (targeting men, women and children), Positive Parenting sessions, Case Management (CM) for CP and GBV cases, as well as referral services.

In 2020, Concern’s partnership with Himaya aimed at building their capacity to implementing integrated, family-based interventions especially in relation to Engaging Men, as a means to preventing GBV. Likewise, Concern was capacitated on best ways to delivering Focused Psychosocial Support (FPSS) sessions for children at risk of child marriage and child labour.

 

The outcomes of this programme (alongside their associated indicators) are various, six in total, listed below:

 

Intended Outcomes

Indicators

1. Male Syrian refugees and host community members promote safety in their communities through positive attitudes and behaviour

• Average score on scale relating to Gender Role Attitudes among male Syrian refugees and host community members
• Average score on knowledge of key course topics (Adult)
• Average score on attitudes relating to key contextual protection issues (IPV, GBV, other) among male Syrian refugees and host community members
• Average psychosocial (personal, interpersonal and capacity) wellbeing among male Syrian refugees and host community members

1. Women and female youths are empowered to reject violence and demand their rights

• Average score on scale relating to Gender Role Attitudes among women and female youths
• Average score on knowledge of key course topics (Adult)
• Average score on attitudes relating to key contextual protection issues (IPV, GBV, other) among female Syrian refugees and host community members
• Average psychosocial (personal, interpersonal and capacity) wellbeing among female Syrian refugees and host community members

1. Children are empowered to recognise protection risks and to respond appropriately to dangerous situations

• Average score on scale relating to Children's Gender Role Attitudes
• Percentage of children (of those participants who only took part in Engaging Men sessions) who have experienced nonviolent discipline, psychological aggression, and physical discipline from caregivers in the home setting within the past month
• Percentage of children (of those participants who took part in both Positive Parenting and Engaging Men sessions) who have experienced nonviolent discipline, psychological aggression, and physical discipline from caregivers in the home setting within the past month
• Average score on knowledge of key course topics (Children)
• Average score on the strengths and difficulties questionnaire

1. Individuals (families) at risk have access to vital services and information, including case management

• Percentage of cases closed (Case Management)
• Percentage of participants in awareness-raising sessions on services and rights who reported an increased understanding of the subject matter covered during session

1. Boys and girls at risk and survivors of violence, exploitation and abuse have access to an improved and equitable prevention and response

• Number of children who have access to FPSS sessions
• Percentage of children at-risk of child labour and child marriage with enhanced knowledge of CP and life skills
• Average score on the Child and Youth Personal Resilience Measure (CYRM)

1. Improved resilience for MSME, Syrian refugees and migrant workers in areas affected by the Beirut Port Explosion

• Percentage of MSME operating business as usual three months after receiving a grant
• Number of days Syrian refugees and other migrant workers employed by MSME on CfW basis
• Percentage of affected Syrians and other migrants receiving CfW reporting immediate needs met

 

 

Evaluation: Criteria & Questions

This final evaluation, to be carried out by the (successful) consultant (in close collaboration with the global consultants), should focus on the design, implementation, activities and results ¾ covering the time period from 2019 to Q1 2021 ¾ and offer concrete and actionable recommendations (in line with the intended outcomes) for improving programming going forward.

 

Evaluation questions should be based on the extended OECD-DAC criteria (see below). Additional country-specific questions could be added when and as needed in order to ensure this evaluation would end up being useful as a learning tool for country teams. The number of evaluation questions has to be reasonable and in line with the scope and objectives of the evaluation, and the expected length of the final report of findings (25 pages in total, of which 15 focused on findings and conclusions against the extended OECD-DAC criteria). 

 

OECD-DAC

Evaluation questions

Indications

Relevance

Were the objectives and design of the intervention relevant and appropriate to the context and needs of recipients?

• Clear rationale for the selected intervention (Mandated by Irish Aid)
• The intervention was aligned with the needs of communities and-or people targeted
• Adaptation in response to arising changes in context and lessons learnt (Mandated by Irish Aid)

Effectiveness

Is there evidence that anticipated results are being achieved at the current stage of the programme?

• Changes as a result of the programme (Mandated by Irish Aid)

Is the programme achieving the planned outputs and outcomes and are they on time?

• Can those changes be attributed to HPP funding? If so, to what extent? (Mandated by Irish Aid)
• Concrete evidence justifying that the anticipated results are being achieved at the current stage of the programme
• The intervention was effective and delivered in a timely manner (in the eyes of communities and-or people targeted)
• The response team coordinated effectively with external stakeholders

Efficiency

Did the intervention use the available resources in the most efficient manner to achieve its outputs?

• Sufficient and appropriate use of resources
• Paying competitive prices for inputs
• Alternatives for achieving the same results with fewer inputs

Impact

Did the intervention achieve the intended impacts?

• Evidence of change at impact level, positive or negative
• Wider effects of the programme on individuals, groups, communities, institutions, other
• Unintended impacts, positive or negative

Connectedness

How were long-term problems given consideration during implementation of emergency activities?

• Link between HPP and IAPG (Mandated by Irish Aid)
• Country team’s capacity to ensure synergy between humanitarian and development programming
• Communities supported and capacitated to better deal with future emergencies (Mandated by Irish Aid)

Coherence

Was there consistency between Concern’s activities and the relevant policies?

• Intervention aligned with the fundamental humanitarian principles
• Adherence to the Red Cross Code of Conduct Sphere, Core Humanitarian Standard, and other sector-specific standards
• Evidence that the intervention adhered to relevant Concern’s policies

Coverage

Is there evidence that Concern’s intervention reached the most vulnerable?

• How the programme targeted those in greatest need?
• How the programme accounted for People with Special Needs (PWSN)?
• Linkage between conflict and gender analysis (on one hand) and project design and implementation (on the other hand)
• Efforts made to ensure all programme participants, direct and indirect, are safe throughout

Additional cross-cutting themes

 

• Potential linkages between this intervention and other interventions which were also planned and implemented by Concern (related to the HPP)
• Covid-19 influence on the intervention in general

 

Evaluation: Methodology

The (successful) consultant would be in charge of the entire evaluation process, with access to a focal point in Concern’s Head Office in Dublin. Besides working in close collaboration with the global-level evaluation team (already appointed), the consultant would be fully supported by Concern’s team in Lebanon, too. For this final evaluation, it is highly recommended that the consultant opts for a mixed methodology, combining primary and secondary data, quantitative and qualitative, that would allow him (her) to sketch a conceptual framework for the HPP intervention, deliberate with and gather needed feedback from beneficiaries (direct and indirect) and relevant stakeholders (local governments, partners, staff, other), and assess the overall impact of the programme. The prescribed methodology includes the following activities:

 

• Detailed programme document review including Country Context Analysis (if applicable); HPP 2019-2022 Programme of Work; HPP Results Frameworks 2019-2022; Report of HPP2018 Final evaluation
• Assessment of achievement of indicators under each outcome using data from Concern Digital Data Gathering (DDG) system
• Conduct Key Informant Interviews (KII) with key programme staff e.g. partner staff, relevant local authorities, key programme participants and community leaders, as well as representatives of various sectors
• Conduct Focus Group Discussions (FGD) with programme participants (appropriately disaggregated). Where FDG are not possible, remote data collection with programme participants should be considered (to be mentioned as part of the inception report)
• Data triangulation as a method to cross-checking existing data gathered from various sources (e.g. 2020 HPP Evaluation Report, and Baseline and Endline surveys carried out from 2019 onward) in an attempt to search for regularities in and to enhance the credibility of the evaluation findings.

 

The consultant must attend a standard training on the prescribed methodology, which would include a package of tools to be used in the course of this evaluation and would be delivered jointly by Programme Approaches and Learning Unit (PALU) and the global-level evaluation team. This is to ensure a standard approach is followed by all country-level evaluators to allow comparison and aggregation across countries by the global consultants in producing a final global report. Moving on, the consultant is expected to develop an appropriate, participatory work plan for the said evaluation ¾  one that is based on the global evaluation approach and relevant to the country context (Lebanon).

Evaluation: Process & Outputs

The consultant will use the following evaluation process to generate the following outputs:

 

Process

Expected Outputs

• Attend e-information session on prescribed methodology and get acquainted with the tools to be used. Data collection tools will be designed by the global consultants team

N/A

• Meeting either virtually or in-person with Concern team in Lebanon before or while drafting the inception report

• Desk-based research and inspection of existing data from baseline vs. endline surveys, Knowledge, Attitude and Practices (KAP) surveys, other

Literature Review (Secondary Data)

• Draft Inception report (+ work plan)

Inception report including detailed work plan

• Collect and compile data: quantitative and qualitative

Raw data files, other records (Primary Data)

• Triangulate and analyse data

Data Reports

• Evaluation Report writing (including an executive summary of 3 pages maximum, outlining the evaluation process , the objectives of the intervention, its achievement and likely impact, highlighting the lessons learnt, and making recommendations to inform future similar interventions)

Report of findings

(Draft version)

• Oral presentation of initial findings to CWW team in Lebanon

Presentation in PowerPoint

• Proof reading and finalization of Evaluation Report (the final version of the Evaluation Report must consider previous comments and feedback shared)

Report of findings

(Final version)

 

Concern (Lebanon, RD-DO and Humanitarian Advisor) will allot up to three days for revision and feedback on the initial draft report.

All field related expenses will be on the charge of the The final payment to the consultant will be issued upon approval of a high-quality, and structured final report of findings, as well as upon successful delivery of all outputs in the Terms of Reference). The final report will be then shared by RD/DO/Lebanon team with the global consultants, as well as with other relevant stakeholders (internal and external). Aspects of the Lebanon report will be incorporated into the organisational report that Concern HQ will be finally sharing with Irish Aid.

The final version of the consultant’s evaluation report of findings should include an appraisal of how well the programme has fared against each of the extended OECD-DAC criteria using the following 5-point grading:

• (5) Outstanding (exceptional) Performance
• (4) Very good overall performance (above expectation)
• (3) Good overall performance (as expected)
• (2) Acceptable performance but with some shortcomings
• (1) Barely acceptable performance with some major shortcomings
• (0) Totally unacceptable performance or insufficient data to make an assessment

As a final point, the global consultants have developed an evaluation framework that must be completed as part of the evaluation process and which the consultant will be asked to fill. The consultant will be asked to provide a completed version of this framework, including references to all quantitative and qualitative data sources to the global consultants, along with any data sets used in data analysis included in the final report, as well as notes, recordings, images, etc. from key informant interviews and focus groups for validation and quality assurance.

 

Experience, Skills, Roles & Responsibilities

The consultant will:

• • Lead liaison with Concern HPP Coordinator, IAPG HPP global evaluation team and in-country stakeholders. The successful consultant will directly report to Concern’s Country Director (Lebanon)
  • Lead planning and preparations for country programme data collection and collation, ensuring all permissions necessary for conducting data collection and collation have been granted and are adhered to (including any ethical clearance or permissions).
  • With Concern country programme staff and programme implementers identify key informants for KII and FGD participants
  • Facilitate key in-country meetings and workshops with Concern country programme staff and other key stakeholders where necessary
  • Conduct in-country data collection and collation (observations, KII and FGD, etc.)
  • Conduct HPP country programme evaluation data analysis and interpretation
  • Write HPP country programme evaluation report.

 

The above-mentioned tasks are only guidance, and should not be viewed as an inflexible specification as they could be changed after deliberations with the selected consultant and CD. 

The (successful) consultant (currently residing in Lebanon or willing to travel to Lebanon for the duration of the evaluation) should have the following skills:

• Post-graduate degree in Social Sciences with emphasis on Gender, Protection and Psychosocial Wellbeing (a must-have)
• Previous experience conducting mid- and end-term evaluations of humanitarian programmes in general and protection programmes in specific (a definite plus)
• Familiarity with and has technical knowledge about key themes such as Gender, Protection and Psychosocial Support
• Acquaintance with participatory methodologies and best ways to implement them (participatory evaluation)
• Able to develop, use and apply gender-sensitive evaluation methodologies (hybrid)
• Numerate with excellent data compilation and analysis skills
• Able to assess the quality of programmatic data collection systems, analyse budgets, results frameworks, M+E reports and surveys, and other documents
• Has exceptional writing and verbal skills in both English and Arabic i.e. previous experience writing evaluation reports to a high standard in English
• Previous experience working as part of a dynamic team.

 

Reporting Structure

The successful consultant will directly report to Concern’s Country Director (Lebanon) and will closely liaise with the Lebanon Protection Programme Manager and MEAL Manager.

Timeline

The evaluation of the HPP in Lebanon will be carried out between 13 July 2021 and 20 August 2021. The duration of the final evaluation will thus extend over a 4-week period.