Terms of Reference - Final Internal Project Evaluation of “Enhanced capacity of the government and social partners to develop a national labour policy, and mainstream SDGs relating to employment and DW into national development and crisis response in Leb

Evaluation Background

1. ILO considers evaluation as an integral part of the implementation of development cooperation projects. Provisions are made in all projects in accordance with ILO evaluation policy and based on the nature of the project and the specific requirements agreed upon at the time of the project design and during the project as per established procedures. The Regional Evaluation Officer (REO) at the ILO ROAS supports the evaluation function for all ILO projects.
2. According to the project documents, a final internal evaluation will be conducted. It will be used to assess the achievements of results, identify the main difficulties/constraints, assess the impact of the project for the targeted populations, sustainability of project interventions and formulate lessons learned and practical recommendations to improve future similar project. This evaluation will also look at the effect of COVID-19 on the project’s timeline and its impact on project implementation. The findings of the evaluation will be used in in the design of new or potential future similar projects in the region.

Purpose

1. The purpose of the final evaluation is to assess the overall achievements of the project against its planned outcomes and outputs to generate lessons learned, best practices and recommendations.
2. It will provide analysis according to OECD criteria at country level and will examine the efficiency, effectiveness, relevance, potential impact and sustainability of the projects. The evaluation report shall reflect findings from this evaluation on the extent to which the different components have achieved their stated objectives. This evaluation will also identify strengths and weaknesses in the project design, strategy, and implementation as well as lessons learned with recommendations. Furthermore, it will touch upon cross cutting issues such as gender equality, disability, social dialogue, environmental sustainability, and international standards, and covid-19 in terms of challenges and opportunities for tackling the most vulnerable segments in line with guidelines and protocols set by EVAL/ILO.
3. The evaluation will comply with the ILO evaluation policy including the protocols and guidelines set by EVAL/ILO, which is based on the OECD DAC and United Nations Evaluation Norms and Standards and the UNEG ethical guidelines.

Scope

1. The evaluation will assess the project duration covering August 2020 - April 2022. It will look at the project achievement at the level of each milestone. The evaluation will take into consideration the project duration, existing resources and political, security and environmental constraints. It will also look into the link between the project’s objectives and the ILO’s P&B strategy, DWCP in Lebanon, and the UNSDCF in Lebanon.
2. The evaluation will take place from July till September through online/field work to collect information from different stakeholders. The consultancy shall start with initial briefing with the project team and the Regional Office for Arab States (ROAS).
3. The evaluation will integrate gender equality, inclusion of people with disabilities, environmental sustainability, ILS and social dialogue, and Covid-19 as crosscutting concerns throughout its methodology and deliverables, including the final report. This is based on EVAL’s protocols on crosscutting issues including the one on covid-19.

Clients of Evaluation

1. The primary clients of this evaluation are ILO, ILO ROAS, ILO EVAL, ILO constituents in Lebanon: Ministry of Labour, the Association of Lebanese Industrialists (ALI), workers’ organizations, the Central Administration of Statistics (CAS), and other UN agencies.. Secondary users include other project stakeholders and units that may indirectly benefit from the knowledge generated by the evaluation. 

Evaluation Criteria and Questions

1. The evaluation utilises the standard ILO framework and follows its major criteria while integrating gender equality as a cross cutting issue throughout the evaluation questions:

• Relevance and strategic fit – the extent to which the objectives are aligned with sub-regional, national and local priorities and needs, the constituents’ priorities and needs, and the donor’s priorities for the country;
• Validity of design – the extent to which the project design, logic, strategy and elements are/remain valid vis-à-vis problems and needs;
• Efficiency - the productivity of the project implementation process taken as a measure of the extent to which the outputs achieved are derived from an efficient use of financial, material and human resources, including re-purposing in the mitigation of Covid-19 impacts;
• Effectiveness - the extent to which the project can be said to have contributed to the project objectives and more concretely whether the stated outputs have been produced satisfactorily with gender equality, including in the Covid-19 context; in addition to building synergies with national initiatives and with other donor-supported project;
• Impact - positive and negative changes and effects caused by the project at the national level, i.e. the impact with social partners, government entities, beneficiaries, etc.; special attention should be given to secondary job effects, which are expected to occur in economic infrastructure like agricultural roads, markets or irrigation.
• Effectiveness of management arrangements - the extent of efficient operational arrangements that supported the timely, efficient, and effective delivery of the project
• Sustainability – the extent to which adequate capacity building of social partners has taken place to ensure mechanisms are in place to sustain activities and whether the existing results are likely to be maintained beyond project completion, in the case of infrastructure this refers concretely to whether operation and maintenance agreements are actually being implemented; the extent to which the knowledge developed throughout the project (research papers, progress reports, manuals and other tools) can still be utilised after the end of the project to inform policies and practitioners,

Relevance and strategic fit:

• How well did the project approach fit in context of the protracted crisis in Lebanon? Were the problems and needs adequately analysed? Was gender prioritized?
• How well were the project’s objectives aligned with the framework of the ILO Decent Work Country Programme of Lebanon, the ILO’s Project and Budget (P&B) 2020-21, and the SDGs?  
• To what extent did the ILO project provide a timely and relevant response to constituents’ needs and priorities in the COVID-19 context?

Validity of design:

• Were the project’s strategies and structures coherent and logical (the extent of logical correlations between the objectives?
• Were project’s assumptions and targets realistic, and did the project undergo risk analyses and design readjustments when necessary?
• To what extent did the project design take into account: Specific gender equality and non-discrimination concerns relevant to the project context? As well as concerns relating to inclusion of people with disabilities, environmental sustainability, ILS and social dialogue?

Efficiency:

• Were all resources utilized efficiently to reach the project’s objectives?
• How efficient were the coordination efforts with the CAS, worker and employer representatives, and other constituents?
• To what extent has the project been on track in terms of timely achieving the assigned milestones? If not, what factors contributed to the delays?
• To what extent has the project leveraged new or repurposed existing financial resources to mitigate COVID-19 effects in a balanced manner? Does the leveraging of resources take into account the sustainability of results?
• What were the intervention benefits and related costs of integrating gender equality?

Effectiveness:

• Were all set targets, outputs, and outcomes achieved according to plan?
• How effective was the coordination with the different stakeholders in supporting the project’s objectives?
• How have stakeholders, particularly women, been involved in project’s implementation?
• How did the outputs and outcomes contribute to ILO’s mainstreamed strategies including gender equality, social dialogue, and labour standards?
• What positive or negative unintended outcomes can be identified?
• Has the project fostered ILO constituents’ active involvement through social dialogue in articulating, implementing and sustaining coherent response strategies to mitigate the effects of the pandemic on the world of work?

Impact orientation:

• How were the survey results incorporated into the policy development?
• What are the expected impacts of the policy on the world of work in Lebanon?

Sustainability:

• Are the results achieved by the project so far likely to be sustainable- in terms of conducting surveys and drafting policies?
• What measures have been taken to ensure that the key components of the project are sustainable beyond the life of the project? Are they sufficient?

• How effectively has the project built national ownership? Is the CAS equipped to continue with the implementation beyond the project’s lifespan?

Effectiveness of management arrangements:

• What was the division of work tasks within the project’s teams? Has the use of local skills been effective?
• How effective was communication between the project’s teams, the regional office and the responsible technical department at headquarters? Has the project received adequate technical and administrative support/response from the ILO backstopping units?

Challenges, Lessons learned and Specific Recommendations for the formulation of new Phases:

• What good practices can be learned from the different phases of the project that can be applied to similar future projects?
• What were the main challenges identified? How were these different from the risk assumptions? What were the mitigation steps taken?
• What are the recommendations for future similar projects?
• What are the challenges, lessons learned and the recommendations regarding the cross-cutting issues of gender equality and inclusion, social dialogue, ILS, COVID-19 and environmental sustainability?

Methodology

1. The following is the proposed evaluation methodology. Any changes to the methodology should be discussed with and approved by the evaluation manager.
2. This evaluation will follow a mixed method approach relying on available quantitative data and primary qualitative data collected through interviews.
3. This evaluation will utilize all available quantitative and qualitative data from progress reports to research studies and database. The information will be analysed in light of the main thematic questions and results will be integrated with the data from the primary collection.
4. The primary data collection will mainly focus on a qualitative approach investigating the perceptions and inputs of the different stakeholders that had some form of interface with the project. Triangulation of data will also be done using both the secondary and the primary data collected. The analysis will follow a thematic examination of the main evaluation areas as guided by the evaluation questions. Considering the nature of the project and its stakeholders KIIs will be conducted. Gender will be mainstreamed throughout the methodology from data collection to data analysis. Where appropriate, the methodology will ensure equal representation of women and men throughout data collection. The evaluation will follow the ILO EVAL Guidelines on integrating gender equality3.  The specific evaluation methodology will be provided in the inception report prepared by the evaluation team and approved by the Evaluation Manager.  Tool: The interview guide will be developed in light of the evaluation themes and main questions as well as the type of stakeholders. Sample: The study sample should be reflective of all relevant stakeholders taking into consideration the scope of the project and its evaluation as well as data saturation. All analysed data should be disaggregated by sex. The results shall address the crosscutting issues described above (including Covid-19).

Work Assignments:

Internal briefing by the project team(s):

The evaluator will have an initial consultation with the evaluation manager, REO, relevant ILO specialists and support staff in ROAS. The objective of the consultation is to reach a common understanding regarding the status of the project, the priority assessment questions, available data sources and data collection instruments and an outline of the final assessment report. The following topics will be covered: status of logistical arrangements, project’s backgrounds and materials, key evaluation questions and priorities, outline of the inception and final report.

• Preparation of the inception report
• Report to be shared with Evaluation manager for comments
• Report to be shared with key stakeholders for comments
• Inception report revised and interviews to begin

Desk Review:

The evaluator will review project’s background materials before conducting any interviews.

These include:  

• Project documents (Logic Framework, Theory of change,…)
• Baseline reports and related data (if available)
• Monitoring reports conducted during the project
• Progress and status reports, extensions and budget revisions
• Previous phase or related evaluation reports of the project (if available)
• Other studies and research undertaken by the project
• Project beneficiary documentation

Individual Interviews:

Following the initial briefing, the desk review and the inception report, the evaluator will have meetings with constituents/stakeholders together with interpreters supporting the process if needed.

Individual interviews will be conducted with the following:

1. Project staff/consultants that have been active in ILO (including Chief Technical Advisor, technical, administrative, and finance staff);
2. ILO ROAS DWT Director and DWT Specialists, RPU, Employers’ and Workers’ Organisations;
3. ILO Headquarters technical departments;
4. Interviews with constituents and national counterparts

Debriefing

Upon completion of the missions, the evaluator will provide a debriefing to the stakeholders to validate the evaluation findings, conclusions and recommendations.

Evaluation Management

The evaluator will report to the Evaluation manager in ROAS. The Evaluation Manager will be the first point of contact for the consultant as well as the project team for any technical and methodological matters related to this evaluation. All communications with regard to this evaluation must be marked to the evaluation manager. The ILO ROAS office and the project team will provide administrative and logistical support for the interviews.

The Main Deliverables:

• Deliverable 1: Inception Report
• Deliverable 2: Draft evaluation report
• Deliverable 3: Stakeholder debrief, PowerPoint Presentation (PPP)
• Deliverable 4:  Comments log of how all comments were considered and taken on board by the evaluation team or not and why not.
• Deliverable 5: Final evaluation report with executive summary (report will be considered final after review by REO. Comments will have to be integrated).

Inception Report

The evaluator will draft an Inception Report, which should describe, provide reflection and fine-tuning of the following issues:

1. 1. Project background
   2. Purpose, scope and beneficiaries of the evaluation
   3. Evaluation criteria and questions
   4. Methodology and instruments
   5. Main deliverables
   6. Management arrangements and work plan

Final Report

The final version of the report will follow the below format:

• Title page
• Table of Contents, including List of Appendices, Tables
• List of Acronyms or Abbreviations
• Executive Summary with methodology, key findings, conclusions and recommendations
• Background and Project Description
• Purpose of Evaluation
• Evaluation Methodology and Evaluation Questions
• Status of objectives
• Clearly identified findings along OECD/DAC criteria, substantiated with evidence
• Key results (i.e. figures and qualitative results) achieved per objective (expected and unexpected)
• Clearly identified conclusions and recommendations that are linked to findings (identifying which stakeholders are responsible, priority of recommendations, and timeframe)
• Lessons Learned per ILO template
• Potential good practices per ILO template
• Annexes (list of interviews, TORs, lessons learned and best practices in ILO EVAL templates, list of documents consulted, etc.) Annex: Different phases’ log frames with results status, by phase.

The quality of the report will be assessed against the relevant EVAL Checklists. The deliverables will be submitted in the English language, and structured according to the templates provided by the ILO. 

Roles And Responsibilities

1. The Evaluator is responsible for conducting the evaluation according to the terms of reference (ToR). He/she will:

• Review the ToR and provide input, propose any refinements to assessment questions, as necessary;
• Review project background materials (e.g., project document, progress reports, etc.);
• Prepare an inception report including a matrix of evaluation questions, workplan and stakeholders to be covered;
• Develop and implement the evaluation methodology (i.e., conduct interviews, review documents, etc.) to answer the evaluation questions;
• Conduct preparatory consultations with the ILO REO prior to the evaluation mission;
• Conduct online/ field research, interviews, as appropriate, and collect information according to the suggested format;
• Prepare an initial draft of the evaluation report with input from ILO specialists and constituents/stakeholders;
• Conduct a briefing on the findings, conclusions and recommendation of the evaluation to ILO and other stakeholders;
• Prepare the final report based on the ILO, donor and stakeholders’ feedback.
• 1. The ILO Evaluation Manager is responsible for:
• Drafting the ToR;
• Finalizing the ToR with input from colleagues;
• Preparing a short list of candidates for submission to the Regional Evaluation Officer, ILO/ROAS and EVAL for final selection;
• Hiring the consultant;
• Providing the consultant with the project background materials;
• Participating in preparatory consultations (briefing) prior to the assessment mission;
• Assisting in the implementation of the evaluation methodology, as appropriate (i.e., participate in meetings, review documents);
• Reviewing the initial draft report, circulating it for comments and providing consolidated feedback to the Evaluators (for the inception report and the final report);
• Reviewing the final draft of the report;
• Disseminating the final report to all the stakeholders;
• Coordinating follow-up as necessary.
  1. The ILO REO:
• Providing support to the planning of the evaluation;
• Approving selection of the evaluation consultant and final versions of the TOR;
• Reviewing the draft and final evaluation report and submitting it to EVAL;
• Disseminating the report as appropriate.
• 1. The Project Coordinator is responsible for:
• Reviewing the draft TOR and providing input, as necessary;
• Providing project background materials, including studies, analytical papers, progress reports, tools, publications produced, and any relevant background notes;
• Providing a list of stakeholders;
• Reviewing and providing comments on the inception report;
• Participating in the preparatory briefing prior to the evaluation missions;
• Scheduling all meetings and interviews for the missions;
• Ensuring necessary logistical arrangements for the missions;
• Reviewing and providing comments on the initial draft report;
• Participating in the debriefing on the findings, conclusions, and recommendations;
• Providing translation for any required documents: TOR, PPP, final report, etc.;
• Making sure appropriate follow-up action is taken

Duration of Contract and Timeline for Delivery

The collaboration between ILO and the Consultant is expected to start in July until September 2022 with an estimate of 27 working days.

Evaluation Timeframe TO BE FURTHER DEVELOPED AND AGREED

Supervision

The evaluator will work under the direct supervision of the Evaluation Manager. The evaluator will be required to provide continuous updates on the progress of work and revert to the ILO with any challenges or bottlenecks for support. Coordination and follow-up with the evaluator will take place through e-mail or skype or any other digital communication mean.

Legal and Ethical Matters

1. This independent evaluation will comply with ILO evaluation guidelines and UN Norms and Standards.
2. These ToRs will be accompanied by the code of conduct for carrying out the evaluation “Code of conduct for evaluation in the ILO” (See attached documents).
3. UNEG ethical guidelines will be followed throughout the independent evaluation.
4. The consultant will not have any links to project management or any other conflict of interest that would interfere with the independence of the evaluation.

Requirements:

The evaluator should have:

- An advanced degree in social sciences;

- Proven expertise on evaluation methods, labour markets, conflict issues and the ILO approach;

- Extensive experience in the evaluation of development interventions;

- Expertise in the Labour intensive modality, job creation projects, capacity building and skills development and other relevant subject matter;

- An understanding of the ILO’s tripartite culture;

- Knowledge of Lebanon, and the regional context;

- Full command of the English and Arabic language (spoken and written) is required.

The final selection of the evaluator will be approved by the Regional Evaluation Focal Point in the ILO ROAS.