Terms of Reference Mid-Term Evaluation - The Just Right Start Project

Project Summary

• Project title: The Just Right Start
• Project reference code:326.161711
• Project duration: 01/10/2016 to 01/10/2019 (with a possibility of a no cost extension till December 2020)
• Project implemented by:  Ana Aqra Association
• Donor: Private Philanthropic Organization
• Project Location: Lebanon
• Project Sites: Mount Lebanon & West Beqaa
• Budget in the proposal: Euros, 2,999,466
• Period of the Evaluation: Mid-Term (November 2018)

Background:

The Just Right Start project aims to prepare Syrian children and disadvantaged host community boys and girls aged between 4-6 years old to be enrolled and retained at schools. Many outputs are set to reach this main goal. The project aims to enroll learners, who have no access to early childhood preparatory programs, in the community based shortened Early Childhood Education. Children who completed at least one cycle of the shortened ECE and are of age 6 and up are oriented towards formal schooling provided by MEHE and still have access to the Just Right Start homework support classes within the community based ECE. In support of their continued learning, the project provides Parents of out of school Syrian and host community girls and boys a program that enables safe and learning environment at home.

The project completed to date, 4 of the 6 expected ECE cycles and 2 of the 4 expected Homework support cycle (with a possibility of a no cost extension for 2 additional ECE and HWS cycles) The evaluation is carried out at this point in time to assess and present results (output, outcome), conclusions, lessons learnt and recommendations for the remaining project duration. The evaluation would also provide recommendations for a possible future programmatic directions, that focus more on changing the mindset towards responsibility for learning results and ways to influence relevant (formal) education systems (eg. capacity building, testing of SEL measurement tools.)

Project outcomes:     
1.    Out of school Syrian refugee and host community girls and boys (ages 4-5) have access to safe protected inclusive community based ECE learning environments and are ready to enroll in public schools in Bekaa, Mount Lebanon, and Akkar

2.    Out of school Syrian refugee and host community girls and boys (ages 6-8) who completed at least one cycle of Community based ECE and have enrolled in grade 1 of public school are promoted to the next grade level on the following school year in Bekaa, Mount Lebanon, and Akkar

3.    Out of school Syrian refugee and host community girls and boys (ages 4-8) who are enrolled in the Community based ECE or HWS reveal increased social and emotional well being as they engage in learning and become part of the learning community in Bekaa, Mount Lebanon, and Akkar

4.    Parents of out of school Syrian and host community girls and boys provide safe and enabling learning environments to their children in support of their continued learning (ages 2 to 8)

5.    Teachers (of HWS, ECE, Just Right Start) of out of school Syrian refugee and host community adopt best practices in differentiated teaching for the provision of safe and enabling learning environments in Bekaa, Mount Lebanon and Akkar.

Under the relevant DAC evaluation criteria (relevance, impact, effectiveness, efficiency, sustainability and coordination) the evaluator has to analyze the following points:

1.    The extent to which the project has already achieved its objectives and results or is likely to achieve them, including the extent to which the lives of the beneficiaries (Syrian and host community girls and boys) has already been improved.

1.1.    The form for the results-assessment needs to be filled in by the evaluator as part of the reporting requirement. This is relevant for end term and mid-term evaluations.
2.    The strengths and weaknesses in terms of planning, management, implementation and monitoring

The evaluation will assess the output and outcome indicators undertaken in line with the aforementioned objectives.

IV.    Evaluation questions
More specifically, it is expected that the evaluator answers the following evaluative questions:
1.     Effectiveness:

-    What is the level of implementation of all activities so far? (number of activities implemented vs. how many are supposed to be completed by the end of the project)
-    What level/type of achievement has the project attained when compared with the baseline status?
-    Are the actions taking place enough to reach the project’s goals?
-    What are the major factors affecting the quality of work?
-    What are the trigger factors enhancing the quality of work?
-    How effective is AA at measuring the impact on learning outcomes? Does it have the right tools and systems in place to provide evidence of both intentional and unintentional outcomes?

2.    Efficiency:

-    Were activities cost-efficient?
-    Were objectives achieved on time?
-    Was the project implemented in the most efficient way compared to alternatives?)

Output Indicators
(To be retrieved from Available Progress reports/logframe during the desk review)

3.    Impact
-  How has the education programme (learning space, facilitators, curricula and staff) contributed to the protection, personal, social and emotional development of children in the programme?
- What impact has AA education programme had on access and retention of Syrian children in formal education since the beginning of the programme? How did the ECE and Homework support programme contribute to access and retention of children in formal education?
- What impact has AA Education programme had on community participation and refugee engagement of both parents? How has it improved their lives?
-Which intended and unintended negative and positive effects can be possibly attributed to the project/program?
-What role is technology playing to support the education for Syrian children. Is it appropriate and relevant and which areas should be scaled up or adapted in the future?
-How many girls, boys and parents in total have already benefited from the project/program (immediate impact)?
-How can the impact of the Just Right Start project be maximized?
- Are there any other important aspects regarding impact?

4.    Relevance and Appropriateness
-How relevant and appropriate is the current program design and implementation to the educational, life skills and social needs of Syrian refugee and host community children?
-To what extent has AA adapted to changes in the operating context since 2016?
- What should be done to improve the relevance and appropriateness of the program?
-What programmatic areas should be scaled up or adapted in future?
- To what degree is the project designed and implemented based on the opinions and the priorities of Syrian refugee children and parents? Is it perceived as relevant by them? If so, how?
- To what extent are the different needs of the various sub-groups (children with PSS needs, children with disabilities, children with special education needs) taken into account? How can the programme better target sub-groups to become more inclusive?
-Does the curricula and pedagogy meet the needs of the students? How can they be made more appropriate?
- Does the education programme meet the INEE minimum standards for education in emergency?
-Are the expected results/outputs of the Just Right Start Project consistent with the outcome, immediate impact and overall goal/impact?

5.    Learning and Sustainability
- What are the suggestions for more effective programming to meet the overarching objective of access and retention in quality formal education? (structurally, work modality and staffing)? Identify current gaps and suggest good practices for future responses and organizational learning.
- What, if any, are the scale up options for the education programme in both formal and non-formal education?
- How can AA contribute to a sustainable educational strategy in the host communities? What are the major factors influencing the achievement or non-achievement of sustainability of the Just Right Start Project?
- What steps need to be put in place to ensure the ECE centres and homework support programme can continue to provide quality education for the Syrian Refugee and Host community?
- By reviewing the theory of change and identifying existing as well as potential innovative elements of the programme, what could be strengthened to achieve greater impact in areas identified as important by children and their parents?  

6.    Coordination:
-How well is the AA program internally coordinated? And how can it be improved?
-What is the potential for synergies with other relevant education providers (formal and non-formal)?
-What has been the role of AA in the advocacy and coordination of quality certified education for refugee children within relevant MEHE working groups and sector relevant coordination meetings? How can this be improved?
-How well does the education programme coordinate with Formal schools?
-What systems have been established to ensure efficient referrals between formal schools and AA ECE centers? How can these be improved?

V.    Approach and Methods
The evaluation consists of several phases:

Phase 1: Preparatory stage

Desk Review: A desk review of relevant project documents or sources would be conducted. Key documents to review include: project proposal, project logical framework, the monitoring reports (baseline report and second intake report, assessment tools, children’s development and academic progress, relevant minutes from the relevant education working group or task force meetings)

Evaluation matrix and tools development: A detailed evaluation matrix will be produced by the evaluator; which will inform the design of the various evaluation tools. The evaluation matrix and tools (Survey questionnaire and forms) will be part of the inception report that the evaluator will submit to Ana Aqra for approval prior to moving forward in the evaluation process.

Phase 2: Data collection

It is expected that the evaluator sets up a strong data quality insurance process to avoid errors during data collection or data entry. All 3 centers will be equally included in the data collection plan, and will cover all aspects to inform the analysis.

Phase 3: Data Analysis

The data collected will be analyzed and used in order to write the report of the Mid-term evaluation study. Ana Aqra Association expects the evaluator to ensure that the evaluation reports meet the objectives as stated in those TOR, that all indicators data be presented and that clear answers to the evaluation questions appears in the report.

Phase 4: Presentation:
Presentation of key findings (feedback workshop) at the end of the conducted evaluation

1.    Final Draft Report: Submission and presentation of final draft report
2.    Final Report: Submission of final report

For the different phases it is expected that data and information will be obtained through different methods such as: analysis of documents, structured interviews, semi-structured interviews face-to face or by phone, group discussions, others.

VI.    Application Process and Requirements

AAA will be the main focal point for the assignment and direction of the evaluation and will receive directly the evaluation report.
AAA will guide and facilitate the evaluator work in Lebanon by:
-    Providing a briefing for the evaluator to clarify open questions related to the “Just Right Start” project and the mid-term review (methodology, logistics, deliverables).
-    Providing the evaluator with relevant documentation and contacts.
-    Supporting the organization of the field phase.
AAA will:
-    Provide a management response to the results of the mid-term review to the donor, upon receipt of the draft final report.
The evaluator will:
-    Manage his/her schedule and make appointments for interviews.
-    Organize a local debriefing before the end of the field phase.
-    Collect and analyze data.
During the preparation phase, the evaluator will submit a work plan that should clarify the focus of the mid-term review and the TOR, highlighting any reservations regarding the feasibility. The report should not exceed 1 page and should include as a minimum:
o    Key data of the mid-term review (project title, project data, commissioner of the mid-term review etc.);
o    Mid-term review design: Methodology, approach, steps for implementation and suggested timelines;
o    Annex with tools and survey templates etc.
The evaluator will develop the final methodology for the evaluation that will be discussed and approved by AAA and the donor.

VII.    Timetable

The work will be concluded in a period of 15 days, indicatively broken down as follow:
- Preparation (preparation methodology, work plan, inception report) - 2 day
- Briefings / Debriefings - 1 day
- Field work (ideally 12 - 16 November) – 5 working days
- Report writing & data analysis - 7 Days
Total: 15 Days
The start of the evaluation is beginning of July 2018. The final report has to be submitted End of November 2018.

VIII.    The Evaluation Team

Key Qualifications in the team should be:
-    Master level in any field related to early childhood education and/or management.
-    A minimum of three years’ experience and expertise in the field/sector of early childhood education and a deep understanding of the refugee context in Lebanon.
-    Team leader has conducted at least three evaluations in the last five years ideally in the relevant field.
-    Team member has participated in at least three evaluations ideally in the relevant field.
-    Working experience in Lebanon.
-    Experience in project cycle management.
-    Experience preparing and analyzing a theory of change.
-    Experience in participatory, qualitative methods, to complement the significant amount of quantitative data.
-    Experience in social science methods.
-    Excellent oral and written English skills.
-    Sound MS Office and IT skills.

IX.    Reports
The consultants will submit the following reports:
-    an inception report (10-15 pages without annexes),
-    a final draft evaluation report (about 25-30 pages without annexes), including a draft executive summary and the results-assessment form (part of the reporting requirement)
-    and the final evaluation report (25-30 pages without annexes), the final executive summary and the results-assessment form (part of the reporting requirement)

All reports need to be written in English.
The executive summary should summarize key findings and recommendations (three to five pages) and needs to be submitted as part of the final draft report.
The findings and recommendations of the draft final report and final report have to be structured according to the evaluation questions. An outline of the report’s structure needs to be agreed upon during the inception phase.
The quality of the reports will be judged according to the following criteria:

-    Is the results-matrix format part of the report?
-    Does the report contain a comprehensive and clear executive summary?
-    Were the Terms of Reference fulfilled and is this reflected in the report?
-    Is the report structured according to the donor criteria?
-    Are all evaluation questions answered?
-    Are the methods and processes of the evaluation sufficiently documented in the evaluation report?
-    Does the report describe and assess the intervention logic (e.g. log frame, program theory) and present/analyze a theory of change and its underlying assumptions?
-    Are the conclusions and recommendations based on findings and are they clearly stated in the report?
-    Does the report clearly differentiate between conclusions, lessons learnt and recommendations?
-    Are the recommendations realistic and is it clearly expressed to whom the recommendations are addressed to?
-    Were the most significant stakeholders involved consulted?
-    Does the report present the information contained in a presentable and clearly arranged form?
-    Is the report free from spelling mistakes and unclear linguistic formulations?
-    Can the report be distributed in the delivered form?