Terms of Reference End of Project Evaluation - The Just Right Start Project

Project Summary

• Project title: The Just Right Start
• Project reference code:326.161711
• Project duration: 01/10/2016 to 01/10/2019, extended till August 2020
• Project implemented by:   Ana Aqra Association
• Donor: Private Philanthropic Organization
• Project Location: Lebanon
• Project Sites: Mount Lebanon & West Beqaa
• Budget in the proposal: Euros, 2,999,466
• Period of the Evaluation: (March-April 2020)
• Previous Mid-Term Evaluation: November 2018
• Background:

• The Just Right Start project aims to prepare Syrian children and disadvantaged host community boys and girls aged between 4-6 years old to be enrolled and retained at formal schools. Many outputs are set to reach this main goal. The project aims to:

• Enroll learners, who have no access to early childhood preparatory programs, in the community based shortened Early Childhood Education.
• Children who completed at least one cycle of the shortened ECE and are of age 6 and up are oriented towards formal schooling provided by MEHE and
• still have access to the Just Right Start homework support classes within the community based ECE. In support of their continued learning, the project provides:

• Parents of out of school Syrian and host community girls and boys a program that enables safe and learning environment at home. This program is designed to first, raise the awareness of parents through parents’ meetings conducted by the center coordinators; and second, to enable parents with younger children (younger sibling of the enrolled child)

• to enhance their abilities as their child’s first teacher within an enabled home learning environment; as well as the program aims at
• Improve teacher capacity building through trainings and coaching.

• The evaluation is carried out at this point in time to assess and present results (output, outcome), conclusions, lessons learnt and recommendations for a possible future programmatic directions that focus more on changing the mindset towards responsibility for learning results and on ways to scale up Ana Aqra ECE program and influence relevant (formal) education systems.

  In August 2020, the project will be completing, 8 of the 6 expected ECE cycles and 6 of the 4 expected Homework support cycles (2 cycles were added during the no cost extension period). A cycle is defined to be the period of 77 teaching days. Each year is composed of 2 cycles and a summer program. Children who reach the age of formal compulsory education are referred to enroll in the formal school. 

• Evaluation questions

• More specifically, it is expected that the evaluator answers the following evaluative questions:

•  Effectiveness:

• What is the level of implementation of all activities so far? (number of activities implemented vs. how many are supposed to be completed by the end of the project)

• What level/type of achievement has the project attained when compared with the baseline status?
• Are the actions taken enough to reach the project’s goals?
• What are the major factors affecting the quality of work?
• What are the trigger factors enhancing the quality of work?
• How effective is AA at measuring the impact on learning outcomes? Does it have the right tools and systems in place to provide evidence of both intentional and unintentional outcomes?
• What are the minimum conditions that will allow scale up or replication?
• Efficiency:
• Were activities cost-efficient?
• Were objectives achieved on time?
• Was the project implemented in the most efficient way compared to alternatives?)

• Output Indicators

  (To be retrieved from Available Progress reports/logframe during the desk review)

   

• Impact
• -  How has the education program (learning space, facilitators, curricula and staff) contributed to the protection, personal, social and emotional development of children in the program?

• - What impact has AA education program had on access and retention of Syrian children in formal education since the beginning of the program?

  - How did the ECE and Homework support program contribute to retention of children in formal education and to enhancing their promotion rates?

  - What impact has AA Education program had on community participation and refugee engagement of both parents? How has it improved their lives?

  -Which intended and unintended negative and positive effects can be possibly attributed to the project/program?

  -What role is technology playing to support the education for Syrian children. Is it appropriate and relevant and which areas should be scaled up or adapted in the future?

  -How many girls, boys and parents in total have already benefited from the project/program (immediate impact)?

  -How can the impact of the Just Right Start project be maximized?

  - Are there any other important aspects regarding impact?

  Has the partnership with the Private Philanthropic Organization brought any positive or negative impact on the program?

   

• Relevance and Appropriateness

• -How relevant and appropriate is the current program design and implementation to the educational, life skills and social needs of Syrian refugee and host community children?

  -To what extent has AA adapted to changes in the operating context since 2016?

  - What should be done to improve the relevance and appropriateness of the program?

  -What programmatic areas should be scaled up or adapted in future?

  - To what degree is the project designed and implemented based on the opinions and the priorities of Syrian refugee children and parents? Is it perceived as relevant by them? If so, how?

  - To what extent are the different needs of the various sub-groups (children with PSS needs, children with disabilities, children with special education needs) taken into account? How can the program better target sub-groups to become more inclusive?

  -Does the curricula and pedagogy meet the needs of the students? How can they be made more appropriate?

  - Does the education program meet the INEE minimum standards for education in emergency?

  -Are the expected results/outputs of the Just Right Start Project consistent with the outcome, immediate impact and overall goal/impact?

  Learning and Sustainability

• - What are the suggestions for more effective programming to meet the overarching objective of access and retention in quality formal education? (structurally, work modality and staffing)? Identify current gaps and suggest good practices for future responses and organizational learning.

  - What, if any, are the scale up options for the education program in both formal and non-formal education?

  - How can AA contribute to a sustainable educational strategy in the host communities? What are the major factors influencing the achievement or non-achievement of sustainability of the Just Right Start Project?

  - What steps need to be put in place to ensure the ECE centers and homework support program can continue to provide quality education for the Syrian Refugee and Host community?

  - What could be strengthened to achieve greater impact in areas identified as important by children and their parents? 

  -To what degree have AAA established or improved mechanisms, structures and platforms for generating and reflecting on evidence on the impacts of interventions? What has impeded or supported such efforts?

  -To what Degree the evidence generated contributed to influencing AAA internal programming/ national and other NGO ECE platforms/ stakeholders/ and MEHE in Lebanon?

   

• Coordination:

• -How well is the AA program internally coordinated? And how can it be improved?

  -What is the potential for synergies with other relevant education providers (formal and non-formal)?

  -What has been the role of AA in the advocacy and coordination of quality certified education for refugee children within relevant MEHE working groups and sector relevant coordination meetings? How can this be improved?

  -How well does the education program coordinate with Formal schools?

  -What systems have been established to ensure efficient referrals between formal schools and AA ECE centers? How can these be improved?

   

• Approach and Methods

• The evaluation consists of several phases:

   

  Phase 1: Preparatory stage

   

  Desk Review: A desk review of relevant project documents or sources would be conducted. Key documents to review include: project proposal, project logical framework, the mid-term evaluation report, the monitoring reports (baseline report and second intake report, assessment tools, children’s development and academic progress, relevant minutes from the relevant education working group or task force meetings)

   

  Evaluation matrix and tools development: A detailed evaluation matrix will be produced by the evaluator; which will inform the design of the various evaluation tools that will reflect mixed method. The evaluation matrix and tools (Survey questionnaire and forms) will be part of the inception report that the evaluator will submit to Ana Aqra for approval prior to moving forward in the evaluation process.

• Phase 2: Data collection

   

  It is expected that the evaluator sets up a strong data quality insurance process to avoid errors during data collection or data entry both quantitative and qualitative. All 3 centers will be equally included in the data collection plan, and will cover all aspects to inform the analysis.

  The evaluator is responsible of collecting, recoding and compiling data.

   

   

  Phase 3: Data Analysis

   

  The quantitative and qualitative data collected will be analyzed and used in order to present charts, correlations and to write the report of the end-term evaluation study. Ana Aqra Association expects the evaluator to ensure that the evaluation reports meet the objectives as stated in those TOR, that all indicators data be presented and that clear answers to the evaluation questions appears in the report.

   

   

  Phase 4: Presentation:

  Presentation of key quantitative and qualitative findings (feedback workshop) at the end of the conducted evaluation

   

• Final Draft Report: Submission and presentation of final draft report that reflect both quantitative and qualitative findings
• Final Report: Submission of complete final report with recommendations and discussions

• For the different phases it is expected that data and information will be obtained through different methods such as: data from AA system, review and analysis of documents, structured interviews, semi-structured interviews face-to face or by phone, group discussions, others.

   

  Application Process and Requirements

• AAA will be the main focal point for the assignment and direction of the evaluation and will receive directly the evaluation report.

  AAA will guide and facilitate the evaluator work in Lebanon by:

• Providing a briefing for the evaluator to clarify open questions related to the “Just Right Start” project and the end-term review (methodology, logistics, deliverables).
• Providing the evaluator with relevant documentation and contacts as requested clearly and timely.
• Supporting the organization of the field phase as planned and shared.

• AAA will:

• Provide a management response to the results of the end-term review to the donor, upon receipt of the draft final report.

• The evaluator will:

• Manage his/her schedule and make appointments for interviews.

• During the preparation phase, the evaluator will submit a work plan that should clarify the focus of the end-term review and the TOR, highlighting any reservations regarding the feasibility. The report should not exceed 1 page and should include as a minimum:

• • Key data of the end-term review (project title, project data, commissioner of the end-term review etc.);
  • End-term review design: Methodology, approach, steps for implementation and suggested timelines;
  • Annex with tools and survey templates etc.

• The evaluator will develop the final methodology for the evaluation that will be discussed and approved by AAA and the donor.

• Organize a local debriefing before the end of the field phase.
• Collect and analyze qualitative data.
• Compile and analyze quantitative data.
•  

• During the preparation phase, the evaluator will submit a work plan that should clarify the focus of the end-term review and the TOR, highlighting any reservations regarding the feasibility. The report should not exceed 1 page and should include as a minimum:

• • Key data of the end-term review (project title, project data, commissioner of the end-term review etc.);
  • End-term review design: Methodology, approach, steps for implementation and suggested timelines;
  • Annex with tools and survey templates etc.

• The evaluator will develop the final methodology for the evaluation that will be discussed and approved by AAA and the donor.

   

• Timetable

•  

  The work might be concluded in a period of 31 days and shall include the tasks below:

  - Preparation (preparation methodology, work plan, inception report) -

  -  Field work/data gathering &analysis

  - Report writing & presentation

  The start of the evaluation is March 2^nd, 2020.

   

• The Evaluation Team

•  

  Key Qualifications in the team should be:

• Master level in any field related to early childhood education and/or management.
• A minimum of three years’ experience and expertise in the field/sector of early childhood education and a deep understanding of the refugee context in Lebanon.
• Team leader has conducted at least three evaluations in the last five years ideally in the relevant field.
• Team member has participated in at least three evaluations ideally in the relevant field.
• Working experience in Lebanon.
• Experience in project cycle management.
• Experience preparing and analyzing a theory of change.
• Experience in participatory, qualitative methods, to complement the significant amount of quantitative data.
• Experience in social science methods.
• Excellent oral and written English skills.
• Sound MS Office and IT skills.

• Reports

• The consultants will submit the following reports:

• an inception report (10-15 pages without annexes),
• a final draft evaluation report (about 25-30 pages without annexes), including a draft executive summary and the results-assessment form (part of the reporting requirement)
• and the final evaluation report (25-30 pages without annexes), the final executive summary and the results-assessment form (part of the reporting requirement)

• All reports need to be written in English.

  The executive summary should summarize key findings and recommendations (three to five pages) and needs to be submitted as part of the final draft report.

  The findings and recommendations of the draft final report and final report have to be structured according to the evaluation questions. An outline of the report’s structure needs to be agreed upon during the inception phase. 

The quality of the reports will be judged according to the following criteria:

• Is the results-matrix format part of the report?
• Does the report contain a comprehensive and clear executive summary?
• Were the Terms of Reference fulfilled and is this reflected in the report?
• Is the report structured according to the donor criteria?
• Are all evaluation questions answered?
• Are the methods and processes of the evaluation sufficiently documented in the evaluation report?
• Does the report describe and assess the intervention logic (e.g. log frame, program theory) and present/analyze a theory of change and its underlying assumptions?
• Are the conclusions and recommendations based on findings and are they clearly stated in the report?
• Does the report clearly differentiate between conclusions, lessons learnt and recommendations?
• Are the recommendations realistic and is it clearly expressed to whom the recommendations are addressed to?
• Were the most significant stakeholders involved consulted?
• Does the report present the information contained in a presentable and clearly arranged form?
• Is the report free from spelling mistakes and unclear linguistic formulations?
• Can the report be distributed in the delivered form?
• Is the report well-structured and designed for friendly and ease to read and understand.