Terms of Reference for Evaluation of Inclusive Citizenship Education Programme in Lebanon

Introduction
Adyan is a foundation for Diversity, Solidarity and Human Dignity. It was founded 6 August, 2006 and is registered in Lebanon as an independent, non-profit and non-governmental organization (NGO). Adyan operates in Lebanon, as well as, regionally and internationally, and implements projects in cultural, educational, media, policymaking, social, and spiritual fields.
Vision: Adyan envisions a world where diversity between individuals and communities is lived as enrichment, generating mutual understanding, inclusive citizenship, creative development, sustainable peace, and spiritual solidarity.

Programme Overview
In 2007, Adyan launched the Alwan programme (‘Alwan’ is the Arabic word for colours). The two-year programme consists of after school clubs for students in Grade 10 and 11 (age 15 to 17). In place of societal – and often inherited – sectarianism, ignorance, prejudice and wounded memories, Alwan educates and provides opportunities for discussion, critical thinking, and new experiences.

Over the two years, Alwan builds the capacity of students to positively manage diversity and create a new culture of living together. This involves lessons, discussion and interaction in year one, and inter-communal community service in year two.

The aim is to give young people faith in their capacity to be a change-maker, create networks of supportive peers, and improve youth attitudes and skills in favour of inter-religious understanding and collaboration, and away from sectarianism.

Alwan started as a non-formal education programme, designed and implemented by Adyan in private schools. In 2011, Alwan entered the public sector, in partnership with the Lebanese Committee of UNESCO and the Ministry of Education and Higher Education.

Over the past 13 years, Alwan has created numerous young change-makers – from diverse faith and geographical – backgrounds across Lebanon. The current cohort (Alwan 2020-2022) is underway, involving 25 schools across Lebanon. Due to the Covid-19 pandemic, in 2020 the Alwan programme shifted to online. In 2021, the programme will likely involve a blended approach of online and offline learning and interaction.

From June – August 2020, Adyan contracted an external consultant to undertake an Evaluability Assessment of Alwan, in preparation for this evaluation. The Evaluability Assessment Report will be shared with those who express interest in this evaluation by the deadline listed above. We anticipate that this report will be highly beneficial to the design of a full proposal for this evaluation.

Terms of Reference Overview
With this Terms of Reference, Adyan is inviting proposals to design and deliver an evaluation of the Alwan programme from its inception up until and including the ongoing cohort. Thus the evaluation requires both a backward-looking study and a forward-looking quasi-experimental evaluation.

We expect the impact evaluation to commence with collaboratively designing the evaluation methodology, using the Evaluability Assessment Report as key starting point. A key question would be the feasibility/relevance of reconstructing a baseline. The Evaluability Assessment suggested the development of an interim baseline against which the future trajectory of the programme can be assessed.

Next step would be to refine and finalise the programme theory of change, which was developed during and following the Evaluability Assessment.

We expect the evaluation to assess the extent to which the Alwan programme has made, and is making, an impact on the attitudes, behaviour, skills and lives of participants, including former and current students, educators, partner schools and communities (over the past 13 years, the Alwan program has reached a total of 67 High-Schools, 208 educators and 4,716 graduates).

The issue of causality and attribution need to be fully examined. This will add valuable evidence-based statements on the impact of the programme.

We also expect the evaluation to propose methods for a strengthened M&E framework, to continually strengthen the evidence base about the long-term outcomes of the programme. In addition, we expect the impact evaluation to involve a stakeholder mapping, and to propose methods for improved engagement/reaching new stakeholders.

We also expect the evaluation to involve tailored capacity building of core members of the Adyan and Alwan team, including on designing and managing a relevant and appropriate theory of change, logframe and M&E framework.

This should include training on how to evaluate changes in behaviour and values, both online and offline, and how to avoid confirmation bias. It should also include training on writing appropriate survey questions, the types of data most relevant to the programme, and the mechanisms that need to be used to gather and analyse them in appropriate formats. This will help the team to identify the most appropriate tools to capture any evidence of impact, which at the moment is largely recorded through anecdotal data.

Evaluation outputs would include an inception, mid-term and final report, and audio-visual materials that can be used as a communications tool to explain key findings. All communication and outputs will be in English. Workshops will be conducted in English, or with translation into Arabic for participants as required.

Key Impact Evaluation Details:
Programme budget: Between €100,000 – 150,000 Euros
Evaluation duration: May 2021 – October 2023 (30 months)
Location: Lebanon (participating schools are based in all regions)

Programme Summary
Problem Analysis

Lebanon is known for its religious diversity, with 18 officially-recognized sects. Despite this, the society is also deeply divided, with segregation evident between religious groups. Over the past few years, these divisions have grown with the rise of populist political messaging, including sectarian-based narratives that stoke fear and mistrust between communities.

In contrast, the anti-government protests that started in October 2019 - in which young people including Alwan alumni played a significant role - demonstrated the appetite that young people have for helping to move Lebanon past its sectarian past and present. However, the challenges that 2020 brought – including a financial crisis, and social and economic pressures resulting from the impact of 

COVID-19 – have dampened the hope and faith in civic ability to effect meaningful change that prevailed in late 2019.

The devastating explosion in Beirut on 4 August brought a new level of despair to the public in Lebanon - emotionally and socio-economically. With the evident failure of national leadership - not only leading to the explosion but also in their actions since – the sense among many young people is that the sectarian system is inexorable.

Thus, this is a crucial time to work with and support young people in Lebanon. There is an urgent need to implement initiatives that give young people faith in their capacity to be a change-maker, create networks of supportive peers, and improve youth attitudes and skills in favour of inter-religious understanding and collaboration, and away from sectarianism.

Currently, schools in Lebanon offer limited opportunities for interaction between young people from different communities. Education, however, has huge potential – and responsibility – to shape attitudes and behaviour. To address this, and to develop the skills of Lebanese youth, in 2007 Adyan launched the Alwan programme.

Target Group
The ultimate beneficiaries of the programme are the wider circles of communities (in-person and online) across Lebanon who the Alwan students interact with, during and long after the end of the project. The ultimate beneficiaries thereby include community members, including those who benefit from the students’ community service activities; other students at schools running Alwan activities but not directly enrolled in the programme; parents of Alwan students; principals and teachers at schools where Alwan is implemented; and followers on Alwan social media pages, including Alwan blog/related social media readers.

The direct beneficiaries of the project activities are the high-school students, in Grades 10 and 11, who undertake the Alwan programme and implement its activities (in-person and online4) in their own communities and social spheres. For the 2020-2022 Alwan cohort, this involves 375 Alwan students.

The programme also directly benefits the educators6 (25) and coaches7 (5) who undergo training and coaching on the programme’s materials, and how to manage teaching and monitoring (currently including online). Direct beneficiaries are also the schools (25 for the 2020-2022 cohort) who embed Alwan curriculum and clubs in their education system.

The main participants of the project’s learning activities are the 375 Alwan students, who form the 25 Alwan clubs, undergo the learning activities in Year 1, and the community service activities in Year 2. The 25 Alwan educators are also main participants, as they lead the clubs and the two-year learning programme, with support by the 5 coaches.

Programme Theory of Change & Outcomes
To note: this Theory of Change and the below outcomes were developed in 2020, for the 2020-2022 cohort. The logframe is included as Annex I. Previously, there was no Alwan Theory of Change or logframe.

Theory of Change:
IF young high school students from diverse communities across Lebanon have opportunities to increase their intercultural and interreligious knowledge, skills, behaviours and experience, via a structured program led by trained educators, involving inter-club activities and community service projects;
THEN they will be able to better engage, communicate and collaborate with others, enabling them to be positive change-makers in their communities who represent and promote active and inclusive citizenship;
SO THAT growing circles of people in Lebanon are empowered and supported to be sustainable ambassadors for social cohesion and peaceful coexistence, helping to overcome sectarianism in Lebanon and instead sustainably embed values of diversity, solidarity and human dignity.

Outcomes:
1. Online Alwan programme addresses the evolving situation in Lebanon, including the necessity for digital/blended learning
2. Improved knowledge, attitude and behaviour among Alwan students towards religious diversity and active and inclusive citizenship
3. Improved intercommunal partnership between young students, and increased positive engagement with communities across Lebanon
4. Increased and sustainable engagement within inclusive citizenship and peacebuilding initiatives by a strengthened national Alwan community, and by members of the public.

Evaluation Purpose
We require a team of evaluators who can accompany us and a core team of programme partners on a collaborative journey to better understand the impact Alwan has made on participants, and by extension what works and doesn’t work in terms of inter-religious dialogue/social cohesion methods.

This knowledge, and the process of gaining this knowledge, will allow Adyan and partners to design more effective projects, including future iterations of Alwan (cohort 2022-2024 and cohort 2023-2025 will be designed during the timeframe of this evaluation). We also expect the evaluation to propose methods for ongoing programme data and evidence collection, including a system for sustainable communication with former participants.

We expect the evaluation to also include a needs assessment, to ascertain relevance of the programme and to identify gaps in programme curriculum and implementation.

We also expect the evaluation to involve a stakeholder mapping, and to suggest engagement strategies for each of the core target groups and important influencers for the programme. This should also outline relevant stakeholder groups not yet reached, and proposed strategies for their engagement. This should include potential donors as well as government institutions.

The evaluation findings will also be of value to other stakeholders working in the fields of inter-religious dialogue and education.

It is of particular importance for Adyan to have this understanding now, in light of three factors. Firstly: the huge societal changes Lebanon has been undergoing over the past 18 months, with anti-government protests, increased sectarian narratives from political representatives, political instability, and the societal impacts resulting from the financial crisis, COVID-19 pandemic and the 4 August explosion in Beirut. These have had a negative impact on children’s learning, with schools closed for an extended period of time. They have also exacerbated sectarian tension in the country.

Thus is it crucial to understand how Alwan can continue to address and promote social cohesion, at what may be a pivotal time for the country. Through an evaluation, Alwan can bring a new perspective to the interrogation and operationalisation of the core concepts of active citizenship, inclusivity, and interreligious dialogue. It will provide preliminary answers on what elements of interreligious dialogue are working well within the surrounding conditions and what elements need to be further developed, changed or disseminated through different delivery methods.
The second factor is to improve Alwan’s sustainable funding situation, by assessing and re-designing Alwan, and with the evaluation data as proof of its impact and potential. This could also facilitate the scaling-up of the Alwan program within Lebanon. (Currently Alwan has funding for the 2020-2022 cohort, but only for Grade 10 in 2020-2021, and grade 11 in 2021-2022. Adyan is currently seeking funding to start a new cohort 2021-2023, and to scale-up to more schools.)

The third factor is the increased interest in Alwan from other countries, with educational organisations showing an interest in adapting Alwan to their own contexts. An impact evaluation would provide needed information about program strengths and weaknesses, enabling other stakeholders to replicate and/or build upon the lessons of Alwan, and/or for Alwan to be rolled-out elsewhere by Adyan. This would greatly increase the reach of Alwan and its model of inter-religious dialogue within a school system, and would enable Adyan to share lessons with others, supporting a community of practice.

Two main target audiences of the evaluation are:
Internal audience
1. Adyan professional team, located in Beirut, Lebanon
2. Adyan Executive Board
External audience
3. Donors and potential donors
4. Inter-religious dialogue partners and potential partners (national and international)
5. Other citizenship/peacebuilding/diversity education NGOs (national and international)
6. National education stakeholders
Evaluation Objectives
The main objectives of the evaluation are as follows:
1. Assess contribution Alwan has made, from its inception in 2007 until present day, on changes in the attitudes, behaviour, skills and lives of participants, including former and current students, educators, partner schools and communities.
2. Assess the impact of year 2 of the Alwan cohort 2020-2022, which involves Grade 11 from September 2021 – May 20228.
3. Capacitate the staff of Adyan in evaluation techniques and M&E methodologies and tools, leading to a strengthened system for programme refinement and continued monitoring, evaluation and knowledge-sharing.
4. Refresh the stakeholder map for the programme, including institutions and individuals and issues of power and interest in the programme.

Timeframe
The evaluation will consist of three phases:
o May 2021 – August 2021: Preparatory phase, including finalisation of evaluation methodology, inception workshop with the team, refinement of Theory of Change, and delivery of inception report
o September 2021 – August 2023: Evaluation of Alwan programme, including backwards looking from 2007 and forward-looking9 of Year 2 of Alwan Cohort 2020-2022
o September 2023 – October 2023: Preparation of final report
The final evaluation deliverables, such as summative evaluation report and audio-visual materials, would follow within approximately two months of programme completion, in order to give enough time for data collection, analysis and writing.

Questions
We have identified evaluation criteria questions, at the macro level, to guide the evaluation of this programme.
Key Questions:
1. Does the intervention address relevant key causes and drivers of inter-communal tension?
2. To what extent has/is the Alwan programme made/making a positive impact on the attitudes, behaviour, skills and lives of participants, including former and current students, educators, partner schools and communities
3. What changes in attitudes, behaviours, relationships or practices can be ascertained?
4. What major factors contribute to the achievement or non-achievement of objectives?
5. Is the intervention achieving its stated (or implicit) purpose, or can it reasonably be expected to do so?
6. What is the programme’s contribution within a wider context, for example on school policy, collective of schools and/or national education policy?
7. What were the roles of and relationships to other stakeholders and how can these be improved?
8. What systems can be improved/put in place for improved programme M&E, to inform programme refinement, understanding of impact and knowledge-sharing?

Evaluation Questions:
Relevance

• Is the intervention based on a valid analysis of the situation of limited inter-religious understanding/awareness in Lebanon? If yes, how? If no, why?
• Are there other communities who are not targeted and/or addressed via the programme who should be (for example refugee populations)?
• In what ways has the intervention been flexibly adapted to updated analysis over time?
• In the light of the recent political instability and social upheaval/trauma in Lebanon, is the intervention working on the right issues in this context at this time?
• What is the relevance of the intervention as perceived by the local population, beneficiaries and external observers?
• Do activities and strategies fit objectives, i.e. is there internal coherence between what the programme is doing and what it is trying to achieve?
Effectiveness
• Is the theory of change based on valid/tested assumptions? Are there alternative theories of change?
• Is the intervention achieving progress within a reasonable time frame, or will it do so?
• Is the intervention achieving its stated (or implicit) purpose, or can it reasonably be expected to do so? If yes, how?
Impact
• What are the primary and secondary, direct and indirect, positive and negative, intended and unintended, immediate and long-term, short-term and lasting effects of the programme? Does it exert a significant effect on key factors for peace and social cohesion in Lebanon?
• What elements of interreligious dialogue are working well within the surrounding conditions and what elements need to be further developed or disseminated through different delivery methods?
• What are the practical and strategic implications of what we have learnt?
Sustainability
• Which steps have been taken or are planned to create long-term processes, structures, norms and institutions for education on diversity in Lebanon?
• What similar initiatives exist around the world from which lessons can be drawn in order to enhance and scale up the programme, both within Lebanon and to other countries?

Evaluation methodology
Since the intention is to look at both the past and the present, two interventions are needed: 1) the retrospective evaluation, and 2) the impact evaluation.

We anticipate that a quasi-experimental approach, undertaken in a participatory manner, would be an appropriate methodology. Overall, we would anticipate that the evaluation approach would mix qualitative and participatory data with quantitative mechanisms and methodology. It will be important for the bidder to propose and to consider who data is collected from.

We invite bidders to suggest an evaluation methodology that they deem most appropriate on the basis of the information provided in this Terms of Reference and the Evaluability Assessment Report. As stated above, the Evaluability Assessment Report will be shared with all evaluators who submit an Expression of Interest to the contact details listed above.

During the inception phase, once a bidder has been selected, the methodology will be finalised in a collaborative process between the evaluator, Adyan, and other key stakeholders.

The main steps of any evaluation approach proposed would need to include the following:
1. Inception phase:
o Finalising evaluation methodology
o Reviewing and refining the Alwan Theory of Change
We would expect the evaluator/s to facilitate an inception workshop to introduce the evaluation process, and to sense check and refine (as needed) the theory of change.
o Building an evaluation and results framework
The evaluator/s will be required to develop indicators and means of verification.
2. M&E capacity-building of team:
Training will be designed collaboratively with the Adyan team, to ensure it is of maximum relevance to the programme team. It is likely that training topics will include i) designing and refining relevant and achievable M&E frameworks for each successive Alwan cohort; and ii) specific tools such as surveys; iii) tools for M&E of online attitudinal change initiatives
3. Mid-term stocktaking:
At the mid-way point of the evaluation, the evaluator will be required to prepare a mid-term report on progress made so far based on the key evaluation questions indicated above. The mid-term review could also suggest possible further refinement of the theory of change. This will be presented and discussed during a workshop with the Adyan team.
4. Endline and final report session:

Presentation and discussion on the endline and final reports with the Adyan team.

Stakeholder Involvement
Adyan is commissioning this evaluation and as such will be responsible for managing the evaluation overall. The evaluator/s will report to Adyan. However, the donor is a key stakeholder in the delivery of this evaluation and the programme itself and therefore we expect them to be consulted and engaged throughout the evaluation of the programme, from start to finish. Given we have not yet worked out the exact remit, roles and responsibilities of each partner, we are unable to say with certainty how we will work together on this evaluation, but it is likely that we will have the following remits:

STAKEHOLDER & TYPE AND LEVEL OF ENGAGEMENT WITH EVALUATION

ADYAN: Commissions evaluation; Oversees evaluation; Evaluator will be held responsible to Adyan for submitting deliverables and will be first point of contact for any management/ delivery issues or issues of concern

ALWAN EDUCATORS (CURRENT & FORMER): Participate in workshops to provide input into programme theory of change and evaluation data collection plans

ALWAN PARTICIPANTS (CURRENT & FORMER); ALWAN SCHOOL REPRESENTATIVES; FAMILIES & COMMUNITIES OF ALWAN PARTICIPANTS: Participate in providing data and insights

MINISTRY OF EDUCATION * HIGHER EDUCATION: Participates in providing data and insights

DONOR: Review and assess proposals received (Adyan will make final decision); Will input and comment on evaluation methodology proposed, including dissemination of learning outputs, throughout programme development

Evaluation Team
Between the members of the team, the evaluator/s must:
- Be independent of the programme and have no stake in the outcome of the evaluation.
- Have a proven track record in delivering high quality and complex evaluations in the peacebuilding, inter-religious dialogue, education and social cohesion fields and be up to date with key concepts and developments in these fields

- Have knowledge and awareness of the nuances of the social and political context in Lebanon and the current conflict drivers. Some team members should have fluency in written and spoken Arabic. If evaluation team members have close links with/are based in Lebanon, it is of course imperative that they are able to maintain neutrality, objectivity and independence.
- Be sensitive to and respectful of social, cultural and religious differences, and comfortable with working and communicating across difference, including able to communicate complex ideas in accessible ways and gather data from and with partners, grantees and beneficiaries across society
- Ideally, have experience of evaluating online peacebuilding/inter-religious dialogue initiatives
- Be skilled in group facilitation and delivering M&E training, including online if necessary
- Be enthusiastic about learning from what does not work as well as what does.
- Be fluent in English with capacity to prepare written reports in professional-level English
 

Deliverables and standards
We would expect the following deliverables (all in English):
- A baseline study (with the exact nature depending on the chosen methodology).
- Two interim evaluation reports following the inception phase (which is expected to finish by end August 2021). The first at the mid-point (June 2022) and the second at the start of the final calendar year (Jan 2023). These will be for internal use of the core partners and should document key learning about process as well as emerging findings or results based on output/outcome data so far.
- An inception workshop, mid-point workshop, and end of programme workshop with the Adyan team (including Adyan staff and Alwan educators).
- A full final summative outcome/impact report (for an internal audience) and a summary final evaluation report, aimed at an external audience and of publishable quality. These would be delivered within two months after programme completion (meaning end of October 2023). See Annex II for minimum requirements. It is expected that at least one draft of each report will be shared with Adyan for review and feedback, before final versions are produced.
- We expect that infographics (likely a short infographic film) would be explored and used to communicate the key findings of the evaluation. These would be delivered within two months after programme completion (meaning end of October 2023).

We are open to additional proposals from the evaluator/s on the exact format of the deliverables and on additional or alternative recommendations.

Budget and Duration of the Evaluation
The duration of the evaluation is approximately 30 months, starting May 2021 and finishing in October 2023.
We would like the evaluator/s tendering for this work to propose a budget and a time breakdown that fits the methodology they propose. The budget for the evaluator will cover travel to Lebanon, accommodation and travel within Lebanon, translation of any reference materials, facilities and logistics for workshops (including simultaneous translation), any evaluation service providers and the deliverables request and proposed. These can be included as lump sums in the budget.
We would expect the total evaluation budget, covering the duration of the whole evaluation, to fall within the range of €100,000 - €150,000 Euros. However, please note this range is an indication only; we would expect the budget to vary according to the methodology and deliverables proposed.
We will assess proposals according to how well the proposed methodology meets the requirements set out in this Terms of Reference and the expertise and experience of the evaluator/s. As already noted, we are open to alternative or additional suggestions to the methodology proposed above if a convincing case is made and so long as they demonstrate they can answer the key evaluation questions.

Submission Process and Selection Timeline:
Expressions of interest and full proposals should be submitted to Caroline Thomas, Director of Development & International Relations, at [email protected]. Expressions of interest and full proposals must be received by the deadlines stated above.

We would ideally like to have a conversation with those shortlisted for the evaluation in the week commencing 10 May 2021, with a view to appointing an evaluator in the week commencing 17 May 2021.

 

Annex I: Final impact/outcome evaluation report
We would expect a final, full evaluation report, addressing the Key Evaluation Questions, of between 40 – 60 pages and a final summary report of approximately ten pages. Both should be of publishable quality. We are interested in reports, and alternative mediums that accompany the report, which are able to engage a wide audience. We would expect to further discuss and refine the minimum components of the final report and any accompanying deliverables at a later date but the following guidance provides an indication of our requirements.

Executive summary (no more than 5-10% of full report length)
• Summary of the evaluation, with particular emphasis on main findings, conclusions, lessons learned and recommendations
Introduction
• Presentation of the evaluation’s purpose, questions and main findings
Intervention
• Description of the evaluated intervention, and its purpose, logic, history, organisation and stakeholders
Evaluation methodology
• Approach taken, methods and instruments used, as well as limitations to the evaluation
Findings
• Factual evidence, data, observations relevant to the specific evaluation questions
Conclusions
• Assessment of the intervention and its results against given evaluation criteria, standards of performance and policy issues
Lessons learned
• General conclusions that are likely to have a potential for wider application and use
Recommendations (generally no more than six)
• Actionable proposals to the evaluation’s users for improved intervention cycle management and policy, including their justification
Annexes
• Terms of reference, detailed description of methodology for data gathering and analysis, references, people interviewed, etc.

 

 

 

 

 

 

How to apply

Submission Process and Selection Timeline:
Expressions of interest and full proposals should be submitted to Caroline Thomas, Director of Development & International Relations, at [email protected]. Expressions of interest and full proposals must be received by the deadlines stated above.
We would ideally like to have a conversation with those shortlisted for the evaluation in the week commencing 10 May 2021, with a view to appointing an evaluator in the week commencing 17 May 2021.

Expired
Deadline
Friday, 30. Apr 2021
Type of Call
Call for Proposals
Intervention Sector(s):
Education
randomness