External M&E Impact Evaluator

RDPP/LGA – End of Project Evaluation

Terms of Reference (ToR) / Evaluation scope

This document (TOR/evaluation scope) outlines the scope of work, intended work plan, and analysis, providing the opportunity to verify the same understanding of the evaluation objectives among all stakeholders. This document details our understanding of what is being evaluated and shows how each evaluation question will be answered using proposed methods, proposed sources of data, and data collection procedures. The inception report should include a proposed schedule of tasks, activities, and deliverables, designating a team member with the lead responsibility for each task or product. 

 

LGA Program Overview 

The Lebanon Growth Accelerator Program is an AltCity Impact/Bloom initiative to help boost Lebanon’s enterprises and the entrepreneurship ecosystem. It aims to create employment through enterprise accelerator programs run by AltCity Impact/Bloom and partner accelerators, enhancing ecosystem collaboration, developing shared training resources, and improving ecosystem capacities. The goal is to improve short-term and long-term opportunities for Lebanese and refugees, focusing on vulnerable and marginalized communities.

The LGA Program is supported by the European Regional Development and Protection Programme for Lebanon, Jordan, and Iraq (RDPP II). 

 

The “theory of change” for this project 

By building the capacity of national partners to deliver high-quality training, mentorship, and support to MSMEs that bridges geographical barriers, and by increasing the opportunities for discussion, community exchange, and support through an online community of practice, MSMEs will have increased capacity to maintain and create new employment opportunities leading to new income opportunities for refugees, vulnerable, and marginalized communities. 

 

Program Description 

AltCity Impact/Bloom will organize a series of activities to help MSMEs maintain and create new employment opportunities leading to new income opportunities for refugees, vulnerable, and marginalized communities. These activities include a set of direct training and support activities (2 cycles led by the AltCity Impact/Bloom team, supporting 10 MSMEs each cycle) and six cycles provided by Training Partners with the support provided by AltCity Impact/Bloom  (supporting 5 MSMEs each cycle). 

These activities include versions of “best practice” programs used by the accelerator and SME training programs worldwide, adapted for the current context in Lebanon, and enhanced by new methods and approaches developed by the AltCity Impact/Bloom team based on its experience and research, and supported by external research. 

These activities include: 

Sprint Programs: Intensive short programs (5-12 days) of training and support designed to help participating MSMEs improve essential skills (communications, leadership, collaboration, etc, “soft skills” that support entrepreneurship and enterprise success) and get light enterprise development support (in core business domains, marketing, financial planning, etc) while helping AltCity Impact/Bloom and Training Partners assess MSMEs for readiness and appropriateness for selection to the next stage Accelerator Programs. 

Accelerator Programs: Intensive medium-term programs (3-4 months) of training and support, along with funding support, to help participating MSMEs reach a key business objective identified by the MSME in collaboration with AltCity Impact and the Training Partners that is deemed to specifically support the MSME in their effort to maintain and create new employment opportunities leading to new income opportunities for refugees, vulnerable, and marginalized communities. 

This enhanced training and mentorship provided by both AltCity Impact/Bloom and the Training Partners work to bridge geographical barriers, to help MSMEs have increased capacity to maintain and create new employment opportunities leading to new income opportunities for refugees, vulnerable, and marginalized communities.

The two outcomes, along with multiple outputs, work together symbiotically: 

The direct enterprise support programs enable AltCity Impact/Bloom to have a direct impact while continuing to refine the methods, content, and tools used in the programs; 

The support of Training Partners improves the scalability of impact during the program by enabling support for a larger number of times and beyond the duration of the program by enhancing capacities with the ecosystem; and

The Community of Practice engages the broader ecosystem to support both of those elements above, along with the participating enterprises themselves. 

 

Program Outcomes 

OUTCOME 1: An effective community of practice (Training Partners and MSMEs) with improved capacities supports MSMEs to operate in complex economic environments

Output 1.1: A community of practice and platform activated for collaborative discussions and learning that supports MSME growth across Lebanon

Output 1.2: Training Partners with enhanced capacities to deliver MSME support using AltCity Impact/Bloom’s methods

OUTCOME 2: Capable MSMEs operate successfully within complex economic environments.

Output 2.1: MSMEs with Increased Capacities from Sprint and/or Accelerator Programs Provided by AltCity Impact/Bloom

Output 2.2: MSMEs with Increased Capacities from Sprint and/or Accelerator Programs Provided by Training Partners

OUTCOME 3: AltCity Impact/Bloom with strengthened organizational abilities to deliver sustained and growing economic impact for MSMEs in complex environments.

Output 3.1: Organizational Capacity Building is strengthened

 

Project Objective

The objective(s) of the development cooperation among the parties is to have MSMEs capable of maintaining and creating new employment within complex economic environments with the support of national partners.  

The focus of all those activities is to improve job opportunities for the target communities of refugees and host communities both as Direct Beneficiaries and as Indirect Beneficiaries.

 

Project Specific Objective

A core consideration of this project is developing a structure that continues providing value and growing beyond this project’s duration, including benefiting the MSMEs that went through the program, other MSMEs, and the broader ecosystem. This is a core part of the design for the “community/ecosystem building” activities, which will continue after the project ends. Additionally, the project’s skills development and character-strengthening elements will provide continued long-term benefits to the MSMEs, and access to the platform and mentors can continue self-directed after the project ends. 

 

Purpose of the Evaluation

Scope and Focus of the Evaluation

To evaluate the implementation and achieving the program’s outcomes, AltCity Impact/Bloom is recruiting an external M&E Impact Evaluator to provide diligent insights on the program’s impact in relation to the different resulting outcomes from the implemented approach and mitigation measures. The evaluation will cover the entire program implementation period, particularly to AltCity Impact/Bloom’s improvement of best practices following the mid-program evaluation completed on January 31, 2022. The evaluation should also assess the Theory of Change and whether the fundamental assumptions supporting it are still valid in light of the changing circumstances. The evaluation should also provide project execution outcomes, recommendations, and lessons learned. It should also make specific recommendations to help AltCity Impact/Bloom execute future development programs that incorporate MSME support and collaboration with partners within the ecosystem.

 

The Evaluation Questions 

The following is an example of key questions that will guide the project evaluation: 

 

i) Relevance 

How well has the project aligned to the objectives, in addition to the priorities and requirements of the target beneficiaries in terms of income access?

Operating within a challenging economic situation, to what extent were the mitigation measures implemented successful in achieving the overall outputs and outcomes of the project for supporting MSMEs’ operational capacity in complex economic environments (including contributing factors and constraints)?

As a result of the program support, are the “new jobs” created or pre-existing jobs filled by beneficiaries? 

Did the “Sprint” phase of the program, solely without the “Accelerator” phase of the program, have any contribution/impact on the new jobs created?

In what capacities were the “new jobs” created filled by vulnerable communities from refugees and/or host populations as initially intended?

To what extent are the new jobs created sustainable and meet “decent work” conditions?

What are the different “decent work” attributes that were taken into account by different beneficiaries? Were there any challenges in meeting those attributes? Any successes to be highlighted?

Were the inputs and strategies identified, and were they realistic, appropriate and adequate to achieve the results?

Did the project cater to the external challenges, and were they adequately addressed? 

How effective has the project been in responding to the needs of the beneficiaries, and what results were achieved?

 

ii) Effectiveness

Describe the management processes and their appropriateness in supporting delivery

To what degree did beneficiaries’ access to income change due to different approaches? Which approach had the most positive impact, and why? Were there differences between refugees and the host community?

What were the positive and negative, intended and unintended, changes produced by the project? (examples of positive changes could for example be businesses creating more jobs than intended, or growing beyond the plan outlined in the business plan etc.)

To what extent did the Project’s M&E mechanism contribute to meeting project results?

How effective were the strategies and tools used to implement the project? 

How cost-effective is the program implementation? What cost did the program incur to create a new position (including HR costs etc.)?  

What are the future intervention strategies and issues?

 

iii) Efficiency of Project Implementation

How did various change techniques affect effectiveness and adaptation? What elements contributed to this?

Specifically, did the actual or expected results (outputs and outcomes) justify the costs incurred? Were the resources effectively utilized? 

Did project activities overlap and duplicate similar interventions (funded nationally and /or by other donors? 

Are there more efficient ways and means to deliver better results (outputs and outcomes) with the available inputs?

How was the project’s collaboration with national MSME training partners? What are some possible areas of improvement?

What are the variances (if any) in program implementation between AltCity Impact/Bloom and national training partners, including in number of jobs created, quality of services delivered, etc?

How efficient were the management and accountability structures of the project?

How did the project financial management processes and procedures affect project implementation?

What are the strengths, weaknesses, opportunities, and threats of the project’s implementation process? 

How efficient and effective was the online implementation of the program in comparison to in-person implementation in reaching the target audience and meeting objectives? 

 

iv) Sustainability

What are the key factors/areas that will require additional support/attention to improve prospects of sustainability of the project outcomes and the potential for replication of this approach?

How efficient and sustainable was using AltCity Impact/Bloom’s online platform, tools, and library for MSMEs capacity building? 

What major factors influenced the achievement or non-achievement of sustainability of the business supported and project outcomes?

How effective were the exit strategies and approaches to phase out assistance provided by the project, including contributing factors and constraints?

Describe key factors that will require attention to improve prospects of sustainability of Project outcomes and the potential for replication of the approach.

To what extent has the project contributed to building the sustainable capacity of AltCity Impact/Bloom, influencing the effectiveness, quality, and sustainability of project activities and results?

What are some of the key lessons learned as a result of this project that can be shared and replicated?

What are the recommendations for similar support in the future? 

(Nb. The recommendations should provide comprehensive proposals for future interventions based on the current evaluation findings)

 

v) Engagement Cross-Cutting Principles

Assessing the extent to which the project is deepening localization of support and engaging with all relevant stakeholders, being supportive of innovation, and employing a human rights-based approach.

To what extent is the project working towards creating a supportive community of practice within a broader entrepreneurship ecosystem?

To what extent is the project deepening support localization and engaging with all relevant stakeholders?

To what extent does the project support innovation with the outcomes and potential scale-ups?

To what extent is the project employing a human rights-based approach in terms of: 

Participation

Accountability

Non-discrimination and equality

Empowerment 

Legality

 

Methodology for Evaluation

For the evaluation to provide quantitative and qualitative data, the following methods might be used for a representative sample that includes internal/ external stakeholders and direct/ indirect beneficiaries: 

Desk study and review of all relevant project documentation, including project documents, annual work plans, project progress reports, annual project reports, reports of the project steering committee, similar interventions, and approaches.

In-depth interviews to gather primary data from key stakeholders using a structured methodology 

Focus Group discussion with project beneficiaries and other stakeholders.

Interviews with relevant key informants (see attached list of relevant institutions) 

Observations (field visits using checklist)

 

Duration of the Evaluation

The evaluation is expected to start mid of October 2022 and conclude by mid of December 2022, for an estimated date to finalize data collecting, analysis, drafting, and holding the feedback and communications of the draft report, and finally, the final product finalizing date.

 

Expected Deliverables

An initial inception report is to be submitted for review and feedback by the donor and key stakeholders before the implementation of the evaluation. The report needs to include the consultant’s understanding of the TOR, detailed methodology of implementation, work plan, and proposed team members.

A draft report of preliminary findings, prior to the final report, that will be reviewed by the donor and key stakeholders; including the consultant’s understanding of the TOR, detailed methodology of implementation, work plan, proposed team members, and preliminary findings. The report is also to be discussed with respective parties in a preliminary findings meeting.

The content and structure of the final analytical report with findings, recommendations, and lessons learned covering the scope of the evaluation usually include the following:

Executive summary   

Introduction  

Description of the evaluation methodology   

Situational analysis regarding the outcome, outputs, and partnership strategy  

Analysis of opportunities to guide future programming  

Key findings, including best practices and lessons learned  

Conclusions and recommendations  

Appendices: Charts, terms of reference, field visits, people interviewed, documents reviewed