Overview
This Entry indicates when evaluations of emergency operations are conducted and provides general guidance and information on evaluations.
UNHCR applies the definition adopted by the United Nations Evaluation Group (UNEG) in its 2016 Norms and Standards for Evaluation:
…an assessment, conducted as systematically and impartially as possible, of an activity, project, programme, strategy, policy, topic, theme, sector, operational area or institutional performance. It analyses the level of achievement of both expected and unexpected results by examining the results chain, processes, contextual factors and causality using appropriate criteria such as relevance, effectiveness, efficiency, impact and sustainability.
More generally, evaluations should provide answers to the following key questions:
- Are we doing the right thing?
- Are we doing the right thing on a scale that will make a difference in the lives of persons of concern?
- How well have our interventions been and how do we know this?
- What results have been achieved?
- Are there better ways to achieve those results?
- To what extent can a given result be attributed to a specific intervention?
An evaluation that answers the questions above in a credible and timely manner, informed by evidence, and which focuses on use of the results, can positively influence planning, design, budgeting, implementation and reporting. Evaluations contribute to evidence-informed programming, policy-making and organizational effectiveness, and strengthen results-based management in UNHCR.
UNHCR evaluations occur at two levels:
Centralised evaluations are commissioned and managed by the Evaluation Service (ES). They primarily focus (a) on policies, strategies, programmes and themes of corporate significance at global, strategic and institutional level, and (b) on Level 3 emergency operations.
Decentralised evaluations are commissioned and managed by Divisions, Regional Bureaux or regional and country offices. They primarily focus on activities, themes, sectors, operational areas, programmes and projects at regional or country level.
Distinct but complementary
Evaluation is distinct from, but complements, other oversight functions, such as audit and inspection, which focus more on risk management, control and governance objectives, compliance with UNHCR policies and regulations, and risks.
Monitoring and evaluation are often linked. However, evaluation also differs from (continuous) monitoring and other forms of assessments.
Evaluations are expected to generate credible evidence that is sufficient to answer questions about implementation and process, and also questions about the relevance and coherence of interventions, their adaptation to context, cause-and-effect concerns, and their contribution to results.
Main guidance
When and for what purpose
When
The Policy on Emergency preparedness and Response (UNHCR HCP 2017-1) stipulates that within 18 months (or earlier if so decided by the High Commissioner), the Evaluation Service commissions, oversees and completes an evaluation of each UNHCR Level 3 emergency operation. Evaluations of Level 1 or Level 2 emergencies may also be commissioned in accordance with UNHCR's policy on Evaluations.
Note: The relevant Bureau(x) and DESS conduct a real-time review of each Level 2 and Level 3 emergency operation after three months, by means of a joint mission or workshop. Reviews involve the key stakeholders and assess the timeliness, appropriateness and effectiveness of UNHCR's operational response. They recommend adjustments and course corrections that may be required. They also monitor the application of UNHCR's policy on emergency preparedness and response.
For humanitarian system-wide L3 emergencies, an Inter-Agency Humanitarian Evaluation (IAHE) is typically conducted some 9 to 12 months after a system-wide L3 emergency is declared. The Emergency Relief Coordinator (ERC) initiates IAHEs on behalf of the IASC, as advised by the Emergency Directors Group (EDG). The Evaluation Service often contributes and participates in the management of IAHEs.
Regional Bureaux or Regional and Country Offices may also decide to initiate and commission a decentralized evaluation of L1 or L2 level emergency responses. In such cases, they should inform the Evaluation Service, who will provide expertise, support and advice, and review draft reports for quality assurance purposes.
For what purpose
The overall purposes of an evaluation are to support organizational accountability and learning and contribute to the continual improvement of UNHCR's performance in addressing the protection, assistance and solutions needs of refugees, stateless persons and other persons of concern.
In practical terms, evaluations objectively assess results and generate insights into how operations, policies and programmes are performing.
Summary of guidance and/or options
UNHCR's evaluation policy has four overarching principles. These guide best practice in planning, design and delivery and, taken together, provide a comprehensive framework for UNHCR evaluations.
Independence. Independence underpins the legitimacy of evaluation work by reducing the potential for conflicts of interest that might arise if managers and policy-makers were solely responsible for evaluating their interventions. The principle covers both behavioural and organisational independence.
Impartiality. Independence requires impartiality, which in turn depends on the professional integrity of evaluation managers and evaluation teams, as well as the absence of undue influence that might create bias.
Credibility. Evaluations must be credible if intended users are to act with confidence on their results, and take steps to incorporate the learning they generate in policy, advocacy, programming, decision-making, and implementation processes.
Utility. Whenever an evaluation is initiated, there should be a clear, specific, and well-communicated intention to apply its analysis, findings, conclusions and recommendations. This is essential if evaluations are to inform decision-making processes and contribute to operational and programmatic accountability, learning and improvement.
Management of an evaluation
The Evaluation Service commissions and manages centralized evaluations, and offers quality assurance, expertise and support to decentralised evaluations.
Preparation
An evaluation manager is appointed. For centralized evaluations the manager is an evaluation officer of the Evaluation Service. In decentralized evaluations, the manager is a staff member of the UNHCR entity that commissions the evaluation; the Evaluation Service provides support and advice as required.
The evaluation manager has important responsibilities throughout the different phases of an evaluation process, and is the key interlocutor for the evaluation team.
UNHCR's approach to evaluation normally includes the phases described below:
Phase 1 - Preparation and terms of reference
This phase focuses on the evaluation's initial and preparatory steps, namely clarifying its scope, defining its purpose and objectives, and agreeing the key questions that will be asked. This information is used to develop terms of reference (ToR), which are published to seek expressions of interest from individual consultants or firms. Bids from potential consultants or firms are reviewed and the evaluation team is selected.
Phase 2 - Inception
Phase 2 focuses on finalizing contract details, analysing background materials, and preparing and executing inception visits (where applicable). The team also lays the groundwork for data collection and drafts and finalizes its inception report.
The inception report is essentially a tool that enables the evaluation team and UNHCR to agree on how the evaluation should proceed.
Phase 3 - Data collection and early analysis (including field mission)
The evaluation team gathers data from multiple sources, and uses different methods and approaches to assemble and analyse evidence relevant to the evaluation questions. Field missions take place during this phase.
Phase 4 - Drafting the report
The team analyses its data and prepares, reviews and revises a draft evaluation report. This phase ends when the team completes its main deliverable, that is to say when the final report is approved. In the case of centralized evaluations the report is approved by the Head of the Evaluation Service, and in the case of decentralized evaluations by the senior manager who commissioned the evaluation.
Phase 5 - Dissemination and management response
The evaluation report is sent to key stakeholders. Management drafts a response. The management response summarizes stakeholders' responses to the evaluation's findings and recommendations, and includes planned follow-up actions. Both the evaluation report and the management response are published online.
How to implement this at field level?
When a centralized evaluation of an emergency operation is planned, the Evaluation Service informs the Representative(s) and liaises closely with the regional bureau concerned.
Evaluations require the support of both Representative(s) and the country office(s) concerned. They are requested to:
- Inform all relevant parties in advance (UNHCR colleagues, UN and NGO partners, government, donors), introduce the evaluation team, and explain that staff and partners may be approached for interview.
- Allocate a focal point on substance, as well as for logistical and administrative support.
- Make available to the evaluation team documents that capture key developments, decisions and results relating to the emergency operation.
- Assist the evaluation team to set up interviews with key stakeholders. Interviews should include persons of concern and should adopt an age, gender and diversity (AGD) approach.
Representative(s) and country office(s) may also be requested to provide logistical support (subject to operational constraints).
Annexes
Links
Main contacts
Evaluation Service at Headquarters:[email protected]