Introduction

This collection of conference proceedings is based on papers which were presented at the Forum on Impact Studies organised by the Education Division of the Department of International Development between 24 and 25 September 1998. The forum brought together participants from a wide range of projects in India, South Africa, Morocco, Eritrea, Columbia, Mexico and Romania, as well as a number of specialists with experience in participatory approaches to project evaluation. Participants brought to the conference a profound theoretical understanding of participatory approaches to assessing impact - as well as the richness of their personal experience gained from many years of practice. DFID also welcomed representatives from the British Council, CfBT, Link Africa and Book Aid International, who provided a useful stakeholder dimension from a management and/or professional standpoint. The majority of the participants had been associated with Project Impact Assessment, which had been supported by DFID during the 1990s.

The Forum used the following definition of impact, taken from DFID's Glossary of Aid Terms (1998), as its point of departure:

Impacts (also referred to as effects) may be planned or unplanned; positive or negative; achieved immediately or only after some time; and sustainable or unsustainable.... Impacts may be observable/measurable during implementation, at project completion, or only some time after the project has ended. Different impacts may be experienced by different stakeholders.

The original programme was slightly modified after a planning meeting which some participants attended in March. Although there was broad consensus at the Forum about the advantages of following a participatory approach for assessing project or programme impact, it was also apparent that some of the defining characteristics of this kind of evaluation strategy raised questions that needed to be explored and answered by the main stakeholders in the exercise from the very outset. The following extracts highlight the issues that emerged most prominently, and focus on those factors which the Education Division needs to take into account when applying participatory approaches to the future evaluation of the impact of education projects or programmes.

Although participatory approaches to impact assessment were becoming increasingly common during the late 1980s, John Shotton pointed out in his keynote paper that the World Education for All Conference at Jomtien in 1990 marked a watershed in terms of local ownership and control in basic education programmes and that it included a substantial critique of donor- and lender-directed approaches to evaluation. The subsequent decisive shift towards wider stakeholder involvement, not only in project implementation, but also in project design, monitoring and evaluation, substantially altered the imbalance implicit in the donor/beneficiary or donor/recipient relationship. The implicit context of the DFID Forum was therefore the learning process which has impacted on all funding agencies in the nineties - an era described by Kenneth King (1991) as "the Post Jomtien curriculum" - and many contributors referred in one way or another to this transformation of the discourse.

The following themes were used to systematise the papers collected in this volume:

· What is an impact study and how should we do it?
· The role of base line studies in impact assessment
· The relationship between national and external researchers
· Training teachers as researchers
· Topicality vs. sustainability
· Impact assessment and sustainability
· Anticipated and unanticipated project benefits

While the final chapter attempts to draw together the conclusions drawn by the Education Division from the Forum debate, it should not be understood as (necessarily) reflecting the views of DFID as a whole.

SECTION 1: What is an impact study and how should we do it?

1.1
The collection opens with a paper entitled Participatory impact assessment by John Shotton. His paper sets the scene by answering the question What is an impact study? He then elucidates what is meant by a participatory study and in so doing, he contrasts traditional approaches to the assessment of impact by way of participatory approaches - approaches that have come of age in the nineties. Shotton's paper describes the changes in the theory and practice that are evidenced in the field of project impact assessment since the Jomtien Conference - a period which he terms "the post-Jomtien era". Shotton refers to some of the most far-reaching methodological innovations of the decade, such as the implementation of the ideals of local ownership and the development of local capacity through enabling participants to learn on the job. These changes necessitate a shift towards participatory methods, a shift which has radical epistemological implications for the assessment of projects.

1.2
Veronica McKay's paper elaborates on the epistemology underlying participatory approaches. She argues that participatory approaches have an educative function which cannot be replicated by traditional approaches to impact assessment. When located within an action research framework, she adds, participatory approaches offer substantial benefits to all evaluators and impact crucially on the development of local capacity. (She points out that participatory approaches are contingent on the discursive nature of knowledge -an assumption that underlies collaborative enquiry). McKay's paper, however, also draws attention to difficulties that participatory researchers may encounter when attempting participatory research in disadvantaged communities. She mentions - in particular - difficulties associated with enabling and motivating grassroot participation.

1.3
Sasidhara Rao too stresses the importance of the evaluation process being informed by a participatory philosophy. He refers to the way in which the Andhra Pradesh District Primary Education Programme was evaluated and argues that that methods and the instruments that were used for this assessment contributed in different ways to engaging participation at different levels and at different stages of the research enterprise. He indicates that participatory approaches encourage participants to reflect on their own contexts - and that this informs practice. Participatory approaches, used in this way, also mediate a formative function. He indicates moreover that the nature of participatory assessments helps to ensure that data - even quantitative data - are interpreted in context.

1.4
N V Varghese considers the distinction between an impact study and an evaluation. He outlines four main features which differentiate evaluations from impact studies. The distinction, he argues, has implications for who conducts the assessment and whose interests are most likely to be served. He indicates that while communities have more interest in the assessment of impact, the interests of funders lie in the evaluation of projects. These observations resonate with the critique of the donor-lead forms of assessment presented by Shotton in the first paper. Varghese makes a strong case for the use of participatory approaches by referring to the human condition. He argues that human volition compels researchers to use approaches that account for human experience and that participatory approaches, by their nature, take this into account.

While the Forum agreed broadly about the virtues of participatory research, it was nevertheless apparent that this form of evaluation is more complex than it might appear at face value. The subsequent sections in this collection explore these complexities.

SECTION 2: The role of baseline studies in the assessment of impact

2.1
This section opens with Carol Moloney's paper entitled School-focused baseline assessments as a catalyst for change. Moloney too draws on the participatory paradigm and argues that when applied to baseline assessments, the investigation fulfils an important developmental function for teachers. She refers in her paper to her experience in training South African teachers to do baseline studies and she argues that this exercise achieved benefits which go beyond the mere acquisition of data. Indeed, she indicates, teachers acquired a modus operandi for doing collaborative baseline studies while simultaneously coming to grips with many of the new demands that are being made on teachers.

2.2
Samir Guha Roy's paper offers a general framework for evaluating educational programmes. He indicates that although participatory approaches to assessment have gained ground over the past few years, he believes that there is still a need for scientific approaches to assessing impact. He argues that scientific approaches may be usefully employed in assessing those issues of impact which are difficult to assess since they may result from factors that are extraneous to programme activities. While firm baseline data and regular post-baseline checks could offer a way of overcoming this problem, Roy argues that, in the domain of impact assessment, there is a growing interest in scientific assessment as a way of statistically accounting for such changes.

2.3
Tony Luxon's paper is the final paper in this section. Luxon's paper, entitled Issues to consider when planning a baseline study, like Moloney's, deals with the importance of the baseline investigation as a way of ensuring adequate benchmarking. Luxon refers to the evolution of the philosophy informing the methodology used for baseline studies for projects in all types of social and educational contexts. It was this paradigm shift, coupled with his own experiences in the field of impact assessment, that prompted him to compile what he considers to be the twelve essential issues that need to be considered each time a baseline activity is contemplated. These twelve points (which affect all baseline studies) give rise to suggestions about the design of the research approach, the selection of members for the project implementation team and the dissemination of the findings of the baseline study to various stakeholders.

The concern with the stakeholders one that all contributors to the collection emphasised. The next section is devoted entirely to exploring issues related to stakeholder perspectives.

SECTION 3: Stakeholder perspectives

Who are stakeholders? What are their roles? How do we deal with multiple stakeholders with divergent interests? These are the kinds of questions that this section on stakeholder perspectives attempts to address.

3.1
This section starts with Dermot Murphy and Pauline Rea-Dickins's consideration of stakeholder perspectives. In this paper, they emphasise how important it is for evaluators to identify stakeholder groupings if they want to make effective use of participatory evaluations in educational development projects. They suggest that most definitions overlook dimensions of power and power differentials, and, as such, are inadequate. This claim underlies their view that what is needed is a framework which is more robust- a framework that takes issues of power into consideration. Murphy and Rea-Dickins present an outline of such a framework - for which they find support in their research - and they present an elucidation of the implications their findings might have for the practice of participatory research. In their discussion of stakeholders, they inevitably mention the role of the external consultant. In this regard, they coin a term FIFO consultants to refer to those consultants who rapidly fly-in/fly-out. They argue, in line with the emphasis that they place on participation, that a more sustained consultancy role needs to be factored into project designs. Their view of the role of the FIFO consultant has repercussions for insider-outsider involvement in participatory studies - an issue which is dealt with in detail in the fourth section of this collection.

3.2
Clara Inés Rubiano and Dermot Murphy in their paper entitled Considering the audience - an important phase in project evaluations, emphasise how important it is for evaluators to give consideration to those audiences for whom the evaluation is intended. They interrogate complexities associated with notions of audience, and refer to the differing interests, differing statuses as well as the differing power relations that are inherent to the concept of audience. They argue that the identification of, and consideration for, the audience/s is central to notions of the practical utility of the recommendations of an evaluation and to the compilation of evaluation reports. The authors draw on critical incidences pertaining to the audience/s, which manifest in the evaluation of the Colombian Framework for English Project.

3.3
In this paper Coco Brenes and Tony Luxon consider the variety of audiences that are implied by multi-partnered projects and the implications of the varied audiences for the dissemination of the project report, and for the mode of reporting. The paper considers the complexities implied by the reporting process, and in particular addresses issues such as: Who writes the a report? Who reads it? In what language is it produced? and How is it disseminated? Each of these questions is addressed against the backdrop of the ODA ELT Project in Nicaragua.

SECTION 4: The relationships between nationial and external researchers

4.1
Tesfai Bariaghaber's Note on a participatory impact study, which is the result of his personal involvement in Eritrea, explores the relations between national and external researchers. He begins his paper by stating that the assessment study was characterised by both highs and lows. Hence, while his paper acknowledges the many advantages of participatory impact assessment, he nevertheless refers also to some of the lows. The lows as described by him might be likened to the effects of FIFO (defined earlier in paper 3.1). For him, the lows are primarily a result of the geographic divide between the local and external evaluation team. He contends that in the case of the Eritrean assessment, the external evaluators moved out of the project too soon to allow them to make a meaningful contribution to the development of local capacity. Their early departure had ramifications for their stake in the ownership of the assessment project. The departure of the external consultants prevented the local team from being able to contribute further ideas or recommendations to the research report. He nevertheless concludes by indicating that collaborative research is beneficial to both internal and external researchers.

4.2
Harvey Smith and Paul Bennell also, as their paper title indicates, draw out the complexities associated with the Relationships between national researchers and external researchers. Their paper is based on their personal experience of impact studies in which they were engaged in Angola and Eritrea. As with paper 4.1, this paper describes a series of conceptual and practical project issues that impact on the relationship between internal and external researchers - in particular those that give rise to ownership-type problems. They argue that there is a need to achieve the "correct balance between local and external ownership". They believe that this correct balance can only be achieved by a research design that ensures that the study meets local needs, and that the external funding agencies are empowered to rate a project's achievements. In their conclusion they ponder the kinds of compromise that might be possible.

4.3
Mohammed Melouk, in his paper entitled The role of an insider/outsider in impact assessments, also explores complexities pertaining to the relationship between internal and external researchers. He bases his discussion on his experience of the Moroccan ELT project, and refers to some of the many complexities implied by the researcher's roles. He argues against the imposition of investigations or project designs without these being grounded in a solid local perspective. In highlighting the need for insiders to participate in the assessment of impact, he coins the phrase insider/outsider, which refers to those locals who are outsiders to the project - but who are insiders to its situational context. He outlines several good reasons why insiders/outsiders should be included in impact assessment teams - not only because they are communication facilitators but also because of their ability to mediate data and thereby contribute to insightful and contextually appropriate conclusions. In this sense he echoes Rao's sentiments in paper 1.3.

4.4
Dave Allan's paper is the final paper in this section. His paper, entitled Impact assessment in educational projects: some perspectives on the 'insider-outsider' debate, also stresses the importance of a consideration of the roles of insiders-outsiders in project assessment. These roles, he argues, have implications for who does the evaluating and who decides whether the outcomes are judged as either successful or not. In order to situate his own position in this debate, he draws on four case studies of evaluations undertaken in Bangladesh, Estonia and Morocco. These evaluations reflect a variety of permutations on a continuum from, on the one hand, being a single outsider researcher to, on the other hand, working as an insider with a range of insider-stakeholders. Allan makes various recommendations for good practice on the basis of his observations and critique.

SECTION 5: Training teachers as researchers

In this section, both Peacock's and Sekgobela's papers focus on training teachers to do assessments. They claim a wide range of benefits as a result of this training. Earlier papers also made some reference to teachers doing research: the papers of Moloney and McKay, for example, draw attention to the many benefits of this form of training.

5.1
Alan Peacock, in his paper entitled Helping teachers to develop competence criteria for evaluating their professional development, discusses interventions in South Africa and Sri Lanka which were intended to help teachers to develop competence criteria for evaluating their professional development. He elucidates various stages of the process which enabled teachers to evaluate their own performance by developing criteria for assessing competence in teaching. Teachers may apply the criteria, which they have generated, as part of a collective enterprise in their classroom situations. He argues that the reflection and thinking underlying this approach enabled teachers to become aware of the need to establish levels of achievement in any given skill area. In practice this meant that teachers are given the responsibility to develop their own competence. This obviously has a number of positive spin-offs for their teaching practice.

5.2
Elijah Sekgobela also trains teachers to do research. In his paper, entitled Combining the teaching of research methods with the assessment of project impact, Sekgobela describes how, while using the University of South Africa's (Unisa) training course for adult educators to teach research skills which are needed by students for the fulfilment of curriculum requirements for trainee educators, he simultaneously uses his teaching opportunities to conduct impact assessment. In this paper, he describes the process which required students to participate in all spheres of the research process – from the initial conceptualisation of the research to the final stage of recording of data. This paper discusses the process and benefits derived from teachers' undertaking an evaluation of their own contexts.

SECTION 6: Topicality vs. sustainability

6.1
Jeff Samuelson and Sara Harrity consider the debate which has arisen from attempts to answer the questions What outcomes are we looking for in terms of impact and what are the implications for the approach that we may adopt? In answering these questions they draw on two projects with which Book Aid International is associated. These projects, they argue, focus more on outputs than on issues of sustainability. They argue that, by their very nature, these outputs may be described as addressing questions of topicality rather than sustainability. The necessity for maintaining an accurate focus is made more complex by the requirement that evaluators determine the extent to which a particular project's intervention (as opposed to any number of external influences) has brought about whatever changes may have been observed. If the assessment is to determine what impact has been a direct result of the intervention and what impact should be attributed to extraneous factors, it must consider a number of other features such as, for example, the political, social and economic context in which the project has been operating.

6.2
Cleaver Ota's paper echoes the concern expressed by Samuelson and Harrity - that it is essential to address features of the context in our endeavours to attain a prognosis for project sustainability. His paper outlines the approach employed in the determining the outcomes of the South African Book Aid Project (SABAP) and certain concerns pertaining to project sustainability. While he concludes that the project had achieved the outcomes defined in the project document, he points to extraneous factors which impinge on these achievements. Accordingly, he asserts that it is not possible to assess impact, or to speculate on sustainability without locating the project within its socio-economic and political context. To do so, would be tantamount to decontextualising the possibilities for delivery. This, he argues, is because there are a number of extraneous factors which impinge on the actual implementation and which have a bearing on the potential for sustaining the project. With regard to the SABAP project, he identifies two such features: namely the role of government in financing the post-donor phase of the project, and the complex relations implied by collaborative multi-partnered implementation.

SECTION 7: Impact assessment and sustainability

This section focuses on the relationship between the assessment of impact and project sustainability. The papers take as their point of departure, the way in which the form of assessment could contribute to the enhancement of project goals, and to the capacitation of local players. These discussions are juxtaposed with the approach employed to assess the impact made by Mexican Advanced Professionalisation Project.

7.1
This section opens with the speech delivered by Carew Treffgarne on behalf of the Latin America, Caribbean and Atlantic Department (LACAD), DFID at various regional conferences on the impact of the professionalisation of the english teacher training in Mexico. Her talk outlines the rationale underlying the design of the model for assessment, and as such, provides an informative backdrop for the subsequent papers in this section. Her paper is intended to situate the collection of papers in this section - all of which are based on the evaluation of the Mexican Advanced Professionalisation Programme. Treffgarne indicates that the decision to use a participatory approach to evaluating project impact, was based on the understanding that it was not possible to do justice to a programme (of the scope and scale of MAPP) by utilising traditional approaches to evaluation. Her talk outlines some of the benefits of participatory assessment - in particular with regard to furthering the achievement of project outcomes, and to enhancing possibilities of sustaining project benefits. Her paper suggests that the approach employed, would be of direct benefit to participating universities, and more broadly, to the sector. Her paper provides an informative backdrop for the subsequent discussions of Morrow, Basich and Rodriguez.

7.2
Following Treffgarne, Keith Morrow's paper is concerned with the extent to which projects are able to sustain their impact after the project is concluded. His paper focuses on the assessment of impact on participants in the Mexican Advanced Professionalisation Project, a project intended to upgrade the professional qualifications of teachers working in language departments. Morrow describes the approach used to gain a sense of the impact made on teachers - an approach to the assessment of impact which doubled up, in formative fashion, as a component in building of institutional capacity. In this sense, Morrow views participatory research approaches as being essential for sustainability because they provide participants in particular with an opportunity to undertake a qualitative and quantitative assessment of impact. He suggests that this is one way to enhance the professionalism of those involved. He also indicates that the process of evaluation, constructed along similar lines, could contribute to the aims of the project - while at the same time contributing to sustainability.

7.3
Kora BasichPeralta's paper also elucidates aspects of the Mexican Advanced Professionalisation Project's assessment of impact. She outlines the research approach employed by her university in assessing impact. She mentions that once the assessment had begun, the research team was amazed to discover the achievement of outcomes which were not initially anticipated. In particular, she refers to the achievement of sector-wide, as well as institutional and policy outcomes. Basich, like Morrow (see the previous paper) indicates that the process of evaluation - especially the reflective component - achieved more than just the necessary required data. It also, she indicates, contributed to the enhancement of project goals in terms of qualitative improvement of English teacher training.

7.4
In this paper, Jorge Anguilar Rodriguez describes the method of assessment used in the Mexican Advanced Professionalisation

Scheme (MAPS). He indicates that although the research design utilised in this project was similar to standard research designs used elsewhere, the emphasis in this kind of assessment is different. The emphasis in the research design was directed at uncovering inter alia the unanticipated outcomes – and these, once discerned, played a significant role in ensuring project sustainability through their being posed as benchmarks for the continuation of the MAPS programme and for new projects which might be started.

SECTION 8: Anticipated and unanticipated project benefits

8.1
Mfanwenkosi Malaza, like Samuelson and Harrity, also suggests that if the assessment is to determine what impact has been a direct result of the intervention and what impact should be attributed to extraneous factors, it must consider a number of other features such as, for example, the political, social and economic context in which the project has been operating. He also examines the anticipated and unanticipated benefits of a project. His paper focuses on the types of impact made by the Mpumalanga Primary Schools Initiative. He argues that the determination of a project's benefits is more complex than it may appear to be at face value and that it is necessary to distinguish between anticipated and unanticipated outcomes. He argues that every project has degrees of both intended and unintended outcomes - whether they are positive or not - and that the impact of the unintended outcomes very often outweighs the intended ones from the local people's point of view. He elaborates on the unanticipated outcomes, which were not predicted at the start of the intervention but which nevertheless make a significant impact. He argues that these need also to be considered when evaluating project impact. He, like Samuelson and Harrity, contends that when identifying unanticipated benefits, it is necessary to look at the wider context of a project's operational environment. This becomes vital if one wishes to guard against attributing effects to the project that are merely incidental to it.

8.2
Mirela Bardi and Roy Cross also give consideration to the question of project outcomes. Their paper deals with an assessment of the impact of the Project for Special Purpose English in Romania (PROSPER). The paper describes how, apart from measuring the impact of the project, the evaluation specifically takes into account the sectoral impacts that led to the ripple effects of the project. These ripple effects mean that the project affects not only the sector, but also those institutions which were not participating in PROSPER. Bardi and Cross point out that it is necessary to consider the consequences of such ripple effects on the sector.

8.3
In her paper, Roopa Joshi attempts to provide a review of a critical area of project management in the District Primary Education Programme (DPEP), namely that of the assumptions underlying the practices and strategies for the assessment of project impact. She illuminates three broad issues, which she argues, informed the approach:

Firstly, it was necessary to address the question of how the DPEP impact assessment model should be designed. The how, she suggests, refers to the design on both a conceptual and operational level.

Secondly, it was necessary to consider the content and range of existing DPEP practice as it manifested across the various states and at various levels of decision-making. In terms of this, it was pertinent to establish how this practice might influence the various stakeholders of the project.

And thirdly, it was necessary to consider what the possible way forward might be for DPEP in the arena of assessment research.

8.4
Finally, Carew Treffgarne presents reflections on the contributions which emanated from the Forum on Impact Studies. In this paper she offers her concluding comments. Her reflections include an acknowledgement of the value of a participatory approach to impact assessment as well as the complexities associated with the process -particularly with regard to local ownership, insider/outsiders and, of course, the FIFO factor. Her paper draws attention to what DFID ought to take into account as it attempts to resolve the problems and confront the issues that evaluators have delineated in their papers. She analyses the papers in this collection in terms of the same thematic categories which have been used to organise this volume.

Treffgarne recommends that DFID's Education Division pay serious attention to the recommendations about the importance of allocating adequate time for assessments, budgetary considerations, and of the actual timing of such assessments. Considerations such as these need to be factored into project documents and project budgets. Her paper makes fundamentally important statements about the assessments of baseline studies and project impact - as well as about the sustainability of projects. Carew Treffgarne concludes by indicating that the Forum on Impact Studies has been instrumental in helping Education Advisers in DFID to identify some of the important lessons learned from the two-day workshop, issues which might constructively inform the future practice of the Department.

Veronica McKay & Carew Treffgarne