4. RELATIONSHIP BETWEEN NATIONAL AND EXTERNAL

4.1 A note on a participatory impact study in Eritrea: exploring the relations between national and external researchers
4.2 The relationships between national researchers and external researchers
4.3 lmpact studies: the role of an insider/outsider
4.4 lmpact assessment in educational projects: some perspectives on the 'insider outsider' debate

4.1 A note on a participatory impact study in Eritrea: exploring the relations between national and external researchers

Tefsai Bariaghaber
SETER Project
Eritrea

My experience of the Eritrean project impact assessment and other local field research leads me to believe that a participatory research approach is the most effective way of assessing impact. The Eritrean Impact Study clearly demonstrated that the participation of local researchers and their consequential interaction with external researchers created benefits and learning opportunities for both local and external teams.

The two teams participated in the planning stages and jointly produced an appropriate research design. This activity was squarely based on our agreement about our central purpose, which was to ascertain whether or not the project had accomplished its aims. Although collaboration was one of the main factors that influenced the relations between the local/national researchers and their external counterparts, different perceptions and expectations of the project on the part of the two groups of researchers created different expectations about the ultimate aims of the project.

Because local and external researchers were somewhat at variance about what might constitute an optimal research design and adequate goals, and because both parties were motivated by differing perceptions and motivations, this variance of opinion and purposes inevitably influenced both the design of the instruments and the research agenda. This, however, in no way detracted from the benefits of collaboration. In contrast, one might argue that the differences actually enriched the ultimate research design as well as the final report.

2 Setting the scene

The arrival of external researchers for the assessment study conducted in Eritrea was met with enthusiasm and was welcomed by Eritrea's Ministry of Education. For the duration of the research exercise, the relationship between the national research team and the external researchers was harmonious, understanding and cooperative. This positive relationship between the two teams was evident from the moment the external team arrived. In their initial interactions, the external team orientated the local team about what they were hoping to achieve during their stay in the country by describing the technical details of the process. The local team responded to these briefings by outlining their hopes, expectations and plans for the project. After much discussion and a fruitful exchange of ideas, both teams were able to reach agreement about a framework for the project.

This initial orientation was crucial since it formed the cornerstone of a positive relationship between the two sides. The researcher considers such mutual interaction to be an the essential part of any process of this kind because it affords both sides the opportunities to exchange ideas about what and how things will be done. Although initial bonding between the two teams was strong, their harmonious relationship was unfortunately complicated in the latter stages of the process and this caused several difficulties. The difficulties that arose were the following:

2.1 Geographical separation

After our initial orientation, the local research team moved back to the sites of the research (i.e. back to the schools) while the external researchers returned to their home country. This geographical separation unfortunately created a break in continuity between the two teams of researchers. What the local team had hoped for was continuous cooperation and immediate feedback. Instead, feedback was delayed and the separation resulted in a number of associated problems.

2.2 Lack of communication

Another factor which is associated with geographical separation and which influenced the relationship between local and external researchers was that the two teams of researchers were not able to interact and hence were unable to exchange ideas about emerging problems and other factors that influenced the work of the project as it progressed. Thus, for example, when local researchers thought of new categories or ideas that they felt might enhance the research, they were unable to test such ideas with the external researchers. Certainly, the communications in Eritrea were such that it was not easy to incorporate any new components into the research process after the external researchers had departed.

2.3 Lack of capacity building

Local researchers in developing countries often tend to rely on external partners to assist with the development of local capacity. This was one of the expectations that the local team had entertained – but because of the separation, was not achieved. The local team, for example, felt unable to cope as well as they would have liked during the data processing stage of the project. By the time the project entered this phase, the external consultants had already left for home. The local researchers had recognised their need to develop the capacity for data processing – especially with regard to the tabulation of information and the categorisation of items during the data collecting process.

When problems began to emerge at this stage, the local researchers realised that the external researchers could have made a decisive contribution by helping to enlarge the skills base of the locals, and that this contribution might radically have affected the quality of the intended results. Because the national research team was limited in their understanding of the theoretical basis of data analysis, it would have made all the difference in the world if the British team had been able to stay longer in Eritrea If they had been able to stay, they could have helped local researchers to gain competence and confidence in the theory and practice of data analysis at this crucial stage This would have yielded better results and more meaningful recommendations in the final report, and the impact assessment might also have contributed to sustainability.

3 Conclusion

Although this paper outlines some of the problems inherent in the relationship between internal and external researchers, it is nevertheless clear that we, as local researchers, were able to experience many of the benefits that arise out of the process of participatory research - in spite of the difficulties engendered by the problems (the chief of which was the premature – in our view – separation of the two teams of researchers).

The participatory approach is extremely beneficial to national researchers because it gives them the chance to refine their knowledge of research methods and techniques Because it does this, it contributes to the development of local human resources On the other hand, the process surely also benefits visiting researchers because it enables them to gain easy access to the local context Indeed, it is my contention that this process of immersion in local culture (a consequence of participatory research and collaboration with the local team) benefits external researchers long after they have left the original site of research activities.

4.2 The relationships between national researchers and external researchers

Harvey Smith (in conjunction with Paul Bennell, Consultant)
Senior Education Adviser
Centre for British Teachers

Although an impact study of an aid-funded project may, in theory, be undertaken entirely by external consultants or entirely by researchers of the country where the project is taking place, such studies in practice are most likely to be undertaken by a mixed team. This paper considers some of the issues that arise as a result of collaborative relationships between local researchers and the external consultants.¹

The paper draws on the experience of the DFID-funded English Language Teaching (ELT) Impact Study in Angola and Eritrea. This study, undertaken in 1997 and 1998, looked at the impact of ELT projects funded by ODA/DFID in the two countries and involved teams of local researchers trained and supported by three external consultants. When the studies in each of the countries have been completed, a comparative analysis will be undertaken. In a briefing to the consultants, DFID staff emphasised that the study should not be a conventional evaluation based on project logical frameworks, and that local perceptions should be gathered from stakeholder meetings and research activities undertaken by the local teams.

Both Angola and Eritrea are countries which are deeply involved in violent conflict and they therefore experience special difficulties that may not by typical of the countries where impact studies are usually undertaken. Many of the educated nationals of both countries live abroad and there is a limited capacity within the countries - especially within the institutions and departments of the ministries of education - to undertake research.

Undertaking such studies has enabled the consultants to reflect on the nature and practice of impact studies, and a number of issues have been identified for discussion here. These have been divided into ones that may be labelled conceptual.

2 Conceptual issues

Conceptual questions raise questions about the nature of an impact study as an undertaking. They of necessity raise questions that relate to the practice or processes of undertaking such a study. The kinds of questions that were raised in the assessments of the aforementioned projects were as follows:

2.1 Ownership

Just as the success of externally initiated projects depends in part on the extent to which local ownership and commitment can be established, the same is true of an impact study (although there is often less time is available to achieve this). The ELT impact studies in Angola and Eritrea were initiated by the agency (ODA/DFID) which had established and funded the ELT projects in the two countries and not by stakeholders in the countries. The success of the studies therefore depends in part on the extent to which local motivation can be established and ownership achieved. This, in turn, depends in part on the ability and willingness (of the personnel of the projects whose impact is being studied) to raise the awareness of the ministry of education or other institutions² about the need for such a study and about the lessons which might be learnt from the results. If one considers the experience of the Angolan and Eritrean studies, one sees that the response of local institutions varied according to the extent to which they are able to identify ways in which the study might resonate with their own agendas.

2.2 Perceptions of the role of consultants

The role of the external consultants may be perceived differently by different stakeholders. Such perceptions may depend on many factors, such as:

· the nature of the involvement expected of the consultants in each of the main stages of the impact study³

· the extent to which the national researchers perceive themselves or are perceived as being able to act independently of the consultants

· the nature of the relations established between the consultants and the expatriate project personnel (if a project is an ongoing one)

2.3 Using a multidisciplinary team

A strength of the ELT Impact Study is undoubtedly attributable to the fact that it comprises a multidisciplinary team of consultants whose fields of expertise cover ELT, economic and social development (including gender issues). This has encouraged a broadening of the field in which the country studies look for evidence of impact. Although the external consultants perceive themselves as a team, and in spite of the fact that they have made individual visits to each country, they may not be perceived locally in the same way. Moreover, it was not possible for the local players to replicate our notion of a multidisciplinary team since they were unable to field people with experience in investigating socio-economic issues.

2.4 Baseline data

It is difficult for local researchers to assess impact where little or no baseline data are available. Baseline studies would have contributed to the measuring of the post-project situation by enabling comparisons to be made against pre-project data. In both studies, the external consultants had access to documents compiled by DFID, which describe the pre-project situation in outsider terms. (Local researchers would not normally have access to such documents.) Equivalent descriptions in insider terms are unlikely to exist. Local researchers may not have been around before the project started or they may themselves have been involved in a very specific part of the project and so be unaware of the wider picture.

2.5 Levels of impact

This issue refers to the balance between looking narrowly (in the case of an ELT impact study) for impact on ELT activity and looking for wider socio-economic impact. This is especially problematic when local researchers are drawn from a narrow group in the ELT field or from a ministry of education. In such cases there may be a desire to focus largely or entirely on looking for evidence of impact in the ELT classroom and even a reluctance to go beyond the direct outputs of the project. This can cause tensions between local and external researchers.

2.6 Training researchers while simultaneously undertaking research

The role of the external consultants includes training the local researchers and assisting them with the design of the research, the development of the instruments, the analysis and interpretation of the data and the presentation of the findings. How does one balance these roles? Although the training role is essential to building on local capacity, this may cause the external consultants to perceive this training as being more important than their roles as facilitators and managers of the research. They may therefore not consequently fulfil their terms of reference.

2.7 Remuneration and responsibility

The relationship between external and local researchers is also influenced by whether or how the local researchers are remunerated. Ownership of the study by a local institution would imply that the institution remunerates the researchers (even where an external agency assists with the budget for this), and the external researchers are not seen as buying the research. In one of the countries where an ELT Impact Study was undertaken, local institutions were unable to compensate local researchers. In this case, they were paid a fee by the external consultants. The perception created by this was that individual researchers were employed by the outside consultant agency and that institutional (and hence also local ownership of the study) were therefore limited.

3 Practical issues

While conceptual questions raise questions that relate to difficulties arising from the research process, there are several logistical or practical issues that impact on both the research enterprise and the relationship between the local and external researchers.

3.1 Location

A problem that we experienced while managing the ELT Impact Studies for Eritrea and Angola was that, in both instances, the management and coordination was located outside the countries concerned. The management was therefore perceived as driven by the client (DFID) - at a time when engendering a sense of local ownership was seen as critical to the success of the studies.

3.2 Number of institutions involved

In both studies we experienced difficulties with coordinating research and ensuring cooperation because in both cases more than one institution was involved in assessing the impact of the respective projects. This was particularly noticeable in those cases where there was a mixture of government and autonomous or semi-autonomous institutions, such as a ministry of education and a university. If there is no institutional hierarchy, the person who is appointed as team leader or as research coordinator may find it difficult to secure the necessary collaboration. In such cases, requirements such as obtaining agreement about deadlines (to name but one issue) may prove to be extremely difficult. When this happens, external consultants might find themselves assuming a coordinating role by default.

3.3 Selection of researchers

There are a number of practical issues which are relevant to the way in which a local team is selected and managed, and to what the role of external consultants might be in this process. These include the extent to which external consultants can influence the size and composition of the research team, their gender balance, the level or mix of research skills, the identification and inclusion of disinterested parties, and the commitment or level of involvement of the team members. Also relevant are issues such as whether or not researchers are released from other duties for the period of the study and whether or not remuneration is necessary. In the Eritrean and Angolan studies, there was reluctance on the part of participating institutions to have people from outside those institutions in the research teams. The result of this was that the teams did not have the insider/outsider balance which the external consultants would have wished to see. Such differences can lead to tension between external consultants and institutions.

3.4 Timing

The timing of assessments is also a significant issue which needs to be considered. In Angola and Eritrea, the funding agency (and not the local players) decided when the impact assessment should take place. In both cases, the process started while the projects were still being implemented and while expatriate project staff were still in their postings. This posed difficulties for both the internal and external researchers. For the former, the timing of the investigation increased the difficulties that local researchers experienced in assuming a detached attitude to the actual project - and this meant that they were less able to look for impact beyond the current project activities. The consequences for the external consultant were that they experienced difficulties (in the limited time available) in setting up the research programme and convening stakeholder meetings immediately after they had arrived in the country.

3.5 Local realities and external deadlines

Progress in the research depends on the extent to which local researchers are able to undertake the work without disruption from unforeseen events (the ELT Impact Study in Angola and Eritrea had to compete with wars and strikes) or are able to work at their tasks without being asked to undertake other activities. Since impact studies generally have to be completed in a very short time-scale, any delays resulting from unforeseen local events (such as in war-torn countries) will have a greater effect, as there is less capacity to absorb them.

4 Conclusion

In essence, the issues above relate to the difficulty of finding the right sort of balance between local ownership and achieving the externally imposed terms of reference. If an impact study is to be locally owned and is to provide a ministry of education, or another local institution with information about the effect that a project has had, then the study must meet local needs and must be undertaken in a manner that is locally acceptable.

If the investigation is to be an external activity that enable an external funding agency to find out how effectively its projects have contributed to development (assuming that it is possible to attribute evidence of impact to the external funding), then some way must be found of making externally imposed criteria and expectations acceptable to those who are locally involved. The key question then becomes, What sort of compromise is possible under the circumstances?

Footnote

1. Following the brief given to the writers, the paper will deal solely with the perspective of the external consultants.

2. This would need to be done well before the actual study is started.

3. This includes consultation with stakeholders, design, data collection, data analysis and report writing.

4.3 lmpact studies: the role of an insider/outsider

Mohammed Melouk
Mohamed V University
Rabat, Morocco

Assessing the impact of an educational project is not an easy matter, especially when the aim of the exercise is to determine exactly, in both quantitative and qualitative terms, the exact nature and extent of change which is being sought by the project. This complexity may present an insurmountable hurdle - whether the evaluative enterprise is carried out by people directly involved in the project (the project designers and implementers) or by outsiders or external experts (or a combination of both).

I contend that the sometimes baffling complexities of impact assessment are caused by the fact that education and educational change are situated in a complex matrix of causes and effects that include cultural, socio-economic, psychological, material and administrative processes. Impact assessments, by their very nature, require a diverse number of role players, including, among others, decision-makers, funding organisations, project designers, administrators, inspectors, teachers, students and pupils.

Quite apart from this, the unique geographical, social, economic, political and educational conditions that gave rise to the project in the first place cause project evaluators to become enmeshed (whether they like it or not) in a complex of feelings and attitudes that invariably impact strongly on the assessment process. Educational projects which are designed for developing countries are far too often designed without any input (and in the absence of) those for whom the project is intended. This inevitably gives rise to a variety of attitudes on the part of the local population to the project. Attitudes may range from enthusiastic adherence (the rarest!) to qualified but sullen acceptance, indifference, or simple outright a rejection -at least at the psychological level.

Although the emergence of such attitudes among the target population may be attributed more to negative emotional reactions rather than reasoned intellectual objections, the effect on the project itself is nearly always emasculating or debilitating. This, sadly, is inevitably the case when the local target population have not been involved in the initial stages of the project and do not fully understand its methods, aims and objectives - and hence, of course, cannot appreciate its potential benefits. In such cases, an accurate assessment of the impact of the project cannot be obtained without considering the potential effect of such attitudes on the outcome of the project. The assumption here is that the contribution of insiders, who are outsiders to the project but insiders to its situational context (hence the term insider/outsider), may be able to contribute insights of crucial importance.

2 The rationale for an insider/outsider in an impact study

In the light of these considerations, it is necessary to discuss the insider/outsider dimensions of impact studies. The discussion is based on conclusions drawn from personal experience in a collaborative impact study entitled The Moroccan Item Banking Project.¹ In this paper I have attempted to highlight the importance of local human abilities in any endeavour to assess the impact of an educational project, without, of course, completely ignoring the legitimate concerns of objective educational evaluation or of funding bodies.

Given the nature of the educational enterprise, the implementation of any educational project is bound to generate a dynamic of its own, which is often not anticipated in the original design or sufficiently catered for or monitored during the implementation stages. In addition, any project involving many people will engender varying levels and degrees of involvement. The people directly concerned or those for whom the project has been designed may take a route (in terms of objectives and priorities) slightly different from the one originally intended. They may also have a hidden agenda which may not correspond to the actual aims of the project, but which can decisively affect the desired impact. The presence of these side effects are generally linked more to attitudes than to technical aspects of the project (knowledge, skills, etc).² They raise the question of how to deal with these phenomena and what place they should have in the study. In other words, what relationship should hold between predicted and unpredicted outcomes in the light of the type of attitudes generated by the project?

A number of strategies and techniques, borrowed from educational research and testing, management education, sociology of education and econometrics are used to investigate the impact of an educational project. Care is taken to collect relevant information and also to triangulate data. Depending on whether the objective of the evaluation exercise is to assess the degree of achievement in terms of value for money (quantifiable indicators) or simply to measure the nature and degree of change in qualitative terms, qualitative data is generally used to supplement or reinforce quantitative data or the reverse. Furthermore, the evaluator may adopt the project designer's perspective, or a monetarist one (value for money), or even an educational or academic perspective (the requirements and constraints of research itself).

However, although it is possible to gather a rich database, reading or interpreting data to assess the real impact of the project, if not carried out from various perspectives, may lead to conclusions which only partly reflect the true state of affairs. This is not to suggest that the above-mentioned perspectives are not important, but simply that they need to be gauged in terms of local features and characteristics. In fact, the local perspective - in terms of population, local conditions and so on - is rarely taken into account. Hence the need in any impact study to incorporate the local culture- in its widest sense - through the association of an insider/outsider. In terms of this approach, the triangulation of data can be paralleled with some form of triangulation of interpreting data, drawing conclusions, and making recommendations.

3 The role of an insider/outsider in an impact study

Educational impact may be investigated in either qualitative or quantitative terms (or both), and its scope and extent may be assessed in terms of change in the following areas:

· Knowledge: complete, incomplete, partial, new information building on old information, updating old information

· Skills: new skills, transfer of skills, building on old skills

· Attitudes: positive, negative, neutral, indifferent

3.1 Some benefits associated with the inclusion of an insider outsider

An insider/outsider, if well prepared and trained, can contribute from a vantage point (whether the evaluation is summative or formative) in the following ways:

· Facilitate communication

If the insider/outsider is able to facilitate a consideration of local habits and customs as well as what is unique about the local mentality and socio-cultural values, he or she may enhance and enliven communication with people directly or indirectly involved in the project. Interaction between an external expert and the target population can be facilitated through the collaboration of an insider who can help to establish on firm ground a common language (concepts, values, assumptions) between the evaluators and all those concerned in the project. In this way, it is possible not only to identify the exact nature of the impact, both present and future, but also to give meaning to the impact, especially from the perspective of the target population.

· Solve field-related problems

The insider/outsider may help to resolve apparent or real ambiguities, contradictions and misunderstandings in all matters relating to the project, not only between the target team and the evaluator, but also between the target team and other people involved.

· Contribute to the development of research instruments

The insider/outsider can help in the adaptation of instruments used to gather data and relevant information as he or she takes into account local features and the target population.

· Contextualise findings

The insider/outsider can play a role in the reading and interpretation of data in the light of the context of the project and the singularity of local conditions.

· Contribute to recommendations

The insider/outsider can assist in drawing appropriate conclusions and recommendations if he or she bears in mind the local context of the impact study in terms of both objectives and implications.

It goes without saying that none of these things can be done if the insider/outsider does not have the appropriate skills or is not appropriately equipped to deal with the various problems that may arise. In addition to the mastery of theoretical and practical skills required in the design, implementation and evaluation of educational projects, he or she should first and foremost have a good understanding, not only of the project itself (objectives, stages of implementation, requirements), but especially of its implications in terms of the desired change and the potential impact that such change might have. It is obvious from the above that the insider/outsider needs to fully understand local culture and the way it might affect the impact of the project at different levels - as well as in terms of degrees of involvement. But, most importantly, and in order to maximise objectivity, he or she should not have a stake in the project. If the insider/outsider were to have a stake in the project, it would be inevitable that his or her point of view and whole approach to the evaluation would be, to some extent, biased.

In addition to mastering the communication skills required by the evaluation, an insider/outsider needs quickly to develop a good and productive working relationship with the target population and the external experts. This is all the more important as the quality of evaluation in its different stages depends on it. Not only does he or she need to fulfil the role of an informant and a communication facilitator; he or she also needs to be a full participant evaluator. The combination of insiders and external evaluators can help to uncover aspects of the impact which would not necessarily be highlighted in the type of evaluation generally carried out solely by external experts or by experts who have a homogenous point of view. This is all the more important because certain crucial decisions are made on the basis of the study.

3. Conclusion

In the light of the foregoing and given the nature of the evaluation enterprise, which seeks to determine the nature and scope of an impact, the role of an insider/outsider can be crucial. In addition to adding a human dimension to the study, he/she may play the role of moderator throughout the various stages of the study, not only as a communication facilitator between the target population and the evaluator/s or researchers, but as an active participant researcher - with no objective other than to enrich the database of the project, to facilitate the correct interpretation of data, and to contribute to drawing insightful and appropriate conclusions.

It may be argued that an external researcher or expert can easily develop the kind of skill and expertise generally brought in by the insider/outsider. Although this may hold true for the few, it is rarely the case for the many. In addition, the exercise may require time and effort (and hence money) which might more profitably be spent on the study itself. In fact, the time and energy which can be saved if one factors in an outsider/insider's contribution may contribute to more productive and meaningful evaluations and swifter completions of impact studies.

Footnote

1. Originally, the project aimed at modernising the Moroccan ELT curriculum and assessment, but during a new project phase, the focus shifted to assessment with the entire project concentrating on technical aspects of item banking.

2. Throughout the various stages of the evaluation process, it became clear that most teachers and testers do not fully understand the ultimate aims of the project though the latter can identify immediate objectives Moreover, some negative attitudes are in sharp contrast with what has successfully been achieved in the area of testing.

4.4 lmpact assessment in educational projects: some perspectives on the 'insider outsider' debate

Dave Allan
Norwich Institute for Language Education

In this paper prepared for the DFID Forum on Impact Studies¹, I wish to draw on my experiences in project evaluation to address the question of Who does the evaluating? and then to explore the benefits and disbenefits of participatory approaches to impact studies. The paper is seen as yet another contribution to the insider-outsider debate in so far as it elaborates on the relationship between what are sometimes called national researchers and external researchers.

The terminology we use is, in fact, quite important. Keith Morrow has pointed out, in his paper entitled, Sustaining Impact: the Mexican Advanced Professionalisation Project, that our understanding of what researcher means will take us a long way towards clarifying a particular view about evaluation and how it may best be carried out in the context of educational projects and programmes. Such a view might immediately predispose us to a belief that specific expertise in the field of research methodology is a prerequisite for effective evaluation. However, my own experience, time and again, has been that some of the most insightful verdicts on the impact of educational projects have come from those who have little or no formal training in evaluation but who do have the ability to reach common-sense conclusions about what they see and hear. Such people are teachers, parents, and, above all, the learners themselves - the students and pupils.

The validity of impact assessment is frequently neither an issue of 'objective truth', nor a consequence of the degree of correspondence with a project framework. It is more likely to depend on whose agenda is being used as a cue in evaluations.

2 Insiders vs. outsiders

One of the more obvious problems that arises when evaluation is carried out solely by an outside expert is the problem of the culture gap. Outsiders, for all that they may be experts in the methodology of evaluation and have wide-ranging experience, may quite simply not see and not hear the reality of the outcomes in a field as difficult to measure as education.

The other major problem for the outside expert is quite simply a question of the timescale. Much worthwhile educational change is not measurable over the timescales typical of many recent language education projects, with their cost-conscious focus on short-term measurable outcomes and sustainability, let alone by evaluators who are working to deadlines defined in terms of weeks and who have tight budgets. My own experience in Bangladesh was of being asked to come to crucial decisions about a multi-million pound language education project on the basis of a two-week visit tacked on at the end, with an evaluation budget which was a tiny fraction of a percentage of the total sums involved before and after the evaluation.

The use of national and external to label apparently opposed (and certainly often juxtaposed) personnel and perspectives can also be misleading, as Keith Morrow (see paper 7.2) and I both well know from our separate and our shared experience of educational project evaluation. Do non-nationals have to be regarded as outsiders? From my experience in Tunisia I wish to argue that non-nationals can be very much insiders as far as a particular project is concerned. During my sustained period of work in Tunisia, I was often described as an honorary Tunisian. On the other hand, nationals may be perceived as outsiders to particular projects or programmes, but fully-fledged insiders in terms of language, cultural awareness, access to longer-term perspectives and their ability to interpret tiny clues. These advantages give them the skills to read between the lines of the bare statistics. Such was the role played by Mohammed Melouk in Morocco (see paper 4.3), where he was one of a team of three charged with carrying out a major formative evaluation of a long-term curriculum development project which was funded by DFID, with the British Council playing a major management role.

3. Revisiting the insider outsider debate

My own views in the insider-outsider debate have been shaped by a number of significantly different experiences over a period of twenty years -a time during which I worked as a teacher-trainer and as a language consultant in some 40 countries. In recent years four of these experiences have stood out. In the sections which follow, I shall elaborate on these experiences to illustrate the differing positions that may be located on an insider-outsider continuum.

3.1 Case study: The single outside expert

In both Bangladesh and in Estonia, I was asked to go in as a single outside expert to evaluate and make recommendations with regard to national language policy. Before starting in Bangladesh, I was given the opportunity to visit Bangladesh for two weeks since it was a country I had not visited before. My task was to provide an informative report on how successful a Bangladeshi project had been in providing structures and systems to facilitate through the school curriculum the re-introduction of English (a language which, for the previous generation, had been part of their everyday life).

It was necessary for me to focus all my skills and experience, to read every available report and document, and to interview stakeholders as varied as the Education Minister, the heads of examining boards and training colleges, and kids in rural schools. Because I wanted to get it right, I worked long days and I eventually produced a long and detailed report. I really cared about the outcomes and agonised over the recommendations I made, but I never knew if they were the right ones. There were no insiders working with me (apart from those who were my informants) to tell me what they thought, and I have not heard a word from anyone in Bangladesh since I submitted the report.

I have since heard this kind of evaluation described as a FIFO² or FIO – Fly In, Fly Out and... well, you can guess the rest!

3.2 Case study: The single outside expert - but with a sustained role

While what I had experienced in Estonia was similar in some ways to what I had experienced in Bangladesh, it was also very different. Although I was once again chosen to work as a single outside expert in Estonia, I was also asked to act as a consultant on behalf of the Council of Europe in order to review how effectively the National Language Board was developing national language policy. The board's aim was to enshrine and support Estonian as the national language while making effective allowances for the needs and problems of a large Russian-speaking minority. The problem of citizenship and the requirement for competence in Estonian had generated enormous internal friction and dissent, and were, in addition, viewed as a possible source of destabilisation in Eastern Europe. In fact, as I got off the 'plane, I had Russian and Estonian television interviewers shoving microphones in my face, and asking me for my 'expert verdict' about whether the language level being required for citizenship was too high or too low. I was only too aware at that moment of the limits of my expertise. But I did not have that FIFO feeling in Estonia because my experience in the country was sustained over a longer timescale. I was able to return regularly to Estonia and to evaluate progress over a period of nearly two and a half years. During that time relationships were built; trust was developed where there had only been suspicion, and even hostility had dissipated. Those who had initially been the objects of the evaluation became in effect collaborative members of a team. We had established a sufficient number of important agreed-upon objectives and our shared concerns helped to bridge the insider-outsider gap. I found that my theoretical expertise, my professional knowledge and my change-management skills proved their value in practice as I worked with insiders over a period of time. I had been able to give many of the insiders an outside perspective, and this had helped them to assess more accurately, systematically and humanely the impact of the work that they were doing. They in turn enabled me to become in part an insider. This really enhanced my ability to evaluate what was happening. The insights gleaned enabled me to understand the complexities of the project rather than to think of it in terms of the simplified images with which I had been initially presented. There are some organisations that worry about their personnel 'going native'. I now find it difficult, after my experience in Estonia, to imagine how any educational project can be properly evaluated, without a proper 'native' perspective.

3.3 Case study: An insider outsider evaluation

In Morocco I was asked to lead a team of three people - myself and two Moroccans - who were required to make a formative evaluation of a major ELT Curriculum Development Project. This project had become generally known as The Item Banking Project because of the central role played by item-banking in the assessment side of the project. This, in fact, had become the main focus of activity for those involved. Although the evaluation took place over a period of one year and involved three visits by myself (the outside expert), it also required continuing work (between my visits) by the two Moroccan members of the team. The Item Banking Project had developed over a period of eight years and had latterly acquired potential significance for the ways in which other subjects might also be assessed in Morocco. The team chosen to do the formative evaluation3 was designed to reflect the maximum advantage obtainable from using three individuals with different backgrounds but appropriately complementary expertise. The team included: · a full outsider, the UK expert (myself), · an insider/outsider in the sense of being a Moroccan national with awareness of the project and the professional issues involved, but an outsider to the project, and · one insider, a member of the project, an inspector who was the leader of a regional test writing team. Though there were problems with funding and communication, the team worked well and produced a series of reports which, in my view, represented a much more accurate and balanced assessment of the project's impact than anything that had been produced before. No single agenda could be given precedence because of the make-up of the evaluation team. This meant that difficult issues were addressed in a positive way. The evidence soon made it very clear that a number of benefits had accrued from the way in which the evaluation team had been set up. In the next section I will elaborate on the benefits, along with some caveats.

3.4 Case study: An outsider who had become an insider

Recently, in Tunisia, I was able to see it all from the other side. The Secondary ELT Project, in which I had been involved as the project leader, was evaluated by a team which deliberately included senior members of the inspectorate who were a part of the project working alongside the outside expert. This was a project I knew as an outsider who had become an insider. It was a situation about which I was passionately concerned, and I wanted those who were evaluating to have the necessary professional awareness to assess the project's impact across a wide range of outcomes (some of these outcomes were accounted for in the project framework while other outcomes - some of them, very important indeed - were entirely unanticipated). What was clear, and gratifying, was that the presence of the insiders allowed issues to be raised which might otherwise have been missed. This consolidated the continuation of a crucial sense of ownership on the part of those who would soon be solely responsible for future success or failure of the project. One cannot speak of sustainability if the long-term stakeholders do not have a major say in the assessment of impact.

4 Contrasting insiders and outsiders

So what generalisable conclusions may one draw from these different experiences? (Whether or not what follows has a wider application is something you will have to decide for yourself on the basis of your own experience and your awareness of the experience of others.) What can outsiders and insiders respectively bring to the evaluation of educational projects and programmes?

5 Conclusion

It will be clear from what I have said that most of my experience leads me to favour a team or collaborative approach to impact assessment in language education projects and programmes. What, then, is the distinctive contribution that a team may make - if one compares it to the contribution that may be made by an individual (whether insider or outsider)? The following advantages (in no order of significance) seem to characterise the use of teams rather than individuals in educational project evaluation:

· Teams collectively gather more and more varied expertise and experience.
· Teams have the ability to multi-task.
· Teams can to cope with unavailability caused, for example, by illness.
· Teams have the potential to reduce the effects of prejudice.
· Teams can offer a wider range of evaluation perspectives.
· Members of teams can be mutually supportive.
· Teams offer opportunities for regular interactive reflection and analysis.
· Teams can engage in cross-checking and ongoing articulated critical analysis.

Strengths	Weaknesses
The outsider in an evaluation
If well chosen, an outside expert will bring: · specific, relevant professional expertise and experience · a wide range of perspectives (not just local ones) · the ability to see the big picture · the potential to be an open-minded, unbiased listener and observer · the potential to be committed but dispassionate · the potential to arrive at non-partisan judgements · the ability to ask difficult but important questions · the power and authority of an outside expert · a clear focal point for communication · only one set of potentially conflicting workloads	But s/he will inevitably tack: · all the plus factors of a team (and a team may · an in-depth awareness of the local culture · local contacts who could be sources of · language proficiency at the required/ideal level · the ability to really understand what is going on
The insider in an evaluation
An insider can potentially bring: · a high level access to the local languages · national/regional/local cultural awareness · an extensive awareness of the environment · a sense of history, and when it matters · a knowledge of which doors to knock on · the influence/authority to open doors · the ability to detect/identify smokescreens · a knowledge of local vested interest · a sensitive 'bullshit' detector · an ability to see project goals as a recipient · a long-term perspective – outsiders go away!	But there are significant disadvantages and dangers: · The insider is always vulnerable to local pressures. · The insider may be a part of the vested interest. · The insider may have a personal axe to grind. · The insider may be unable to see the big picture. · The insider may tack the authority of an outsider.

· Teams may have the ability to cope better with unexpected events.
· Teams cost more but often provide better value for money.
· Teams create more potential for communication problems.
· Teams are more likely to be become embroiled in time-wasting internal disputes.
· Teams need to allocate time for communication and liaison.
· Teams need to be managed and so require leadership.
· There are more likely to be workload and prioritisation problems in teams.

· Teams are more effective than individuals.

· The best teams are characterised by carefully selected complementary expertise and awareness.

· Insider/outsider combinations can be very effective.

· Impact assessment needs to be planned in from the very start - and not 'tacked on'.

· Effective impact assessment requires adequate time and funding.

Footnote

1. My involvement in the DFID Forum on Impact Studies came about not because I have a particular theoretical perspective to support, nor a strong academic background in development issues, but because of my long-term interest in evaluation in relation to language teaching.

2. A term coined by Dermot Murphy and Pauline Rea-Dickins (see paper 3.1).

3. This had been costed into this funding period of the project as a significant phase and with a reasonably realistic budget.