Monitoring and Evaluation of ICT in Education Projects :
A Handbook for Developing Countries Daniel A. Wagner, Bob Day, Tina James, Robert B. Kozma, Jonathan Miller & Tim Unwin Publication: infoDev, 2005 A Handbook for Developing Countries – is intended as an  ntroduction and guide for busy policymakers and practitioners grappling with how to         understand and assess the ICT-related investments underway in the education sector. The handbook comprehensively covers a domain of interests to help decision makers to develop a stronger knowledge base through Monitoring and Evaluation to make better investment and innovation decisions in ICT in education.

The challenges facing education systems in most of the developing world are formidable. Evidently, in recent years ICTs are being regarded as the solution to most of these challenges. ICTs are being used widely to aid education in many developing countries. However in view of the resource constraints in most of these developing countries, there may be potential great risks associated with ICT use in education in developing countries. Relatively little is actually known about the effectiveness of investments in ICTs in education in promoting educational reform in general. Little hard evidence and  consensus exist on the proper, costeffective utilization of ICTs to  meet a wide variety of some of the most pressing educational challenges The power of ICTs as enablers of change (for good, as well as for bad  is undeniable but if policy advice related to ICT use in education is to be credible, it need  to be backed up by a rich database of lessons learned, impact evaluations and cost data. The Monitoring and Evaluation of ICTs in Education handbook is specifically designed to meet the needs of developing countries and to answer some pertinent questions of implementation fidelity of an intervention. The handbook would help decision maker to understand the outcomes of intervention in terms student learning and new skills learnt, teachers outcomes with respect to development of technology skills and new pedagogical approaches and other outcomes in terms of increased innovativeness in scho l and increased access of the community to adult education and literacy (chapter two). The decision maker would also be able to identify ‘Core Indicators for Monitoring and Evaluation Studies for ICT in Education’ (chapter three) that relates to specific components of the ICT intervention and their implementation, and include both input variabls (such as classroom ICT resources, teacher training, and classroom pedagogy), as well as output variables (such as student learning of  chool subject, learning of ICT skills, and “21st century” skills. The chapter on ‘Developing a Monitoring and Evaluation Plan for ICT in Education’ would help policy-makers in understanding ways to assess results at the end as related to the original objectives set for the project through appropriate, realistic andmeasurable indicators. Policymakers  would also understand the need of ‘Capacity Building and Management in ICT for Education’ (chapter five) to develop the appropriate skills to deliver these programs effectively.The chapter on  Pro-EquityApproaches to  Monitoring and Evaluation: Gender, Marginalized Groups and Special  eeds  Populations’ guides decisionmakers to be inclusive in their approaches. Chapter seven  discusses the ‘Dos and Don’ts in Monitoring   nd Evaluation’ that  identifies what decision-makers  should do to  enhance the impact  and effectiveness of ICT4E  programs and the    oncomitant  things that one should not do.  In short, the book is an one   top  guide for policy makers and  practitioners on how to evaluate  ICT   nd Education Initiatives. ?

The DiploFoundation is a small non-profit  rganization created by  theGovernments of Malta and Switzerland, co-located in Malta and Geneva. It’s mission: ‘to assist all countries, particularly those with limited resources, to participate meaningfully in international relations. Diplo promotes a  ultistakeholder approach, involving participation of international  rganizations, civil society and other actors in international affairs. Diplo’s activities include education and training programs, research, and  the development of  nformation and communication technologies for diplomatic activities.’ (www.diplomacy.edu) Diplo’s flagship course is its one year Post Graduate Diploma or Masters of Art course in contemporary diplomacy and use of ICT. Diplo is accredited by the University of Malta, with EU-wide  recognition. Limited to about 25 participants, the course draws  young men and women from around the world: working diplomats, students of international affairs, those working in international organizations, NGOs and others. Diplo offers scholarships to participants from  developing and transition countries, on the basis of funding from Swiss Aid, and support from entities such as the EU, the Commonwealth and other donors. In addition, Diplo runs an expanding number of short courses (usually of 10-week duration), and a series of programs covering Internet governance, plus diplomacy-related topics, aimed  at single countries, special groups,as also general participants. A common feature is that these courses are either entirely run through the Internet, or are offered as blended programs to participants from Europe, with a strong distance learning component. Teaching methodology The Internet-teaching methodology used has evolved through  experience, and consists of textbased learning, with very limited use of multimedia. A significant proportion of Diplo course users  are based in Africa, Asia, Oceania, places where broadband is still an unrealized promise, and the dial-up  connection, sometimes accessed by  articipants relying upon Internet  cafes is the simple reality. The  underlying  technical system and  the support infrastructure have  evolved over these  years, through  singularly dedicated work by  Diplo’s visionary Director, Jovan  Kurbalija, and his dozen-strong  team, based in Malta, Geneva and  Belgrade.  I   ined the teaching faculty of Diplo seven years back; that story  is one of chance   nd serendipity,  best reserved for personal  conversation! Let me, in this paper, first examine the relevance of e-  learning for  situations where    ntensive   acherstudent dialogue is imperative,  then narrate the  way we use elearning  at   iplo  and describing  another challenge  of e-learning —  the concept of the pure   selflearning’  course,  which some may  regard as the holy  grail of distance  learning, and others may see as   an unattainable goal.  Why e-Learning?  In   ome circumstances, e-learning is more efficient and economical  than   raditional learning, or ‘tlearning’. <!–Ads1–> Foreign ministry  mid-career training (when  most  personnels are on assignment  abroad ) and other ‘continuing education’   rograms, for example,  find a natural fit with the Internet. This method also proves less expensive for training locally the locally-engaged staff working in embassies abroad.  Another characteristic of diplomatic work is that much of it  involves craft skills. When the course participants are people with  many years   f experience, the ‘training’ is actually a mutual  learning process among the  lass,  where the faculty function as discussion catalysts and  facilitators, and   ven have much to learn from the course participants.  The Diplo method Diplo   as evolved its methods over  several years; described below is the current   echnique, in a process  that changes continually.<!–Ads3–> How does  our faculty-led  teaching work, with  a class size of 15  to 25 drawn from  10 or more countries,  distributed around the  world?  A typical module  consists of eight lectures and   wo  assignments, besides a final  exam. Under  universitymonitored standards,   uch a module typically  counts for four   units of postgraduate training. Diplo  uses sophisticated text-based  methods because many of its ‘class participants’,   s Diplo prefers to describe those enrolled in its  programs — do   ot have access to  broadband or multimedia modes.   After reading the lecture  osted on   the class web-page (usually of 3,000 to 4000 words), the  participants highlights some words or part of a sentence from the  lecture text   nd adds his or her   comment on it and the faculty add  his observations. The  lecture thus gets thoroughly reviewed by the  class. One can also view all the comments in their totality (without  opening each text-box), through a  ‘discussion tree’. The participants can also engage in a open-ended  dialogue, or   synchronous (i.e., not  in real time) exchanges, among the class and its   nstructor through a ‘lecture blog’. A part of the  participant’s final evaluation is based on the number and quality of  comment. All the comments are  ublic,  visible to the entire class,  but an option for private teacherparticipant   mment  hrough e-mails is also available.  Real-time online debate on points arising from  a particular lecture is incorporated in the e-learning process as the third major  ctivity     for each lecture. Thus each generation of course  members and  lecturers create a  new layer of meaning and     examples, enriching the initial   text. Unlike comments made in a traditional class, all the ecomments are  accessible anytime,  anywhere, and produce a   ermanent record as well. A  ‘resources’ button on the home page of the e-learning lecture site, takes one to a  ollection texts of   relevant documents as well as links   o other relevant web    sites.  Finally, at the end of a 7-day cycle, the teacher ‘closes the loop’ by  summarizing the key issues that the class discussed, and suggests some themes  hat participants may  wish to pursue on their own. The    iploFoundation’s experience shows that the process create a  sense of  community within this far-flung class which is further  augmented with group work among  class members, such as  assignments and simulated   egotiations. Such    chemistry  among the participants is the key   o this entire process, more so  when learning is no longer a  top-down process of dissemination  of basic  knowledge by the faculty to students, but rather mutual dialogue and shared learning . E-learning has to be supported by a  technical team that continually monitors the systems. A course director provides an extra pair of eyes for observing clas   nteractions, acting as the participant’s ally to resolve technical and other issues.  The self-learning format In some ways self-learning represents the ‘last mile’ of  distance learning, where the essence of a single lecture is  distilled to some 9 or 10 ‘frames’ or slides that a participant can scroll through, at his or her  convenience, with built-in questions and pop-up quizzes that  test the learning accomplished. At the extreme end of self-learning,  there may be no faculty participation at all, though it is quite easy to build in some faculty  supervision, depending on the circumstance.   came upon this method through   the Canadian Foreign Service Institute, Ottawa, who asked me in 2004 to convert a textbook I had written two years earlier into such a self-learning course of 6 lectures. We found that a course of 6  lectures could involve around 150hours of work in developing the formatted script, and an additional 700 to 800 hours to convert that into a full multimedia product,  making the process fairly laborintensive.That particular course took one year’s work. How does e-Learning  measure up? One may legitimately ask, is it really possible to overcome distance, replicating the instant, natural communication of the  traditional format? Can online learning match the rapport that a good teacher establishes with  students and   ce-to-face evaluation? Surely no videoconference or online chat room can reproduce the way a good guru assesses at a glance the class’s absorption of the ideas taught.

At first sight, such e-learning programs miss out on the rich interactivity described above. But in practice, once a self-paced program is designed, it is easy to add on faculty intervention, either in the form of exercises whose results go to a faculty member or  via periodic group exercises or simulations that break the apparent isolation of the self-taught format.  There is only one caveat: the faculty add-on is possible only  with server- based programs, not those distributed on CD-Roms.  As for the faculty-led e-learning programs, like those offered by Diplo, a surprising conclusion is  that in some ways the new format  is superior to the old one. Unlike traditional classrooms where the tutor is able to reply to only a few  queries within a stipulated period of time, e-learning classrooms supports a sustained facultyparticipant  engagement as evident  form the intensive scrutiny that each lecture undergoes.

A huge advantage is that lecture comments remain available for  subsequent reference or reflection. Moreover, teachers invest on  average 12 to 15 hours per lecture, much more than a traditional  teacher takes to prepare and deliver the face-to-face lecture.  Most of our participants also find that they end up spending more  than the average of six to eight hours of class work that our courses promise. This may be partly due to the keen involvement  that such courses arouse among all. Some e-learning systems also opt  for more asynchronous activities, including group exercises such as  class assignments and use video  links or other multimedia facilities, depending on whether a diplomatic  service can provide broadband connectivity to all its missions abroad. <!–Ads2–> What about a downside? One can visualize a few potential problems with e-learning. First, if the class fails to establish internal rapport, the entire process becomes very mechanical. Second, it is possible that if many  ndividuals from a single organization take part, and if  in addition the faculty is also drawn from the same organization,  here could be some inhibition among the class in setting out their honest views, especially in written format. Third, if the e-learning class is told that their class  performance will be used for major internal evaluations, such as  promotions, one can easily encounter breakdown or reluctant participation. The moral: treat the  new medium with caution, and do not overload it with an excess of  frills or expectations; treat the first experiences as experimental, learn  as you go along! In sum, the e-learning format is a fascinating addition to the  repertoire of professional education. It is a work-in-progress, with each course that is run,  offering new insights for better application of the format. ?

Rumi Mallick [RUMI@CSDMS.IN], Centre for Science, Development and Media Studies, India

It is approximately 100 years since John Dewey began arguing for the kind of change that would move schools away from authoritarian classrooms, to environments in which learning is achieved through experimentation, practice and exposure to the real world. Today, technologies are  affecting teaching and learning in a big way.  Technology in Education or Educational Technology has been defined as the design, application evaluation and development of systems, methods, and materials to improve the process of human learning (Association for Educational Communications  and Technology, AECT definition). In very simple terms, educational technology includes ‘all  components of the  information technology used in the  delivery of educational materials.’ Since  he last decade, several initiatives (both private and  national governments) have experimented with diverse technologies in educacion. While  training in and training through computing devices has been the  most popular, the outreach of radio and television as education  instruction technol1ogies has been truly  utstanding especially with respect to remote rural areas. Other  educational technologies like alternative (low-cost, low-energy)  technologies, wireless and mobile learning platforms are also gaining  increasing attention. The use of educational technologies in particular and Information and  Communication Technologies (ICTs) in general, has been able to change the instructive  ducation process to a participative lifelong  learning process, be it the cost heavy computer-enabled learning process or the cost-effective  community  radio based mass  education process.

The changing role of   educational  technologies in the developing countries
Most technology integration in  schools in developing countries is premised on the realistic  perception that technologies are becoming increasingly pervasive in  the growing  nowledge-based economy and inaction will lead to  marginalization of these  ountries. With globalisation and the global  agenda to develop knowledge societies, the need to constantly upgrade the skills of human resources has  ecome crucial. Education and training has become the foundation of globalcompetitiveness with technologyenabled education as the key to human  esource development that can meet the challenges of such competition. While developing countries have responded through national education strategies with a focus on integration of technologies mainly ICTs at all levels and  for all purposes of education, most technology interventions in the education sector in many developing countries – especially the most poor, have been through small, uncoordinated pilot projects  initiated by non-governmental groups while most governments in these countries are struggling to cope with the challenges of infrastructure and implementation and education for all. Recent advances in and applications of technologies, have also demonstrated how  increasingly critical private sector involvement and investment is becoming in assuring that the promises of computer and communi-cations  capabilities proliferate in the sphere of public education to fulfillment of national education  and training missions.

Is it worth investing in technology for education?
While the non-governmental organizations in developing countries have experimented with diverse media and communication  technologies with a focus on improving learning outcomes of  students, most government responses have been in integrating technology training (mostly  computers) in schools. In a drive to ‘prepare for the digital  tomorrow’, a number of nationalgovernments are allocating funds to ‘create a digitally literate’ young  generation who have the skills to “survive’ and contribute to the  digital era. In the past years the ‘online’ (Digital Divide Network,  Bytesforall, to name a few) and the ‘offline’ community (the various groups led by the Non- Governmental practitioners) have been debating on several issues of technology especially ICTs in education. Although much  literature exists that tell the story of successful ICT in education endeavours, there is lack of adequate research instruments, indicators and even qualitative evaluation methodologies that aresufficiently general and sensitive to capture the impact in a variety  of situations. Questions have been raised – “What are the outcomes of ICT in education? What are the lessons we have learnt from good models? Are these models scalable? What does all of these cost? A recent study initiated by  infodev (www.infodev.org) on monitoring  and evaluation of ICT in educationprojects has raised these similar  questions. The study indicates that “relatively little is actually known about the effectiveness of  investments in ICTs in education, in promoting educational reform in general, and Education for All(EFA) goals in particular. Despite the billions of dollars of investments in ICTs in education. little hard evidence and consensus  exist on the proper, cost-effective utilization of ICTs to meet a wide  variety of some of the most pressing educational challenges  facing the developing world. To be sure, some good work has been done. These lessons do not seem to  be informing policy related to education in a significant way.”  Seemingly, there is a glaring gap between ‘what has worked on ground’ and a relevant actionable  knowledge base, which can help policy-makers in making more  ‘informed’ rather than ‘assumed’ decisions. Such divergent  scenarios have often resulted in imprecise policies that have allowed the introduction of  computer-aided teaching andlearning in schools without a follow-up with adequate teachers  capacity building programmes and change management in the  education system The message is clear. National policies on ICT in education is not  enough, there needs to be adequate evaluation and monitoring  echanism that can assess theICT related investing in progress in the developing countries. This can provide a more credible evidence-based future of ICT in  education in development andresult in more comprehensive policies in future.
The scenario in India for ICT and education stems from several initiatives that have created enabling environment at the policy and strategy levels. With the advent of the dedicated educational satellite EDUSAT, the ecosystem is  ready to absorb new technologies for achieving the educational  goals. We review some key policy and programme interventions at  he national level. The Right to Information Act The Government of India on June  15, 2005, passed the much awaited Right to Information (RTI) Act  which came into force on October 12, 2005 (120th day of its enactment on June 15, 2005). The  Act extends to the whole of India except the State of Jammu and  Kashmir. The new law replaces a weaker law enacted by the Lok  Sobha (Lower House in Parliament) in 2002. As the  globally renowned rights activist Aruna Roy and Magsaysay  Award winner points out, “For a country proud of its status as the world’s  largest democracy, this is one more step towards making its institutions  and authorities accountable and answerable to the people they  serve”. In view of making the law truly participatory, progressive and  meaningful, the RTI portal (www.righttoinformation.gov.in) of the National Informatics Centre  (www.nic.gov.in) was made a central repository of information for citizens to access information  through a user-friendly search engine.  Citizens can now get more information on the educational   pportunities available to them, citizens can now request for information on funds received,  budget allocations and funds spend by their local authorities for  education and can judge by themselves the educational performance of their district/town/  city. The University Grants Commission (UGC) is responsible  for monitoring the workings of institutions of higher education.  The UGC has notified all institutions of higher education to disclose various kinds of  information (as elaborated in the ‘Returns of Information by Higher  Education Institution’, formulated by the UGC).  The Right to Education Bill  The Right to Education Bill gives effect the Article 21A of the 86th  Constitution Amendment Act, affirming that every child between the age of 6 and 14 years has the  right to free and compulsory education. This Act entails the state  to ensure a school in every child’s  neighbourhood conforming to the minimum standards defined in the
Bill. The Bill entails, also free  education in all government schools and private schools to  provide free education to at least 25% of children from weaker sections. The Bill also makes it  mandatory for the state/Union Territories governments to determine every year the  requirement of schools, facilities, and their locations; establish  additional schools as required; deploy teachers and create  facilities for their training. The states are also required to develop a mechanism to monitor enrolment, participation and  attainment status of every child, and take corrective steps wherever  required. The states also need to make information in this regard available in the public domain,  including on an on-line basis. However, although this Bill  heralds a positive step toward Universal Primary Education, the   Bill has been criticised on several grounds. Not only does it fail to acknowledge factors such as  poverty that forces children out of  schools, it also does not make any provision for children below 6  years and above 14 years. Anil Sadgopal, a former dean of Delhi University’s Department of Education, notes that this exclusion  of some age groups contradicts the United Nations Convention on the  Right of the Child, which describes a child as “every human being below 18 years”. “The government of India is a signatory to this  convention. By excluding those under six, we are ignoring 170  million children,” he says. Educationalist claims that exclusion of children below 6 year  will mainly affect the poorest section of the society. The National Curriculum Framework (NCF)  Prepared by the National Council for Educational Research and Training (NCERT), the NCF  mphasises the words learning without burden and child-centred  education repeatedly. Its array of suggestions, includes cutting down  on the number of textbooks, making assessment methods flexible, and promoting more  inclusive learning. The pathbreaking suggestions of the new  National Curriculum Framework has put the child firmly at the centre of its proposals. The NCF  has been “sensitive” to the needs of children and understands that  the ultimate goal of education is to “motivate”. The framework also has a section on Teacher Education for Curriculum Renewal, which, recognizes that, “Attempts at curricular reform have not been adequately supported by teacher education” and suggests strategies for organising teacher training  programmes. The NCF has devoted a chapter to School and Classroom  Environment, mentioning that enough attention has not been paid to the importance to the  physical environment for learning and heads of school and block functionaries needs to  focus on ensuring that at least minimum infrastructural requirements are met. It also mentions that the ideal number of students in a class should be  round 30. The nation waits to witness how NCF recommendations will become a reality in the  face of crumbling basic infrastructure and  in most cases no infrastructure in schools.  The Sarva Shiksha Abhiayan (SSA)  Sarva Shikhsha Abhiyan  (Education for All), launched in  2001, is a flagship programme of the national   overnment. It aims  that all children complete 5 years  of primary schooling by  2007 and  all children complete 8 years of  elementary schooling by 2010.  The  SSA program combines  centrally set targets and norms for  planning and   osting with  decentralized management, bottom  up planning, community  mobilization, and social audits.  SSA funds annual work plans  submitted by states and districts to  meet the targets. SSA provides  ample flexibility to design   ocally    pecific strategies, encouraging  partnerships with nongovernmental organizations and requiring community oversight to ensure transparency and sustainability. In 2005, SSA approved the ICT@school initiatives, which involves the introduction of  computer education in the primary level in several states.  Mission 2007: Every village aknowledge centre.

This national capacity building programme received support of Rs. 100 crores (USD 22.2 million)from the Government of India in  its Union Budget 2005-06,  hich was presented by the Finance  Minister (FM), P Chidambaram. The FM said that the Government shared the goal of Mission 2007, which is to establish village knowledge centres in every Indian village by the 60th anniversary of India’s Independence Day. The Mission also received International  recognition at the World Summit on Information Society (WSIS  2005) where the International Telecommunication Union (ITU),  an arm of United Nations formally accepted ‘Mission 2007’ as a  flagship programme of its ‘Connect the World’ initiative. The UNDP and the Swiss Development gency (SDC) also joined  the International support group for the  programme along with International Development Research Centre (IDRC), Canadian  International Development Agency (CIDA), the United Kingdom’s  Department for International Development (DFID), the World Bank, UNESCO, WHO, FAO,  GKP, amongst others, promising financial support and help in  technology deployment.