REVIEW JOURNAL : A Holistic Framework on Information Systems Evaluation with a Case Analysis

Title	:	A Holistic Framework on Information Systems Evaluation with a Case Analysis
Author	:	Petri Hallikainen, Information Systems Science, Helsinki School of Economics, Finland, petri.hallikainen@hse.fi Lena Chen, General Education Center, National University of Arts, Taiwan, lenachen@info.ntua.edu.tw www.ejise.com
Source	:	Elecronic Journal of Information Systems Evaluation Volume 9 Issue 2 2006 (57-64)
Abstract	:	Presents a framework for understanding IS evaluation in its broader. The role of IS evaluation is emphasized on integrating the IS development process into business development process. The framework is applied to analyze a single IS project in details. The results show that sometimes formal IS evaluation might not be important or necessary, but rather it may be more important, with an informal and flexible evaluation process, to quickly gain experience of a new kind of business and system to maintain a leading position in the competitive market.
Keyword	:	Information systems project, IS evaluation, organisazional context, holictic framework on IS evaluation
Smithson and Hirschheim, 1998; Irani, 2002; Peffers nd Saarinen, 2002) IS evaluation is a very difficult task involving a variaty dimensions and McAulay, 2002 Various stakeholders Powell, 1992 IS Investments often include intangible benefits and Saarinen and Wijnhoven, 1994 the benefits are often realized during a long period of time Irani and Love, 2001 Ad hoc pratices for IS evaluation are frequently reported and Lederer and Mendelow, 1993 simple methods, like payback period, are used in evaluation Peffers and Saarinen, 2002 Stated that evaluation of IT in financial terms may be biased toward the most easily measured benefits and prone to manipulation to justify predetermined investment decisions, resulting in systematic over or under-investment in IT. Farbey et al, 1992 At organizational level, contingency factors may include, for example, the industry situation (stable or changing) and the leadership role of the organization (pioneer or follow) See e.g. Farbey et al., 1992; Hochstrasser, 1990 At IS project level, contingency factors may include, for example, project types, project sizes, the type of expected benefits (qualitative vs quantitative), the stages of the system’s life cycle, and development and procurement strategy Peffers and Saarinen, 2002 Divided evaluation criteria into five broad categories as follows: -          Strategic value -          Profitability -          Risk -          Successful Development and Procurement -          Successful Use and Operations.
Aim	:	To provide an instrument for understanding IS evaluation in its broader context form both researchers and practitioners.
The Outset situation	:	-          organizational norms and values -          Project specific contextual factors -          resources given to the project
The actual outcomes	:	-          business development process -          IS development -          procurement processes
The outcomes of the IS project	:	Defined by the success of the system with respect to: -          investment perspective -          IS project implementation -          desired IS functionality
Research Methodology	:	A case study in a major Finnish Insurance company to investigate how IS evaluation was conducted in an environment where IT was extensively used in daily operations. The company have 3700 employees A questionnaire was used, because this study was a pilot study for a survey to be conducted later and one objective was to test the research instrument (Appendix 1) First interviewer: the corporate IS manager to get an overview of the use of information technology in the company and to select an appropriate project for the detailed analysis.
Findings of the case study	:	-          Project characteristics -          Development / procurement  process -          Evaluation process -          Success of the information system project
Discussion and conclusion	:	The first impression of the analysis is that evaluation is highly contextually specific. Besides the use of formal evaluation methods, the study was use informal evaluation process. Formal evaluation does not seem to be important or necessary in every case, The evaluation did not cause any major changes in the IS project. There were no major unexpected changes in the business  environment that would have caused a larger scale re-consideration  of the IS investment. The interviewees were rather satisfied with the evaluation process, but on the other hand, based on the study, they are not able to detect the actual effects of the above IS investment on the business functions of the company. They were not able to find any established practices for learning about IS evaluation itself. One of the challenges for the case company, and other companies, would be making tacit evaluation knowledge more explicit so that it could be exploited in future projects. The investigation of how companies adjust their evaluation procedures to the rapidly changing business environment into account would require a continuous evaluation process over the information system life cycle. Another challenging area for the future research is certainly the development of evaluation methodologies for new Internet technology based IS. In contemporary IS projects where development time is short, and modifications easy to make, collecting customer feedback efficiently would play the crucial role. Finally, the question how much IS evaluation should be formalized remains a contradictory issue and an interesting topic for the future research. It seems that the formaliaztion of conducting evaluation might not be applicable in the rapidly changing business environment of today, but evaluation knowledge should rather be developed and managed as a continuos process.