Metrics for Evaluating Information Systems

10. Máj, 2017, Autor článku: Pavlík Lukáš, Informačné technológie
Ročník 10, číslo 5 This page as PDF Pridať príspevok

p18957_iconThe paper discusses the issues and methods of measuring corporate performance. In the first part of the paper, the term metric is defined as well as classification of metrics. Critical factors for the management of information systems and the loop principle in management processes are also presented. The Balanced Scorecard method, which is one of the most used metrics, is also presented and described in detail. The final part uses discussion to evaluate the topic and outline the possibil-ity to use the acquired knowledge for future development.

Introduction

The issue of performance measurement is currently the subject of interest in many fields. Since the pressure on developing information systems continues to grow, it is necessary to monitor costs and investments that are put into these activities. The managers of information systems of organizations deal with this issue, they must be able to measure the effectiveness of information systems in the company. New con-ditions which are changing quite rapidly in the turbulent environment of information systems require new methods and applications. Since most organizations today are dependent on their information systems, measuring performance and efficiency in computer science increasingly gains significance.

Performance measurement of information systems is done via several methods. Among the most common is e.g. the Balanced Scorecard (BSC) which is presented in the form of a simple causal model. BSC also presents a framework for communica-tion between different layers of management. Other methods may include the meth-odology COBIT which is one of the most comprehensive analyses to measure corpo-rate performance. This analytical method allows defining the correlation linkage between resources, requirements and processes in the field of information systems.

IT management is effective and beneficial if the information system supports achieving business goals and optimal business processes. Management and meas-urement is an integral part of business management. The information system objec-tives formulated in an information strategy are a part of an overall structure of cor-porate strategy and its goals.

One of the fundamental problems of the measurability of IT success lies in the fact that it is not the only prerequisite that directly or indirectly affects the achieve-ment of the objectives at the corporate level. There is an illustrative example: a situa-tion in which IT capabilities are completely optimized, however, due to the mistakes in motivating employees, the results are far behind scheduled expectations. This problem can be solved particularly by using the proper decomposition of perfor-mance metrics (e.g. BSC) so that the activators of IT influence on basic business objectives were possible to separate from the activators of personnel management and other environments. [1,2]

1. The concept of metric

The concept of metric can be characterized by several definitions:

  • it is a measurable indicator that is used to determine the quality, quantity and fi-nancial category (there can be a cost or price),
  • it is a specifically defined method of measuring and defined measuring range,
  • it is a performance indicator which combines individual and team efforts to achieve predetermined objectives.

1.1. Metrics their classification according to the measurement object

a) Hard Metrics (HM)

These are objectively measurable indicators to monitor the development of business objectives, business activity or are aimed directly at the customer. They can be easily characterized as:

  • easily transferable to the financial formulation,
  • easily measurable,
  • available at no extra costs.

Properly selected hard metrics should belong to the areas that directly affect the basic competitive advantages, in other words they are formulated according to indi-vidual perspectives of BSC (Balanced Scorecard) methodology if it has been imple-mented. Indicators also belong to hard metrics. The indicator is seen as an indicator which has desirable limits or upper and lower limit specified. If the real value shows devia-tion from the given limits, or limit, it is the deviation from the desired state. If the metric is not an indicator, it must have the desired state defined, to which the true value of the particular indicator is compared and evaluated. [1]

Hard metrics are divided into:

  • earnings (“Leg”) – are aimed at metrics of achieving the objectives,
  • performance (“Lead”) – are focused on performance measurement and support

b) Soft metrics (SM)

Soft metrics are used to measure and evaluate the level of computer support of individual processes and functional areas of the company in auditory way. This type of metric is designed in accordance with the purpose of application, e.g. to be applicable to evaluate the rate of:

  • meeting internal targets in specific areas,
  • achieving potential effects of innovation of the information system (IS). [1]

1.2. Critical success factors for implementing metrics into the fields of in-formation management and evaluation of effectiveness and benefits of the information system

FACTOR 1 – Existence of a corporate and information strategy

If the company has neither a clear concept for its future development nor the idea of what role will be assigned to the IS in this strategy, evaluating the bene-fits and use of IS does not make sense. [2]

FACTOR 2 – Understanding the basic properties of metrics

Metrics must:

  • help identify priorities which the firm should aim at when maximizing its value added,
  • be derived from the structure of business activities and goals that are decomposed from the corporate strategy,
  • not be only a financial indicator, but their interaction with the financial and value-creating system should be ensured, i.e. to apply a balanced system of hard and soft metrics,
  • guarantee a balance in meeting the short-, medium- and long-term goals,
  • be objectively measurable,
  • it must be possible to process them using mathematical and statistical methods such as time series, trends, etc.,
  • be repeatedly measurable, their cost acceptable,
  • be comprehensible and accessible to workers who work with them and influence them,
  • objectively interpretable. [2]

FACTOR 3 – Evaluation based on measurements

For successful application of metrics, it is appropriate:

  • to mainly evaluate trends rather than the results of individual measurements
  • to face a lack of transparency – managers of metrics must receive the monitored values in time according to a predetermined schedule,
  • to establish clear responsibility for the accuracy of measurements for individual metrics. [2]

FACTOR 4 – Knowledge and skills for working with metrics

One of the critical factors for successful implementation of metrics is the possession of skills and knowledge by workers who make up and evaluate the metrics model. When working with metrics, it is essential to use the knowledge of managers of specific areas and the abilities of creative work with measured results and their processing (if possible) through mathematical and statistical methods. To support the evaluation, it is appropriate to apply software tools. [2]

FACTOR 5 – Common sense

Every management field should be treated with respect to all its specifics. When applying metrics in management, common sense plays a major role. Me-chanical and obsolete applications, although using modern methods, will very like-ly prove ineffective in the end. [2]

p18957_01_obr01
Fig. 1 The basic portfolio of skills for working with metrics

1.3. Purpose of applying metrics in organization management

The implementation of a vision through the achievement of goals of a company or institution is a key success factor. This factor is in direct relation to the evaluation of the company’s success as owners, i.e. management. Creating the value belongs among the criteria for success. The value of the company is not only connected to its market value, but also to meeting the needs of customers, employees and the com-pany itself. [1] In current dynamic era, the following principles in business management ought to be respected:

  1. “If I do not know where I am going, I will not get where I want to.” Visions and missions formulate the idea of where the company should be directed in the long term.
  2. “If I know where I am going, I have to control my steps, otherwise it will not get where I want to.”
  3. “I cannot control my steps if I do not have senses that measure where I am currently standing.” It is necessary to focus on what ‘senses’ I need, or what are the things I need to measure and to what extent.
  4. “If I know how to measure, I can make corrections of future steps and, therefore, I have a real chance that I will get where I want to and will become excellent and successful”.

1.4. Loop principle in management processes

From the above mentioned principles, it is based on the basic loop structure of all management processes in the organization as defined by W. Edwards Deming, and it forms the basis for the definition of so-called Deming’s PDCA circle principle. [3]

p18957_02_obr02
Fig. 2 Loop principle in management processes

2. Balanced Scorecard (BSC)

Balanced Scorecard (BSC) is a method of transforming the mission and strategy into objectives and their metrics so that they will comprehensively cover not only the field of financial results of the company, but also all the basic fields of preconditions to achieve these results as well as conditions for the overall success of the company. [3]

In fact, approximately one to two dozens of goals and metrics are used at the corporate level; they are divided into partial BSC of individual organizational units (e.g. of informatics) and in some cases up to a personal “scorecard” for each employ-ee. As seen from the illustration above, the objectives and metrics are organized into four groups within the BSC: financial, customer, internal processes, and learning and growth. Their mutual relations determine the primary sequence of cause and effect and thus function as the definition of the simplified causal model of business. BSC also provides a framework, introduces a mean for communicating the mission and strategy among the individual layers of management and employees. It is used to ensure that all staff is instructed on activators (i.e. drivers) that affect current and future success. [3]

p18957_03_obr03
Fig. 3 Balanced Scorecard

2.1. Prospects for BSC

Financial perspective

Metrics must be implemented with regard to the life cycle of the company, on the success evaluation of which a specific group of metrics is always focused. Within the BSC, three basic stages of this cycle are differentiated:

  • a) Stage of Growth – is characterized by new investments, infrastructure development, etc.
  • b) The stage of growth saturation – is characterized by investments and reinvestments to eliminate bottlenecks, ex-pand the capacity, etc., with the emphasis on the maximum rate of return of the money invested.
  • c) The stage of maturity – called “harvest”, it is characterized by low investments in the support and maintenance of existing capacity, the main emphasis is on maximizing cash flow into the company. [33]

Customer perspective

Customer perspective defines how the company should be perceived by cus-tomers in order to meet business objectives while, at the same time, there was a re-alization of the company vision. Metrics of customer perspective should be fo-cused on market shares and market orientation of the company. For this purpose, such metrics should be defined that can help understand what is the driving force of the company in order to increase the market and customer segments. Another area of evaluation is the offered products in terms of their characteristics im-portant for customers, the level of relationship with the customer and the compa-ny’s image. [3]

Procedural and organizational perspective

When management is procedurally organized, it is necessary to precisely de-fine processes in terms of their position in the process model. In this context, we differentiate management, customer, and resourcing processes. The processes which bring the added value, which means they are customer focused, constitute a generic value chain, at the beginning of which the customer need is identified and the output of which is to meet the customer’s need. [3]

Perspective of development and growth

This view provides a stable basis for the previous three perspectives (finan-cial, procedural and organizational, customer). The perspective of development and growth supports dynamic setting of objectives and their continuous improve-ment. If the company wants to succeed, it must constantly innovate and acquire new knowledge, mainly through “organizational learning”. [3]

Conclusion

Metrics and their applications in a corporate environment is an integral part of the evaluation of the business effectiveness and performance in the industry. Metrics are based on basic attributes that are important for its implementation and determine their functional structure. Currently, information systems and infor-mation technology are an integral part of the organization and in terms of their importance; they also belong to the most vulnerable assets. Therefore, it is neces-sary to constantly improve their security and optimize their performance. The in-formation contained in information systems ought to be protected at the highest level, since its leakage, or eventually misuse, can have dire consequences for the company.

To prevent such events, insurance against cyber risk can help which provides coverage of loss or damage of information and can significantly prevent or reduce damage. To assess the value of the information system and the information con-tained therein, which are an important stage of the insurance process, metrics are mainly used. Their implementation provides measurable outputs in terms of which the price of the information system in the organization can be evaluated.

Grant affiliation

The paper is developed as a part of the IGA called Design
methodology for determination of prices also of the information system organization in terms of cyber risks, registered under IGA/FAI/2017/008

References

  1. UČEŇ Paul et al. Metrics in computer science: How to objectively determine the benefits of the information system. Praha: Grada Publishing, spol. Ltd., 2001, 140 pp. ISBN 80-247-0080-8.
  2. NOVAK, Daniel. Evaluation of the effectiveness of information systems. Brno, 2005, 40 p. Bachelor thesis. Masaryk University. Supervisor RNDr. Pavel Hain
  3. KOSTIHA, Francis. Measurement and evaluation of the quality of information systems. Prague, 2012, 225 pp. PhD thesis. Charles University. Supervisor Doc. PhDr. Rudolf Vlasak

Napísať príspevok