A Metrics Generation Model for Measuring the Control Objectives of  Information Systems Audit

A Metrics Generation Model for Measuring the Control Objectives of Information Systems Audit

-

English
11 Pages
Read
Download
Downloading requires you to have access to the YouScribe library
Learn all about the services we offer

Description

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007A Metrics Generation Model for Measuring the Control Objectives of Information Systems AuditMathew NichoAuckland University of Technologymathew.nicho@aut.ac.nzBrian Cusackbrian.cusack@aut.ac.nzfifth among the top IS issues facing the organisation. In Abstr actanother study conducted in the mid eighties byBrancheau and Wetherbe [8] on the key issues in IS, Information Technology governance (ITG) whichmeasuring the effectiveness of information systemswas a relatively new concept in the late 1990s, has ranked 9th on the list of the information systemsgained importance in the 21st century due to factorsmanagers and 4th on the list of general managers. At namely the collapse of Enron Inc, and the need for a the turn of the millennium, Markus and her colleagues better reporting and financial disclosure system as emphasised that system success is one of the most requested by the US Securities and Exchangeenduring research topics in IS [38]. In point “theCommission chairman in 2001. Subsequentmeasurement of information systems success has been legislations namely the Sarbanes Oxley Act (SOX) in on the research agenda for well over thirty years” [54: the United States and the Turnbull Guidance in the p. 3]. A recent study conducted by Pricewaterhouse United Kingdom provided further impetus for the need Coopers, sponsored by the IT Governance Institute for ITG. Other factors that ...

Subjects

Informations

Published by
Reads 21
Language English
Report a problem

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007
A Metrics Generation Model for Measuring the Control Objectives of
Information Systems Audit
Mathew Nicho
Auckland University of Technology
mathew.nicho@aut.ac.nz
Brian Cusack
brian.cusack@aut.ac.nz
fifth among the top IS issues facing the organisation. In Abstr act
another study conducted in the mid eighties by
Brancheau and Wetherbe [8] on the key issues in IS, Information Technology governance (ITG) which
measuring the effectiveness of information systemswas a relatively new concept in the late 1990s, has
ranked 9th on the list of the information systemsgained importance in the 21st century due to factors
managers and 4th on the list of general managers. At namely the collapse of Enron Inc, and the need for a
the turn of the millennium, Markus and her colleagues better reporting and financial disclosure system as
emphasised that system success is one of the most requested by the US Securities and Exchange
enduring research topics in IS [38]. In point “theCommission chairman in 2001. Subsequent
measurement of information systems success has been legislations namely the Sarbanes Oxley Act (SOX) in
on the research agenda for well over thirty years” [54: the United States and the Turnbull Guidance in the
p. 3]. A recent study conducted by Pricewaterhouse United Kingdom provided further impetus for the need
Coopers, sponsored by the IT Governance Institute for ITG. Other factors that prompt companies to give
(cited in [50: p. 8] on a sample of 7000 respondents, more importance to the management, control and
found that one of the top ten problems cited by these measurement of information systems include the risk
respondents was the “inadequate view of how well IT associated with information, the investments made by
is performing” and furthermore 80 percent are of the companies into the IT resource and the need to be
opinion that IT governance or some sort of governance competitive in the marketplace. All of these factors
mechanism was required to solve the issue.emphasize the requirement to measure the
Overall Information Technology (IT) spending is performance or effectiveness of information systems.
increasing at an alarming rate and it is estimated to be The state of performance of various entities, events and
about US $ 2.5 trillion in 2005 which is 50% of the total process of information systems give a ‘dashboard
corporate capital spend [34]. Taking into account the approach’ vision to management. In this paper a
importance of measurement in one of the largest assets metrics generation model is proposed for generating
(IS) in an organis ation, and the relevance of IS audit in metrics that can measure the key performance
the twenty first century, the authors took two models indicators and goals of the control objectives of CoBIT
namely the CoBIT and the GQM model from the IS field by applying the GQM model..
to generate metrics for information technology audit.
While CoBIT is an IT audit framework that had evolved 1. Introduction
over the last eight to ten years, the GQM is a metrics
generating model used in software engineering for
Measuring the effectiveness of info rmation systems
generating metrics to measure the various goal related
(IS) has been one of the top concerns of IS and
aspects of the software development process. The
corporate managers since the 1980s. A study
objective of this paper is to transpose the GQM model
conducted by Dickson, et al. [15] on the key
into CoBIT by taking the detailed control objectives of
information systems concerns, revealed that measuring
the CoBIT and follow the GQM guidelines for
and improving IS effectiveness/productivity ranked
Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 1530-1605/07 $20.00 2007 IEEE © 1Proceedings of the 40th Hawaii International Conference on System Sciences - 2007
generating metrics that can measure the process and acknowledgement that the board is responsible and
entities that are audited during an IT audit exercise. accountable for the effective governance of its
information and IT assets” [29: p.1].
2. IS Auditing Framework
3. Attributes of IS Measurement
Strous [51: p. 264] defined IT audit as “an
independent and impartial assessment of the reliability, It has been stated by DeLone and McLean [13] that
security, effectiveness and efficiency of automated out of the large number of studies conducted in
information systems, the organisation of the information systems in the nineteen-eighties, half of
automation department and the technical and them relate to identifying the factors that contribute to
organisational infrastructure of the automated information system success. There is overlapping of IT
information processing.” IT auditing is a new auditing and information systems measurement
profession that extends the concept of control in the attributes, such that some of the attributes of IS entities
form of quality assurance, benchmarking and and process are similar. This is evident from the
measurement. It is also used in some organisations to definition of IT auditing proposed by the Dutch
implement IT governance. CoBIT as an IT audit Association of Registered EDP auditors who stated
framework addresses the need for management and that “an IT auditor assesses and advises on the
control of information and information technology [36]. following aspects of information technology:
It focuses on five IT governance areas namely, effectiveness; efficiency; exclusiveness; integrity” [51:
strategic alignment, value delivery, risk management, p. 265]. A review of measurement factors in information
resource management and performance management systems literature shows a similar set of attributes for
[28]. Tools such as the balance scorecard (BSC) for evaluating information systems success namely
IT/business alignment [22], maturity models for efficiency of resources and effectiveness of the users
benchmarking, key goal indicators (KGI) for measuring [26]. Attributes of IS measurement namely: precision,
the outcome, and key performance indicators (KPI) for accuracy and reliability of information quality;
performance measurement which are within the CoBIT completeness, relevancy, timeliness, and up-to-
framework lends a multi perspective approach to IT datedness of information contents; format, clarity and
audit. quality of the IS product [46: p. 110] defines
While the control and governance of information ‘exclusiveness’ and ‘integrity’ from various
systems and related technology were considered a perspectives. Both of the disciplines attempt to
subset of management information system (MIS) during measure attributes of information systems success.
the early stages of MIS development, measurement and DeLone and McLean [13: p. 61] in their study on
tested measurement however, are more generally found information systems success stated that “if information
in the field of software engineering. The late 90s saw systems research is to make a contribution to the world
the emergence of the concept of IT governance that of practice, a well-defined outcome measure (or
emphasised high level control rather than measures) is essential. It does little good to measure
‘management’ with emphasis on compliance, control various independent or input variables, such as the
and measurement rather than generic ‘management’ of extent of user participation or the level of I/S
IS. Business orientation is the main theme of CoBIT investment, if the dependent or output variable —I/S
[36]. Commenting on this, Kordel [35: p.1] states that success or MIS effectiveness—cannot be measured
“to be successful, the business side of an organisation with a similar degree of accuracy”. DeLone and
has to be involved in and committed to what IT does. McLean’s definition of success at the three levels also
To deliver the services the organisation requires IT has reflects the overlap of IT audit measures with IS
to be managed by the business as a business.” Apart measures. At the technical level success is defined as
from the above motives, other reasons for the accuracy and efficiency of the system (Shannon and
widespread adoption of IT audit framework include the Weaver, 1946, cited in [13]). At the semantic level
enactment of SOX in 2002 in the US, the Turnbull success is the information conveying the intended
guidance and the Combined Code in the UK. Australia meaning. At the effectiveness level success is the
has developed its own standard for IT governance, the effect of the information on the receiver. At a functional
AS 8015:2005 standard which “is the first formal level, Chang and King, [12] focused their efforts on
standard for IT governance. It has emerged and creating a functional scorecard for measuring the
recognises that the heart of IT governance is the performance of information systems, based on three
Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 2Proceedings of the 40th Hawaii International Conference on System Sciences - 2007
system output dimensions namely system performance, multiple objectives. The financial aspects of software
information effectiveness and service performance. quality measurement, such as the return on investment
calculation (ROI) are also critical. Jones [30: p. 36]
states that “software quality measurements provide one
of the highest ROIs of any technology and are far 4. Measurement Relevance
easier to get started than a full reusability program.” He
cited the example of IBM to prove his point where Work on software measurement started during the
IBM’s dominance of the computer industry has been 1960s [18, 55] and the first dedicated book on software
attributed to its use of quality measures by its founder metrics was published by Tom Gilb in 1976 [18]. During
very early in the corporation’s history.the same time software complexity metrics that were
In IS field, metrics is a relatively new concept and is introduced by McCabe and metrics based on the
incorporated in the CoBIT framework. There is a lack of source code of programs by Halstead are used even
research in metrics or metrics generation in the IS field. today [55]. From 1980s onwards “there was recognition
The lack of metrics to measure information systems that the scope of measurement should include the
performance prompted Zahedi [53: p. 791] to complain entire business organisation” [17: p. 3]. The importance
that “although millions of dollars are spent onof measurement in software project management and
developing information systems, little attention hasquality as surance was highlighted by Ince et al. [27: p.
been paid to formal metrics of information system59] when they stated that “once something can be
performance.” The critical nature of IS for internal and measured, you move away from the world of opinion
external organisational success, was emphasised bytowards the world of fact”. They complained that “most
Boynton et al. [7: p. 32] when they stated that “ITmeasures of project progress are informal, and hence
resources are being used to solve business andopen to interpretation” while giving advice that “the
strategic challenges associated with cross-functionalcareful use of numerical measures can introduce
integration, coordination and control of mutuallyprecision and clarity to the process”. It has been stated
dependent value chain activities, and teamby Finkelstein (1982, cited in [19: p. 7]) that “one of the
development across organisational and geographicaims of science is to find ways to measure attributes of
boundaries.” From a financial perspective, measurement things in which we are interested.” Certainly interest
of IT resources is gaining greater importance due to the grew among the various IS stakeholders to measure IS
high risk of IT investments as is evident from the fact success or effectiveness as it was critical to the
that over 20% of the corporate IT budget, which is US understanding of the value and efficacy of management
$500 billion, does not achieve its objectives [34]. actions and IS investments [14].
Taking into account a broader perspective of
5. Performance Measurement in COBITmeasurement, that is from an information systems point
of view Basili et al. [5] stated that measurement is a
mechanism for creating a corporate memory as it helps The twenty first century has seen more
to support project planning, determine the strength and organisations eager to embark on IT
weakness of the current process and products, provide governance/auditing exercise for a variety of reasons
reasons for adopting a technique and allow to evaluate (mentioned in section 2). It has been predicted by
the quality of products and processes. Furthermore Pucciarelli et al. (1999, cited in [22]) that by 2003, 60
there are many instances where measurement has been percent of large enterprises and 30 percent of mid size
used effectively to inform management. A good enterprises had adopted a balanced set of metrics to
measurement framework is also instrumental in aiding guide business oriented IT decisions. The term
technical decisions based on facts and objective ‘performance’ is a multidimensional perspective of
evidence [10]. During the mid-1970s software metrics concurrent viewpoints (with dimensions as quality of
developed by McCabe and Halstead were used for work life, efficiency, effectiveness and quality), that
improving software quality and productivity [41]. impact productivity and profitability [11]. This
Measurement of performance is critical because quantitative view of performance has led researchers to
monitored measures get high visibility within an produce, adapt, and adopt measurement models and
organisation, and people strive to achieve high schema for measuring IS performance namely, the BSC,
performance with respect to these measures [1]. A and the Information Systems Functional Scorecard
balanced performance management system drives (ISFS) of Chang and King [12] while models namely
decisions that target optimizing the performance across
Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 3Proceedings of the 40th Hawaii International Conference on System Sciences - 2007
GQM were produced and used in the software senior management, but is not effective for translating
engineering field. those measures for everyone in the organisation as it
Measurement of performance that traditionally doesn’t tell companies what they need to do to reach
emphasised the financial aspect was extended to the targets it sets out. Kenny [31] pointed out that the
include more perspectives other than financial aspects. main reason why the BSC doesn’t work in all situations
Kaplan and Norton’s Balanced Score Card ensured is because the four perspectives developed by Kaplan
these additional measures drive organisations towards and Norton were originally modeled on the corporate
its strategic goals [22]. This shift by organisations to scorecard of Analog Devices, US which was a highly
measure performance through other perspectives using innovative organisation and thus may not fit all
the BSC saw many firms adopting the BSC. By the end organisations.
of 2000 “60 percent of Fortune 1000 companies have
experimented with the BSC’ (Silk, 1998, cited in [39: p. 6. Software Metrics and IT Audit
29] and according to Gartner Group (cited in [39]), over
50 percent of large US firms had adopted the BSC by Software engineering is a discipline that involves all
2000. Even out of those not currently using it, 43 those activities dealing with the various aspects of
percent planned to use one soon and a further 48 software development, implementation and control. The
percent were considering (Downing, 2001, cited in [39]). use of quantitative methods for measuring the software
The popularity of the BSC has seen academics and development process was established in the 1970s
organisations using the BSC framework in measuring (Cote, 1988, cited in [41]). It was the same time when
entities and process other than what BSC was intended interest grew among the researchers to measure the
for. One such study was conducted by Rosemann and various aspects of information systems effectiveness
Wiese [45] that involved the successful use of the BSC and success. CoBIT focused on metrics namely the KPI
for measuring the performance of ERP software. They and KGI for measurement. Goals and metrics are core
added a fifth perspective called the project perspective concepts of CoBIT 4.0 and when goals are set, it is
since they argued that the original four BSC important to measure the achievement against these
perspectives were inadequate to measure IT resources. goals [23]. The metrics are categorised into KGI and
The measurement and management methodology of the KPI where business goals, IT goals, IT processes and
BSC is well suited to the IT governance and the activity goals can be measured by KGI, while KPI
alignment process [22] that it can be successfully measures how well the process is performing.
applied to CoBIT. Apart from its use in software development
Even though the data from Gartner Group measurement the six primary uses of metrics include
suggests that between 40 and 60 percent of large US goal setting, improving quality and productivity,
firms will have adopted balanced scorecards by 2000, it project planning, managing and improving customer
had been claimed that 70 percent of balanced scorecard service [41]. Even though the word ‘metrics’ originated
implementations fail [42]. A similar concern was voiced from software engineering, recent trend had seen its
by Brock, et al. [9: p. 4] when they expressed their acceptance and diverging use in the information
opinion about the BSC by stating that it is inadequate systems domain which is a superset of software
for IT project management, due to two main reasons. engineering. The universality of the metrics program
First of all “its theoretical constructs do not explicitly was highlighted by Feigenbaum [20] when he stated
specify which areas or factors must be considered that metric data is useful for establishing quantitative
under each of its four high-level perspectives” and the improvement objectives for company management,
secondly the four perspectives in the BSC do not measuring productivity, and can be used in conjunction
adequately reflect relevant project management focus with a corporate quality improvement program that are
areas. Neely and Bourne [42] further goes on to say akin to Total Quality Management. Apart from the
that there are two reasons why measurement initiatives measurement aspects of metrics it is also used in the
fail. One is the poor design of the measurement system management and monitoring of software projects.
and the other is the difficulty in implementing it. In a Anderson, [3] stated that if metric data is made
study by Professor Claude Lewy (cited in [32]) on available to software project managers on a frequent
Dutch firms that implemented the BSC it was found that basis, corrective actions can be made to the project
over half of the scorecard implementations fail. One plan to increase the probability of successful
reason for its failure was given by Paul Hesselshwerdt completion of the project, thus relating metrics to
(cited in [32]) who stated that the BSC works well for project success which ultimately lead to information
Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 4Proceedings of the 40th Hawaii International Conference on System Sciences - 2007
systems success. Moller and Paulish [41] went further organisation [6, 40] for implementing an effective
in translating this success factor into user satisfaction measurement program.
by stating that a successful metrics program will
ultimately result in higher quality software system 7. Integrating the GQM Model into CoBIT
products which will in turn increase customer
satisfaction. This success factor is one of the six According to Fenton and Neil [18] there are two
dimensions of information systems success mentioned works with borrowed ideas from TQM that effectively
by DeLone and McLean in their article of 1992. address the mechanics of implementing metrics
Measurement is the process employed by the IT programs. One is the work of Grady and Caswell in 1987
auditing frameworks to oversee the level of assurance, which is an extensive experience report of a
compliance, quality and control. The link between companywide software metrics program and, secondly
software metrics and IT auditing is evident in the the work of Basili, Rombach and colleagues in 1988 on
statement of Fenton and Neil [18: p.149] when they GQM which was a top down approach for quantifying
stated that software metrics not only includes a “wide metrics from goals. This paradigm developed in 1984 [5]
range of activities concerned with measurement in has been applied worldwide in numerous companies,
software engineering” but also “includes quantitative namely NASA, Ericsson, Bosch, Schlumberger, Digital,
aspects of quality control and assurance – and this Motorola, HP and AT & T [52, 40]. The modified
covers activities like recording and monitoring defects version of the model has been used effectively in Nokia
during development and testing.” of Finland for measuring software development [33].
The IT auditor will find the CoBIT version 4.0 of Commenting on this model’s ability to turn goals into
immense use since it gives a detailed list of metrics values, Latum et al. [37: p. 19] stated that the GQM
under each of the 34 high level control objectives that approach is “a systematic way to tailor and integrate an
can be applied to goals, but the auditor’s job of organisation’s objectives into measurable goals and
correlating the metrics to the detailed goal(s) may refine them into measurable values” thus implying that
present implementation problems as they have to trace this model can be used by the CoBIT framework in
the metrics to the goals and objectives. One of the translating its 34 high level and 318 low level objectives
limitations attributed to CoBIT was highlighted by into measurable values. The model also addresses the
Solms [49: p. 100], when he said that “it is not always ‘how’ aspect of implementation as it not only helps in
very detailed in terms of ‘how’ to do certain things.” formulating an effective measurement program but also
The DCOs (Detailed control objectives) are more guides in implementing it [47]. The popularity of the
addressed to the ‘what’ must be done. A similar GQM approach has been emphasised by Fenton and
statement was echoed by Anthes [4] when he stated Pfleegar [19] who stated that the GQM approach to
that it tells what to do but not how to do it. Also it process and metrics has proven to be an effective
doesn’t deal directly with software development and IT approach to selecting and implementing metrics.
services. Hence the specification of a metrics The application of the GQM model involves the
generation model for measuring the control objectives understanding of its three inherent levels at the
of CoBIT will support effective CoBIT implementation. conceptual level where the goals are determined for a
Given the importance researchers have attached to the set of products or process, at the operational level
selection of metrics, it is essential that metrics selection where a set of questions characterize the object of the
is a critical aspect for successful measurement measurement, and at the quantitative level where a set
programs [25, 21]. One of the problems with metrics of objective or subjective data is associated with every
cited by Latum et al. [37] in their study on Schlumberger question [5]. Thus the three components of a GQM
RPS (A Finnish company) is that the measurement data model comprise of a goal statement, a set of questions
was scattered throughout the organisation that it was derived from the goal and a set of metrics that
difficult to trace the goal, purpose or context of the quantifies the generated questions. A GQM goal is
measurement. Hence it is imperative that as far as described with five dimensions that express the object
possible metrics should be derived from the goal which to be measured, the purpose of measurement, the
is the strength of the GQM model when applied to measured property of the object (quality focus), the
CoBIT. The auditor’s job is to use the GQM model in subject of measurement (viewpoint), and the
selecting appropriate metrics through defining measurement context (environment). The set of derived
measurement goals tailored to the specific needs of the questions refine the goal by focussing on the quality
models and relate to variation factors which describe
Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 5Proceedings of the 40th Hawaii International Conference on System Sciences - 2007
attributes that affect quality. The third part is a set of how the DCO of CoBIT can be transposed into the goal
metrics derived from the questions that provide metric definition template of the GQM and defined according
to answer each question. to GQM principles.
There is an overlap between CoBIT and the GQM Determining what to measure or what is being
model. In figure 1 a cascade effect is shown where the measured involves knowing the goal of the project or in
business goals are ultimately broken down into metrics. the case of COBIT the goals of the acquisition and
While in GQM the process starts with defining the implementation phase. After defining the goal, the next
various dimensions of the goal, in CoBIT each of these step is to develop a set of questions. Table 2 illustrates
dimensions are implied in the high or lower level control how a GQM defined goal taken from CoBIT can be used
objectives. Thus instead of generating a goal for the for generating questions that define the goal.
information systems entities, the DCOs of CoBIT can Developing a set of questions involves taking into
be considered for defining the goal in GQM and this account the guidelines for product and process
aspect makes it easier to apply the GQM model in oriented goals given by Basili and Rombach in 1988.
CoBIT. Theoretical testing of the model involves taking They proposed three sets of questions for the process
a low level (detailed) control objective from CoBIT to oriented goal. These are the definition of the process,
generate metrics using the GQM framework. Out of the the definition of the quality perspectives of interest,
four domains of CoBIT, the acquisition and and the feedback from using this process relative to the
implementation phase of CoBIT is the one with closest quality perspective of interest. Table 2 gives the set of
relation to software development processes. Selecting questions developed by the author using the three
the questions involves three major categories of perspectives outlined by Basili and Rombach in the
questions that need to be addressed for each process model. Even though GQM does the job of generating
under study, namely the definition of the process, the metrics, there is a strong link between GQM and CoBIT.
definition of the quality focus, and feedback relative to Metrics are derived from questions, which in turn are
the quality focus or focuses from using this process [6, derived from goals defined according to GQM
16]. While giving guidelines for generating metrics, principles and taken from the CoBIT framework. CoBIT
Basili and Rombach [6: p. 760] suggested that 4.0 provides a robust and convenient framework for
subjective as well as objective metrics can be selected applying the GQM approach to metrics generation
and those aspects that cannot be characterised since the goal for each IS entity, process or activity are
objectively can “at least be categorised into clearly defined in the form of 318 detailed control
quantitative (nominal) scale to a reasonable degree of process. It is to be noted that the GQM model is more
accuracy.” suited for lower level rather than high level control
For the purpose of integrating GQM and CoBIT, a objectives and goals [48]. Thus the GQM goal fits
detailed control objective is taken from the above perfectly with the DCOs. Moreover the DCOs of CoBIT
domain. In CoBIT 4.0, the detailed control objective are in turn derived from the high level control
under the high level control objective (AI 7) that objectives. Basili et al. [5] stated that in order for
corresponds to software development is AI 7.3 measurement to be effective, the goals should be
(Implementation Plan). It reads as: “Establish an specific, applied to all life-cycle products, processes
implementation plan and obtain approval from relevant and resources. While CoBIT helps in providing the
parties” [28: p. 98]. Table 1 modified from Basili et al. [5] goals, generating a set of questions is a critical aspect
incorporating the CoBIT AI 7.3 goal best illustrates in deciding the metrics to be generated [48].
Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 6Proceedings of the 40th Hawaii International Conference on System Sciences - 2007
COBIT Corporate Goals
aligns
corporate
goals with
IT goals IT goals
CoBIT
Control Objectives
Goals Detailed Control
based on Objectives/
purpose, GQM Goals
issue,
object and
viewpoint
Set of Questions
GQM
Metrics or a set of
metrics for each
goal
Figure 1 CoBIT-GQM measurement framework showing the linkage
Hence the elicitation of questions can be done has to be defined from the perspective of the members
through a survey of the project team or via of the project team and based on model of software
brainstorming [37, 48]. Furthermore, the importance of processes and products. This ensures that the goals,
the project team in defining metrics was highlighted by questions and metrics generated are customised to the
Solingen et al. (1995, cited in [48] by stating that for particular organisation.
ensuring correctness and completeness of metrics, it
Table 1: Use of the GQM Model (The basic GQM Model, 1988) in defining the goal of CoBIT (AI 7.3)
DCO of CoBIT 4.0 (AI 7.3) defined according to GQM goal perspective
Object to be Purpose of Measured Subject of Measurement
GQM measured measurement property measurement context
(quality focus) (view point) (environment)
Implementati Better Clear, detailed Head of Organisation/
AI 7.3 on plan understanding of and Development and department
the plan for comprehensive CIO
approval
The last step in the GQM model involves generating technique or by using focus groups. The GQM model
metrics from the quantifiable questions. Each question can be used by organisations at any stage of IT audit
can generate one or more questions or one metric can process. For example it can be used during CoBIT
be generated from two or more questions through the implementation for getting more specific measures and
use of surveys, brainstorming sessions, Delphi even long after CoBIT have been implemented as an
Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 7Proceedings of the 40th Hawaii International Conference on System Sciences - 2007
add-on tool. If an organisation already has a set of metrics also change leading to re-specification of the
metrics then this set can serve as a valuable database goals, questions and metrics. To speed up the metrics
from where metrics can be selected or modified for the generation process, basic automation can be done
quantifiable questions generated from the model. The using spreadsheet. There are solutions to automate the
generation of metrics for all the identifiable information GQM model namely the application software called
systems goals in an organisation is a large undertaking MetriFlame [52] developed and successfully tested by
and once this process is over, it is easy to VTT Electronics of Finland, for generating metrics for
generate/modify new metrics for any new sub process. software development process. This shows that a
Moreover the metrics generation process is a dynamic similar solution (GQM suited for IT audit framework)
process whereby as the organisations change, the can be developed for the IT auditing process.
Table 2: Use of the GQM model (The basic GQM Model, 1988) in defining and generating
questions based on the goal.
(Sample questions derived from the goal AI 7.1 defined as per
Quality of use GQM guidelines)
Definition
- How far is the plan clear, effective and user friendly in
of the
conveying information?
process
Domain of use - What is the competence of the persons (s) involved in the
Implementation Plan?
- How far is the plan user friendly?Quality model used
Validity of the model - Is user friendliness/effectiveness of the implementation plan an
Definition
appropriate way to measure its quality?Validity ofof the
- Are the factors for user friendliness/effectiveness able to Data
quality
collect the right information to measure it?
perspective Model effectiveness
- How far is the quality of the results (reliable/robust)?
Model Substantiation
- Are the results consistent from various perspectives?
- What is the quality level of the present implementation plan?
Quantitative feature - quality
Feedback
- What are the problems regarding quality of the implementationMajor problemsfrom using
plan?
this process
Suggestions for improvement
- How can we improve the implementation plan?
that a study conducted by the Hackett Group (cited in 8. Automation
[42]) found that the average organisation spends 25,000
person days on performance measurement andAutomation of data collection is one of the factors for
planning for every US $ 1 billion worth of sales. This metrics program success [25, 43]. The importance of
necessitates the need for automation.automation was stated by Solingen et al. [48: p. 9] by
proposing three sets of tools for automating GQM
namely, data processing tools, data representation
9. Limitations and areas for further researchtools and data displaying tools and at that time they
complained that “currently no commercial GQM data
analysis tools exists.” Differding [16: p.17] echoed the GQM is a model developed for generating metrics in the
same comment that “no commercial software is software engineering field. Application of this tool into
currently available that directly supports defining GQM other areas such as IT auditing that includes diverse
plans in a top-down fashion, and interpreting data process, products, entities and activities may require
according to plans.” Automation of metrics generation further refining to suit the auditing framework -
process can save organisations valuable human and especially the template for developing the questions.
financial resources. According to Neely and Bourne Secondly application of the GQM model to CoBIT will
[42] performance measures are a very costly affair such not itself assign a high level of quality standards to the
Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 8Proceedings of the 40th Hawaii International Conference on System Sciences - 2007
auditing process as this measurement tool has to be
audited further to determine its validity and robustness. 11. References
While comparing the GQM model in its ability to relate
with the CMM maturity level, Solingen and Berghout [1] Abu-Suleiman, Boardman, A. B., & Priest, J. W.
[48] attributed a maturity level of 3 if a GQM program (2005). A Framework for an Integrated Supply Chain
was fully and effectively implemented. According to Performance Management System. International Journal
them level 4 can only be reached if the GQM program is of Production Research, 43(15), 3287-3296.
audited, while level 5 would need to be a fully
integrated measurement tool environment and the [2] Al-Passori. (2000). Executive Directions. Stamford:
program subjected to a cost benefit analysis. The META Group Inc.
above demonstration is a theoretical application of
metrics generation using the GQM model in an auditing [3] Anderson, O. (1990). The use of Software
framework and if one model is successful in one context Engineering Data in Support of Project Management.
(software engineering), it doesn’t mean that it will be Software Engineering Journal, 5(6), 350-356.
successful in another context (Measurement in IT
auditing) unless the theoretical model is subjected to [4] Anthes, G. H. (2005). Quality Model Mania.
empirical testing in a real world environment. The Retrieved 14/02/2005, 2005, from
impetus for taking the model stems from the fact that http://www.computerworld.com/ developmenttopics/de
the CoBIT framework has borrowed concepts, ideas velopment/story/0%2C10801%2C90797%2C00.html
and models from software engineering (KGI, KPI,
metrics, CMMI) and general management (BSC) areas. [5] Basili, V., Caldiera, G., & Rombach, D. (1994). The
The increasing use of the CoBIT tool by global Goal Question Metric Approach in Encyclopedia of
organisations [44, 36, 24, 2] makes it a contemporary Software Engineering (pp. 528-532): John Wiley and
framework for measurement. Sons Inc.
10. Conclusion [6] Basili, V., & Rombach, D. (1988). The TAME Project:
Towards Improvement - Oriented Software
In this paper the application the GQM software Environments. IEEE Transactions on Software
engineering measurement tool was outlined to provide Engineering, 14(6), 758-773.
metrics for CoBIT’s lower level objectives.
Theoretically the paper had demonstrated that it is [7] Boynton, A. C., Jacobs, G. C., & Zmud, R. W. (1992).
possible to build a trustworthy measurement system. Whose Responsibility is IT Management? Sloan
This is a step towards constructing the numerous dials Management Review (Summer), 32-28
that record the state of an organisation’s information
system entities, activities and processes. It also fills a [8] Brancheau, J. C., & Wetherbe, J. C. (1987). Key
significant gap in the CoBIT literature that has caused Issues in Information Systems Management. MIS
debate and concern regarding the detailed measurement Quarterly, 11(1).
and implementation aspect of CoBIT. Furthermore, the
issue of standardised metrics in CoBIT has been [9] Brock, S., Hendriks, D., Linnell, S., & Smith, D.
addressed since GQM can be used to customise a set (2003). A Balanced Approach to IT Project
of metrics to an organisation. This can provide a Management. Paper presented at the 2003 Annual
competitive edge in four areas. First it can give more Research Conference of The South African Institute of
specific and quantitative information on the present Computer Scientists and Information Technologists on
state of an organisation’s information systems; Enablement through Technology (SAICSIT 2003),
secondly the information can be used to improve the Johannesburg, South Africa.
existing IS structure for the attainment of the stated
quality objectives; thirdly the framework can provide [10] Brown, M., & Goldenson, D. (2004). Measurement
transparency in IS operations due to better control and and Analysis: What Can and Does Go Wrong? Paper
monitoring; and finally, the process can be a much presented at the 10th International Symposium on
needed step for full compliance to the various Software Metrics, Chicago.
regulations and auditing requirements. Further empirical
testing is planned to stabilise and validate the model.
Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 9Proceedings of the 40th Hawaii International Conference on System Sciences - 2007
[11] Buglione, L., & Abran, A. (2005). A Model for [22] Grembergen, W. V. (2000). The Balanced Scorecard
Performance Management and Estimation. Paper and IT Governance. Information Systems Control
presented at the 11th IEEE International Software Journal, 2.
Metrics Symposium, Italy. [23] Grembergen, W. V. and S. D. Haes (2006). "Goals
and Metrcis: Core Concepts of COBIT 4.0." COBIT
[12] Chang, J. C.-J., & King, W. R. (2005). Measuring Focus 1: 2-7.
the Performance of Information Systems: A Functional
Scorecard. Journal of Management Information [24] Guildentops, E., & Haes, S. D. (2002). COBIT 3rd
Systems, 22(1), 85-115. Edition Usage Survey: Growing Acceptance of COBIT.
Information Systems Control Journal, 6, 25-27.
[13] DeLone, W. H., & McLean, E. R. (1992).
Information Systems Success: The Quest for the [25] Hall, T., & Fenton, N. (1997). Implementing
Dependent Variable. Information Systems Research, Effective Software Metrics Program. IEEE Software
3(1), 60-95. (March/April), 55-65.
[14] DeLone, W. H. and E. R. McLean (2003). "The [26] Hamilton, S. and N. L. Chervany (1981). "Evaluating
DeLone and McLean Model of Information Systems Information System Effectiveness - Part I: Comparing
Success: A Ten-Year Update." Journal of Management Evaluation Approaches." MIS Quarterly 5(3): 55-69.
Information Systems 19(4): 9-30.
[27] Ince, D., Sharp, H., & Woodman, M. (1993).
[15] Dickson, G. W., Leitheiser, R. L., Wetherbe, M., & Introduction to Software Project Management and
Wetherbe, J. C. (1984). Key Information System Issues Quality Assurance London: McGraw Hill Book
for the 1980s. MIS Quarterly, 8(3). Company.
[16] Differding, C., Hoisl, B., & Lott, C. M. (1996). [28] ITGI. (2005). COBIT IV. Retrieved 16/12/2005, from
Technology Package for the Goal Question Metric www.itgi.org.
Paradigm (Internal Report No. 281/96). Kaiserslautern:
University of Kaiserslautern. [29] IT Governance Ltd. (2005). Board Briefing on IT
Governance. Retrieved 13/03/2006, 2006, from
[17] Du, G., Ngolah, C., & Thornton, S. (2003). Software www.itgovernance.co.uk
Measurement (pp. 1-29): University of Calgary.
Retrieved 20/03/2006, from [30] Jones, C. (1996). Applied Software Measurement
http://members.shaw.ca/sdthornton/seng621/webdoc/s New York: McGraw Hill.
oftware_measurement.pdf
[31] Kenny, G. (2003, March 5). Strategy: B alanced
[18] Fenton, N. E., & Neil, M. (1999). Software Metrics: Scorecard - Why it isn't working. New Zealand
Successes, Failures and New Directions. The Journal of Management, 32-35.
Systems and Software, 47, 149-157.
[32] Kersnar, J. (1999). Hitting the Mark. Retrieved
[19] Fenton, N. E., & Pfleeger, L. (1997). Software 14/03/2006, from
Metrics- A Rigorous & Practical Approach. Boston: http://www.cfoeurope.com/displaystory.cfm/1735815
International Thompson Publishing.
[33] Kilpi, T. (2001). Implementing a Software Metrics
[20] Fiegenbaum, A. V. (1983). Total Quality Program at Nokia. IEEE Software
Management. New York, McGraw Hill. (November/December), 72-76.
[21] Gopal, A., Krishnan, M. S., Mukhopadhyay, T., & [34] Knowledge@Wharton. (2005). Why so many big
Goldenstein, D. R. (2002). Measurement Programs in IT investments do so little for shareholder value.
Software Development: Determinants of Success. IEEE Retrieved 14/03/2006, 2006, from
Transactions on Software Engineering, 28(9), 863-875. http://www.phptr.com/articles/printerfriendly.asp?p=40
2223
Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)
0-7695-2755-8/07 $20.00 © 2007 10