By Darrin May

SWSE 625

April 21, 1997

Standards. Quality. Improvement. Measurement. Words such as these can be found in the themes of most of today’s software engineering literature. Indeed, there is much interest in solving the "software crisis" by encouraging that minimal standards be met, that quality software and continuous improvement be more commonplace, and that a means of measuring and evaluating success be used. Software engineering is not alone in this respect. With the globalization of the marketplace, increased opportunity and competition have driven the need to produce better products and services in every field of endeavor. The need has been felt, but where do we receive direction on how to proceed? What are the steps we should take and how do we know if we’re succeeding? To answer these questions in software and other disciplines, a series of quality management standards called ISO 9000 was created by the International Organization for Standardization. Specifically, ISO 9000-3 is the guideline for the application of the standards to the development, supply, and maintenance of software. In this paper, I will provide a brief overview of the standard and its organization, the certification process, and a discussion of the potential compliance risks.

The ISO 9000 family of standards was created to ensure quality across a wide range of activities, from manufacturing to servicing. Because of the special nature of software development, with its emphasis on research and design, a separate guideline was needed. ISO 9000-3, like the rest of the 9000 standards, is based on the premise that a "high quality process results in the production of high quality goods and services" [1]. Such an emphasis means that organizations receive ISO 9000-3 certification, not the products they produce. This emphasis on process rather than product is reflected in the description of the principles underlying the ISO 9000 quality standards used by Bailetti and Fitzgibbon [2]:

Similar to these principles, Shmauch [3] describes an ISO 9000 quality system as having the following attributes:

2. Commitment (the QMS is a top priority at all levels)

With this in mind, the organization of the 9000-3 standard is readily understandable. Its six parts include: 1) Scope , 2) Normative References, 3) Definitions, 4) Quality System-Framework, 5) Quality System-Life-Cycle Activities, and 6) Quality System-Supporting Activities. The last three sections dealing with the framework and activities of a quality system comprise the bulk of the guideline's substance and will be the focus of this paper.

The framework of a quality system described in ISO 9000-3 details the responsibilities of the supplier and purchaser management, the attributes of the engineering environment, and the existence of internal audits and corrective action. Specifically, the supplier management should

In addition, the purchaser management should

The Quality System itself should have the following characteristics:

The Quality System-Life Cycle Activities section of the standard gives a description of the various phases normally encountered in the development of software. Specifically, the standard mentions Development Planning (5.4), Quality Planning (5.5), Design and Implementation (5.6), Testing and Validation (5.7), Acceptance Testing (5.8), Replication, Delivery and Installation (5.9), and Maintenance (5.10).

A Development plan that conforms to the guideline will have the following elements: Definition of the Project (objectives), Organization of the project (includes team structure, responsibilities, and subcontractors), Development Phases (description of the phases, defined inputs and outputs, verification procedures, and risk assessment), Project Management (how measuring and monitoring will occur, organizational responsibilities, and interfaces between groups), Development Methods and Tools, Project Schedules (tasks to be performed, when, and by whom), Related Plans (the Quality Plan that will be used for example).

A 9000-3 Quality Plan will itemize the various quality activities that will be used for a particular project. It will specify objectives, inputs and outputs, the types of tests that will be performed, who will be executing the various tests, and how defects will be controlled and corrected.

The Design and Implementation phases in 9000-3 are described in general terms so as not to restrict or favor one method over another; yet, the guideline does mention some common characteristics such as the identification of design considerations, a design methodology, use of past experiences, allowance for programming rules and languages, and defined implementation methodologies. The guideline does say that the design phase should be carried out in a disciplined manner.

The Testing and validation activities under 9000-3 should be done so as to ensure that the results are recorded in a meaningful way, that areas requiring retesting are identified, and that the adequacy and relevance of the tests are periodically evaluated. Coordination between supplier and purchaser for acceptance and field-testing is also an important part of the plan before the delivery and installation phases.

Under the guideline, activities in the Maintenance Phase should be stipulated completely in the contract. The Maintenance Plan should include the scope of the maintenance, a description of the initial status of the product on delivery, a list of maintenance activities and supporting records and reports.

The activities classified as "Supporting" include Configuration Management (6.1), Document Control (6.2), Quality Records (6.3), Measurement (6.4), Rules, Practices, and Conventions (6.5), Tools and Techniques (6.6), Purchasing (6.7), Included Software Products (i.e. third party vendors, 6.8), and Training (6.9).

In order to identify, control, and track the different versions of each software item, 9000-3 calls for a configuration management plan that identifies the organizations and groups responsible as well as the activities they will perform and tools they will use. Status reports and procedures should also be included to evaluate the effectiveness of the configuration team's efforts.

In order to be certified under ISO 9000, software organizations must choose the scope of the business activity to be registered, select a registrar, undergo the audit, and after successful completion, pass annual surveillance visits thereafter to maintain certification. In the event that the registrar/auditor finds areas of non-compliance, the organization is given a time limit for corrective action.

In the past decade much thought has been devoted to the application of risk management techniques in the field of software development. Such work has focused on the identification, analysis, and mitigation of the dangers that prevent projects from fulfilling requirements "on time and under budget." At first glance, the idea that there are risks involved in complying with ISO 9000-3, a guideline devoted to increasing quality, seems contradictory. Yet, by closely examining the standard and its potential interpretation by suppliers, purchasers, and compliance auditors, the dangers become apparent. Such risks fall into three basic categories: those dealing with the specification itself, those arising from its implementation by individuals putting conformance ahead of effectiveness, and those found in the audit process.

Section 5.3 of the specification deals with the Purchaser's Requirements Specification. The guideline states that part of purchaser's management responsibility is to make sure that "the requirements for the product are explicitly, accurately, an completely stated." [4]. Certainly this is a standard that all software developers would like to see met; yet, in reality it remains an unattainable goal. Waiting until the specifications are complete and unchanging would force many projects to languish at this stage of development. A more realistic approach would be, in the words of one author, " to call for software specifications that provide for evolution even as the product is being developed, if the customer understands and takes responsibility for a realistic incremental cost and schedule impact of the change." [5].

In addition to the substance of the requirements, ISO 9000-3 calls for the use of "terminology that is agreed by both parties" to describe the requirements. Working with a purchaser that is unfamiliar with formal notation and specification methods is the norm rather than the exception. While using common terminology facilitates understanding between purchaser and supplier in most cases, being limited to natural language could hinder the completeness and accuracy of the requirements.

Another item associated with the Requirements section that could lead to problems is that there is no mention of a detailed requirements phase. Depending solely on a customer's high level requirements could lead to ambiguity and errors in later life-cycle activities.

In section 4.1.2 it states that purchaser's management should identify one representative responsible for supplier interface. While the intent here is to minimize potential miscommunications and misunderstandings between the purchaser and supplier, such a requirement, if strictly followed, could lead to an over dependence on one individual: a single point of failure.

Also section 4.1.1.2.3, Supplier's Management Representative, states that a management representative "shall have defined authority and responsibility for ensuring that the requirements are implemented." This requirement implies that such an individual has authority to give direction to "line" personnel even though s/he is a "staff" manager. Such a situation could lead to disagreements with the "line" managers whose authority and direction this representative would be overriding.

Another danger in the guideline is that the one-size-and-type-fits-all quality system is encouraged. This runs counter to the evidence that projects of differing size, complexities, and criticalities should use the same measures and procedures for ensuring quality. One aspect of this is the need for formal document control as called for in section 6.2.1. Such an approach aids the development of large projects; however, meeting such rigorous documentation standards could choke smaller ones.

The ISO 9000 standards present a special risk for organizations that develop both hardware and software: the 9000-3's interpretation of the design and implementation phases varies from the 9001 standard. This means that two quality management systems need to be set up, thus hindering unity and consistency.

In addition, even though the standard "is intended for application irrespective of the life-cycle model used", the description of the life cycle activities presented is oriented towards a classical waterfall approach. Section 5.4.2.1 states that between the development phases the required inputs and outputs must be defined. If a project calls for the use of rapid prototyping where the outputs are not entirely known at the outset, then the organization runs the risk of either not fulfilling the specification or missing the benefits of an alternative life cycle.

Another less obvious but no less dangerous risk in the guideline lies in the implication that "quality is everyone's business." This sounds like a nice idea, and, to some extent has merit. However, various recommendations and compliance strategies I have read contradict one another on this point. One has suggested that by making everyone responsible for QA, the need for a separate checking procedure is eliminated. Another source disagrees, stating that without independent verification, judgment could be impaired and conflicts over who is responsible for defect discovery will arise. I myself have worked in a large service organization that used the motto "It is my business" when solving a customer's problem, in order to prevent line staff from "passing the buck". A good deal of ill will and bad feeling resulted when enthusiastic workers would over-step job or departmental boundaries on their "total quality" crusade.

Perhaps the aspect of the standard that towers above all others is that it is pass/fail. Yes, quality is often measured by whether the customer is satisfied or not; however, this lack of middle ground puts great pressure on informally managed organizations to quickly rise to a high level. The author of this paper wonders whether a more graduated scale, such as is found in the Capability Maturity Model, might encourage greater participation and more thoughtful compliance efforts. The fact that 9000-3 is wedded to a family of standards covering a wide range of activities seems to discourage this. A manufacturer of airplane parts, for example, would have a tough time selling merchandise that was mostly of high quality.

As Oskarsson points out: "An intelligent application of the requirements in ISO 9001 will make a supplier better."[6] An unintelligent application, on the other hand, could cause disruption and loss in quality. The following is a discussion of the potential hazards that a compliance- seeking organization may encounter.

One of the first hazards stems from the differing amount of support for compliance within a company. Sometimes individuals other than top management first recognize the need for compliance. In such a case, these "early adopters" have the responsibility of fully convincing senior executives that it is a goal worth pursuing. If less than whole-hearted support is given, problems arising later on will not be overcome due to a lack of resources and will.

Another hazard stems from an organization choosing the wrong people to design and direct the implementation efforts. Trouble could arise, for example, when an organization selects outside consultants or untrained personnel from within to prepare specification templates, programming rules, and reviewing procedures. Their lack of knowledge of the existing, "informal" procedures and associated rationale could lead to dissension with engineers and programmers who will be asked to follow the "new" rules. Indeed, those responsible for following and documenting the quality rules and procedures must "buy in" on it. This is not to say, however, that implementation should be placed solely on the shoulders of the line staff. If this were to occur, line personnel would end up duplicating efforts and reinventing the "quality system" wheel for each project. Instead, those individuals with enough authority, such as line managers, should be given this responsibility.

Not only must the right personnel lead the effort; their approach must have structure. Without timetables, budgets, milestones, and progress reports, the effort could stray off course, wasting financial and internal "political" capital.

Another personnel risk associated with compliance is that managers at different levels may not feel that the quality system is important and right. To avoid confusion, wasted efforts, and potential ill will, all individuals responsible for setting the priorities of an organization must believe in it.

Getting everyone behind the effort could lead to another danger: old, less formal procedures that have worked in the past risk being thrown out in favor of reinventing everything in light of the standard. Such a situation could lead to disruption in the operation of the company and thus decrease the level of quality. To avoid this, companies should document existing procedures and polish them where required. Creation of new procedures should generally occur only when there is a genuine need.

Another trap that organizations may fall into is attempting an abrupt switchover to compliance with 9000-3. This could mean the rework of existing projects, leading to possible schedule over-runs. Also, such a "rushed" approach may mean asking people to implement a quality plan without giving them time to adjust to the new system. A system that is thrust upon them rather than one that evolves and empowers is likely to fail. Indeed, a step-by-step phase-in of the quality system should be favored.

In addition to an overly aggressive compliance schedule, there are hazards that could arise from lack of training in the standard itself. An organization-wide "compliance steering committee" without proper training could run the risk of losing support from all levels. Along with the "designers" of an organization's efforts, line personnel who are unaware of their roles and responsibilities risk taking misguided steps in implementation. Such steps range from not taking the effort seriously to putting too much emphasis in certain areas and ignoring others. Also, patience is required on management's part to allow people to move from seeing the QMS as an interference activity to an improving activity that is worth the effort.

Assuming proper training is supplied at all levels, another danger that may present itself is that the QMS could overly dictate direction and thus stifle the creativity needed for innovative solutions. One of the guideline's major strengths is its emphasis on consistency and formality; yet, conformance should not be overly restrictive. An example of taking compliance too literally would be the interpretation of the description of the design phase. The guideline includes it in activities that "should be carried out in a disciplined manner." While such discipline will help to verify that the design is fulfilling the requirements, too much structure could lead to a minimum level of quality rather than exceeding it. This could be true in other phases of the life cycle as well.

In addition to the supplier organization, the purchaser faces risks related to implementation of the standard as well. A purchaser without knowledge of a registrar’s qualifications and area of expertise runs the risk of relying on an unqualified, or at the very least, inappropriate opinion of a supplier. Also, just because a supplier has an ISO certification does not automatically mean that they will deliver the project as requested by the contract. Purchasers who rely too heavily on the ISO rating and do not verify the quality of the product throughout the development process could be in for a rude surprise at the end.

A significant risk in being audited is the prospect of failure. The dangers of failing include the discouragement of personnel and the loss of momentum and support. While failure is always a possibility, companies that perform a pre-audit assessment are much more likely to succeed and less likely to put undo pressure on the implementers. Such pressure puts strain on individuals and processes that are essential to the compliance effort.

One of the major risks involved in the audit process is the danger of selecting an auditor/registrar who lacks sufficient experience in software development, in both general and highly specialized project types. A lack of understanding may lead the auditor to require procedures that are detrimental to the success of the organization.

In addition to experience, an auditor must be chosen that values the spirit of ISO 9000-3 standards as much or more than literal conformance. There should not be any throwing of the baby out with the bath water on the part of either the supplier organization or the auditor.

Yet another audit risk is the possibility of choosing a registrar that lacks credibility in the market that a company is seeking to do business in. The new business that compliance status could bring is one of the main attractions of the standard. This danger should not be taken lightly.

Finally, for smaller companies, the direct and indirect costs associated with registration could be an enormous strain on resources. A 1995 estimate for a small business mentions the following costs[7]:

The amount of time and effort spent in complying with the standard and undergoing an audit may take away more away than the resulting quality system's gain in efficiency and productivity provides.

After reviewing the numerous pitfalls surrounding compliance with ISO 9000-3, some may rightly ask whether certification is worth the effort. Proponents of the standard point to benefits such as raising productivity, reducing costs, providing consistent quality, improving competitiveness and corporate image, and perhaps valued most of all: increasing recognition and approval among customers. As with most choices in business, deciding if compliance is right for an organization should be taken from a cost-benefit perspective. One of the keys to such analysis is being aware of all of the hidden costs and risks. Such risks lie within the specification itself as well as its implementation by individuals lacking experience and/or valuing literal conformance over the true intention of the standard: increased effectiveness of an organization to produce quality software. The answer to the compliance question remains to be answered by individual organizations and preferably by informed management.

[1] A. Bailetti and C. FitzGibbon, ISO 9001 Registration for Small and Medium-Sized Software Enterprises, p. 10, Carleton University Press, 1995.

[2] A. Bailetti and C. FitzGibbon, ISO 9001 Registration for Small and Medium-Sized Software Enterprises, p. 11, Carleton University Press, 1995.

[3] C. Shmauch, ISO 9000 for Software Developers, p. 13, ASQC Quality Press, 1994.

[4] R. Kehoe and A. Jarvis, ISO 9000-3: A Tool for Software Product and Process Improvement, p. 45. Springer-Verlag, 1996

[5] O. Oskarsson and R. Glass, An ISO 9000 Approach to Building Quality Software, p. 126, Prentice Hall, 1996.

[6] O. Oskarsson and R. Glass, An ISO 9000 Approach to Building Quality Software, p. 85, Prentice Hall, 1996.

[7] A. Bailetti and C. FitzGibbon, ISO 9001 Registration for Small and Medium-Sized Software Enterprises, p. 35, Carleton University Press, 1995.