Abstract
This paper presents common software development activities and their associated risks while exploring what if any effect the role of configuration management has in controlling or reducing those risks.
Introduction
The object of this paper is to examine the role that configuration management plays in risk reduction during the software development process. To do this we must identify the activities within the software development process, and the configuration management process, as well as define the terms to be used in this evaluation. Once the terms and activities have been identified, the configuration management activities can be crossed referenced into the software development process. An evaluation can then be made as to the effectiveness of configuration management in limiting project risk during the software development process.
Defining the software development process to be used
Three software development process models widely used today are waterfall [Royce 11], spiral [Boehm 4], and evolutionary [Gilb 5]. The waterfall is a linear progression of project activities, where an input is received by an activity, processed, and the output is delivered to the next sequential activity as the input to that activity. The final output being the product for delivery to the customer. The spiral process model bends the planning, requirements, and design activities of the waterfall back around itself three times to allow these three activities to be injected with activities of evaluation, risk, verification, and planning based on the results of the previous spiral. The implementation activities of the spiral process returns to sequential activities as in the waterfall model. The evolutionary process model is where each development activity feeds output both forward as the waterfall, but also backwards to any activity as necessary to correct or improve the product being developed. This can be compared to a sort of two steps forward one step back dance. When needed this process can go all the way back to the beginning to reevaluate and rework itself.
In all three of these process models the same basic software development activities are taking place. These activities may be implemented differently, their relationship to other activities may change, and the process models themselves offer their own unique management issues [Bersoff 3], but the general software development function of these activities remains unchanged. These general software development activities, not the process models, are the focus of this evaluation as to what role configuration management plays in reducing project risk. Therefore, for the purpose of this paper the development process models will not be considered as factors but rather the basic software development activities that they all use. However, to avoid unneeded complexity in the evaluation these software development activities will be discussed within the framework of the waterfall process model.
Defining terms used in the evaluation
Risk
Before going further a definition for project risk needs to be established. Risks are those factors that may prevent the attainment of a set goal. H. Bersoff has defined the goal of a software development process as, "Producing a product that meets or closely matches the needs of the people for whom it is developed. Achieving this goal is called product integrity" [2]. He goes on to say that product integrity must also included the additional goals of meeting the planned cost, and meeting the planned schedule for producing that product. Therefore, project risk would involve anything that may compromise the attainment of project integrity. It is interesting to note that the current version of the Software Engineering Institute's (SEI) Software Capability Maturity Model (SW/CMM) does not include a risk process area, although a level 3 process area has been included in the draft SW/CMM Version 2 [Paulk 9].
Software Development
The term software development can apply to the total software development project as well as the specific project function of implementing the software product. Paul Rook in his paper on project control[9], presents a multi-dimensional model of the software development project. In his model he depicts project control activities matrixed upon project activities. Project management, configuration management, quality assurance, and others are part of project control activities. Software development constitutes the majority of the project development activities. A somewhat different model is used in the BDM software development methodology [1]. This model presents project management as the basic foundation with total oversight and support responsibility. Upon this foundation is rooted configuration management and quality assurance support and oversight. These three form the project control and oversight platform needed for software engineering activities.
For the purpose of this paper the term software development will characterize the actual implementation stages and their respective activities in the software development project.
Roles
Project Management
The primary oversight role in a software development project belongs to project management. Project management is responsible for the planning, monitoring, controlling and evaluating the status of the project. Project management is generally assisted in this regard by configuration management and the quality assurance.
Configuration Management
The SW/CMM Version 1.1 [Paulk 8] defines four goals for configuration management as part of its level 2 key process areas. Three of the four goals can be mapped into the four activities as identified by H. Bersoff. Configuration management is responsible for identifying configuration items, controlling and tracking changes to those items, providing configuration status accounting reports to the Project Manager about those items, and performing configuration audits [Bersoff 2]. Although Bersoff and others identify configuration audits as a configuration management activity, it is not identified as such in the IEEE standards [IEEE 6]. I agree with Alexa Marshall [7] that a project that has implemented configuration management as discipline reduces the need for certain verifications. In those cases configuration audits may become more the domain of quality assurance.
Quality Assurance
Quality assurance, although confused at times with configuration management, is only concerned that products meets the baseline requirements and that the prescribed processes as set out in the software development plan are followed during the software development process. In the draft version of the SEI SW/CMM Version 2 [Paulk 9], Quality assurance has separate process assurance and product assurance goals.
Configuration Management Activities
As mentioned above, configuration management is a project control activity. Software configuration management activities are designed for control and over sight into software development activities and not into other project control activities of project management and quality assurance. However, the results of configuration management activities can provide project management and quality assurance with information that can help identify developing risk related to schedule and process.
The following configuration management activities will be used for evaluating the effectiveness of configuration management activities in risk management.
Identification of configuration items
This activity is primarily dealt with during the project planning stage of a software development project. At that time items are identified for which configuration management will be responsible. Some of these items will make up the project baseline, such as requirements documents, and source code. Other item not part of the project baseline, such as project archive documents, development tools, and libraries needed to generate the product, should be identified and also come under configuration management control.
Change control and tracking
Once items have been identified for configuration management control, changes to these items and the processes involved with changing these items must be controlled and tracked. The control process is most formal with baseline items. Changes to baseline items generally are presented as Engineering Change Requests (ECR) and are not acted upon without the permission of a Configuration Control Boards approval (CCB). Non- baseline items have less formal control processes, but do require some form of authorization before changes are allowed. Tracking is the documentation part of the controlling activity. Configuration management must record all actions of the CCB as well as the revision history of each configured item. A second change control process is software problem tracking. These problems depending on when they are reported during the software development process may or may not require CCB involvement.
Configuration status accounting - version control
As a project becomes more complex and the number baselines and proposed ECR increase on a project, the activity of configuration status accounting becomes crucial. Configuration management is charged with keeping and providing the project with exact version information on all configured items for each baseline. This activity should be extended to include non-baseline configuration items as mentioned above in the configuration item section.
Software development activities
For the purpose of this evaluation a set of standard activities need to be defined. After reviewing the IEEE Standards for Developing Software Life Cycle Processes [6], Royce's waterfall process model [11], and Boehm's spiral model [4], the following software development activities presented themselves as the basic activities for any process. The names for some of these activities have been generalized for the purpose of this paper.
Identification of software development activities, risks, and mitigation factor
The next sections addresses software development activities following the standard waterfall development model. Since software development maintenance tends to be techno speak for 'etc., etc...' or in some cases 'ad nauseam', it will not be included here. To be done correctly maintenance must follow the same process as initial implementation, if perhaps on a smaller scale. Because the following information is to be applied to both initial work and ongoing maintenance work there are risks cited for failure to establish a baseline or use the correct existing baseline.
Requirements Analysis
Define functional requirements
Risk factors - misunderstanding customer's needs
Mitigation QA Review
- failure to use the correct baseline
CM Configuration status accounting
- failure to identify all requirements
QA Review
- implying that all requirements will be met
PM issue
- failure to version control draft work products
Use version control tools
- failure to baseline results
CM Configuration Item/status accounting
Define interface requirements
Risk factors - misunderstanding customer's needs
Mitigation QA Review
- failure to use the correct baseline
CM Configuration status accounting
- failure to identify all requirements
QA Review
- implying that all requirements will be met
PM Issue
- failure to version control draft work products
Use version control tools
- failure to baseline results
CM Configuration Item/status accounting
Requirements management
Risk factors - failure to record all requirements
Mitigation QA Review
- failure to use the correct baseline
CM Configuration status accounting
- failure to update requirements properly
QA Review
- failure to maintain requirements traceability matrix
QA Review
- failure to baseline modification
CM Configuration Item/status accounting
Prototyping
Risk factors - failure to maintain version control
Mitigation CM Configuration Item/status accounting/version control tool
- may mislead customer by demonstrating something that will not be possible in the production system
PM Issue
Documentation
Risk factors - failure to follow standards
Mitigation QA Review
- failure to include all requirements
QA Review
- failure to record requirement correctly
QA Review
- failure to version control draft work products
Use version control tools
- failure to use the correct document baseline
CM Configuration status accounting
- failure to baseline results
CM Configuration Item/status accounting
Design
Architecture design
Risk factors - failure to design all system requirements
Mitigation QA Review
- mistakes in design
QA Review
- failure to use the correct baseline
CM Configuration status accounting
- failure to create an implementable design
QA Review
- failure to version control draft work products
Use version control tools
- failure to baseline results
CM Configuration Item/status accounting
Data design
Risk factors - failure to completely define data dictionary
Mitigation QA Review
- failure to use the correct baseline
CM Configuration status accounting
- failure to completely define record formats
QA Review
- failure to properly define primary keys
QA Review
- mistakes in design
QA Review
- failure to version control draft work products
Use version control tools
- failure to baseline results
CM Configuration Item/status accounting
Interfaces
Risk factors - failure to design all system requirements
Mitigation QA Review
- mistakes in design
QA Review
- failure to use the correct baseline
CM Configuration status accounting
- failure to create an implementable design
QA Review
- failure to version control draft work products
Use version control tools
- failure to baseline results
CM Configuration Item/status accounting
Detail design
Risk factors - failure to design all system requirements
Mitigation QA Review
- mistakes in design
QA Review
- failure to use the correct baseline
CM Configuration status accounting
- failure to create an implementable design
QA Review
- failure to version control draft work products
Use version control tools
- failure to baseline results
CM Configuration Item/status accounting
Prototyping
Risk factors - failure to maintain version control
Mitigation CM Configuration Item/status accounting/version control tool
- may mislead customer by demonstrating something that will not be possible in the production system
PM Issue
Documentation
Risk factors - failure to follow standards
Mitigation (QA Review)
- failure to use the correct baseline
CM Configuration status accounting
- failure to include all aspects of design
QA Review
- failure to record design correctly
QA Review
- failure to version control draft work products
Use version control tools
- failure to baseline results
CM Configuration Item/status accounting
Implementation
Create source code
Risk factors - failure to implement all requirements
Mitigation QA Review
- failure to use the correct document baseline
CM Configuration status accounting
- failure to use the correct source code baseline
CM Configuration status accounting
- failure to implement requirement correctly
QA Review
- unable to implement design
PM Issue
- failure to version control draft work products
Use version control tools
- failure to baseline results
CM Configuration Item/status accounting
Reuse of available source code
Risk factors - failure to use the right baseline
Mitigation CM Configuration status accounting
- lack of documentation
CM Configuration Item
- unanticipated side effects
SWD Issue
Use of COTS
Risk factors - incompatible versions
Mitigation SWD Issue
- lack of documentation
SWD Issue
- effects of future upgrades
SWD Issue
- failure to use correct version
CM Configuration status accounting
Unit testing
Risk factors - failure to test the correct software versions
Mitigation CM Configuration status accounting
- failure to test the correct system configuration
CM Configuration status accounting
- failure to test the correct requirement baseline
CM Configuration status accounting
QA Review
- failure to test all requirements
QA Review
- failure to test requirement correctly
QA Review
Documentation
Risk factors - failure to follow standards
Mitigation QA Review
- failure to use the correct baseline
CM Configuration status accounting
QA Review
- failure to document all implementation comments
QA Review
- failure to version control draft work products
Use version control tools
- failure to baseline results
CM Configuration Item/status accounting
Integration
Creation of test plans
Risk factors - failure to use the correct test plan baseline
Mitigation CM Configuration item/status accounting
- failure to use the correct requirements baseline
CM Configuration status accounting
QA Review
- failure to use the correct design baselines
CM Configuration status accounting
- failure to version control draft work products
Use version control tools
- failure to baseline results
CM Configuration Item/status accounting
- failure to plan for all requirements
QA Review
- failure to correctly plan requirement test
QA Review
Integration testing
Risk factors - failure to test the correct software versions
Mitigation CM Configuration status accounting/audit
- failure to test the correct system configuration
CM Configuration status accounting/audit
- failure to use the correct test plan baseline
CM Configuration Item/status accounting
- failure to test all requirements
QA Review
- failure to test requirement correctly
QA Review
- failure to track test results
QA Audit
Problem reporting
Risk factors - failure to track problems
Mitigation CM Control & Tracking
- failure to track status of problems
CM Control & Tracking
- failure to track fixes to software revision
CM Control & Tracking
- failure to report status to management
CM Configuration status accounting
Problem resolution testing
Risk factors - failure to test the correct software revisions
Mitigation CM Configuration status accounting
- failure to test the correct system configuration
CM Configuration status accounting
- failure to report status to management
CM Configuration status accounting
- failure to track test results of fixes
QA Audit
Documentation
Risk factors - failure to follow standards
Mitigation QA Review
- failure to accurately record test results
QA Review
- failure to version control draft work products
Use version control tools
- failure to baseline results
CM Configuration Item/status accounting
Delivery
Building software
Risk factors - failure to use the tested baselines
Mitigation CM Configuration status accounting/audit
- failure to use the correct tools or version of tools/libraries to build software release
CM Configuration status accounting/audit
- failure to include all the required files
CM Configuration status accounting/audit
Providing documentation
Risk factors - failure to use the correct document baseline
Mitigation CM Configuration status accounting/audit
QA Review
- failure to include all the required documents
CM Configuration status accounting/audit
QA Review
Evaluating effects of configuration management activities in limiting risk
In the previous section software development activities and associated risk factors where presented along with some mitigating activities. The list of risk factors is not meant to be exhaustive, nor is it based on any published authority. They are taken from problems I have experienced or heard of from follow software engineers.
What can be seen from the breakout of configuration management activities versus quality assurance activities, is that in many places they both tend to work in parallel. Configuration management providing the correct tools and work products for the activity, while quality assurance verifies and validates that the work done meets requirements and has followed the correct processes before the work product is rebaselined. There are slight shifts in this oversight balance throughout the software development process. Quality assurance appears to be the more dominant oversight in the initial activities of analysis and design. During implementation, configuration management takes a more lead role, but during integration testing, they return to shared oversight. Finally, configuration management is just slightly more involved in the delivery activity. This would support the BDM Methodology model of parallel support by configuration management and quality assurance for the software engineering activities [1].
Most of the identified risks that can be mitigated by adherence to configuration management principles can be grouped and reduced to a small set of generalized risk factors:
In addition there are the specific risk factors:
These risk factors map directly to configuration management activities.
Identification of configuration items is a SW/CMM level 2 goal that mitigates risk by identifying all important project work products needed for delivery or future modification. Without a list of identified controlled items, work products can be lost and would have to be totally recreated, if at all possible, when needed. The majority of the work for this activity occurs during project planning, but like most project artifacts, it needs to be periodically updated to reflect changes.
Responsibility for mitigating the failure to baseline work falls to both the identification activity, is it missing from the list of configured items, and the configuration status accounting activity, the current version does not match the project plan.
Another aspect of this activity is also identification of configuration management tools to be used on the project. These tools include version control tools to be used with draft work products, and system development tools that will be used for generating work products for final delivery to the customer. Tool identification is a risk mitigation factor for the software development team to be used during work on draft work products that are prone to the same errors and rework issues in much the same manner as the configuration management controlled baseline work products. Configuration management usually does not take responsibility for version control of draft work products.
Configuration status accounting is the primary version control activity for configuration management to avoid errors and rework due to using the wrong work product version when planning, designing, modifying and decision making. This is a common error that I have seen occur on projects. The use of an out of date baseline can usually be caught through QA reviews and testing, but this again involves unplanned rework. I have also witnessed a case where, in a multiple customer environment, a customer received a maintenance release that included modules containing other customers variations, not their own, and more recently a case where the current modification where applied to a version that was three releases out of date.
Configuration status accounting is a SW/CCM level 2 goal. The SW/CCM goal however, combines the activities of status accounting with change control and version tracking.
Control and tracking is regular baseline change control activity involving ECRs, and normally take place prior to software development in the project planning, or release planning activities of the project. In this evaluation control and tracking are represented by means of the control and tracking subactivity of problem reporting during the integration testing. In the problem tracking process, change control focuses on controlling the configuration of the current work products during testing. The configuration management risk mitigation role that is played during problem tracking has two aspects. The first is to prevent the inclusion of new work product version that have not been successfully test into the baseline, and second is to inform project management as to the status of problems being tracked as a result of testing. In the more quality conscience projects, problems/defects are also tracked as a result of quality assurance work product reviews. These status reports can be clear indicator to project management of possible schedule, process, quality, and resource issues that could adversely effect the project integrity, risk.
Configuration audit is the risk mitigation activity that verifies that all 't's are crossed and 'i's dotted by way of verifying that all baseline configurations are what configuration management believes them to be. This activity should been done periodically when supporting a production system, but especially when preparing a work product for customer delivery. The risk mitigation is obvious here. Without this activity it is possible that a configuration mismatch could be introduced into the product. The effect of such an occurrence can leave a project open to a myriad of possible errors and problems.
Summary
Certain software development risk factors can be effectively mitigated by configuration management. This is not really a surprising conclusion since configuration management evolved to address certain software integrity issues, i.e., risks.
Failure to mitigate the risk factors listed above leaves the software development project open to issues of credibility and loss of customer confidence, as well as additional unplanned costs in time and resources to preform rework.
Risk factors are often given the aura of potentially project shattering events and as a result the big ticket items usually get the primary focus. I fell that there are day to day risks that tend to be over looked or ignored because it is assumed that everyone knows the right thing to do. Oversight of some of these day by day activities falls to configuration management, an area that is also often under rated or under appreciated. However, the existence of configuration management on a project is not enough. Without proper implementation and project acceptance, the projects risks erosion of its credibility, and project management loses a valuable insight into the project. Day by day things slip through the cracks and get lost or forgotten. Customer confidence begins to fade. Small mistakes become embarrassments. Hopefully large mistakes won't happen.
Configuration management, when done well, is a process area that is over looked. When configuration management personnel leave a project without qualified replacements it can become painfully clear of the day to day importance a well implemented configuration management program is in maintaining basic project integrity.
"In the same way that quality cannot be inspected into a product, software configuration management cannot be audited into a product" [Marshall 7]. Configuration management has been around for year on large scale and now smaller scale systems development projects. The sheer presence does not guarantee mitigation of day to day risk. It requires top management support and buy in across the project.