1.0. Introduction 1

2.0. Methodology 1

2.1. Test Template 1

2.2. Test Environment 1

3.0. Test Procedures 1

3.1. Preparation 1

3.2. Initial Link Test Procedure 2

3.3. Rework Test Procedure 2

3.4. Regression Test Procedure 2

4.0 Test Results Summary 2

4.1. Requirements Cross Walk 2

4.2. Interim Builds Summary 2

4.3. Final Build Report 3

5.0 Unresolved (Open) Issues 3

6.0 Recommendations 4

Figures

1 Requirements Cross Walk 4

Appendices

A. QATR 1.0

B. QATR 2.0

C. QATR 3.0

D. QATR 4.0

1.0. Introduction. This report summarizes the results of the test process performed on the MJY Software Risk Management Site. The report covers basic test methodology, checklists (scripts) used during test, test results, and (if applicable) any remaining unresolved (open) items at product delivery to customer with proposed corrective action plan and schedule.

2.0. Methodology. The test methodology, outlined in the Quality Assurance/Test plan, is to test web site connectivity and functionality, and to verify and validate web site contents, against SRS requirements. Quality assurance also used these tests as an opportunity to conduct an informal review MJY team-generated documents. The test program was synchronized to the time-phased delivery (build) of increased web site functionality, and included regression testing to insure continued correct operation of previously delivered capability. Results of each build test were disseminated to the program manager, developer, and (if appropriate) owner(s) of document(s) that requiring rework based on informal document review.

2.1. Test Template. A test template was constructed to ensure consistent and systematic test of all site capabilities. The template layout was patterned on the Build One website design, then updated as required over the remaining builds. Each section of the template (website design) was traced back to its required build delivery as specified in the current MJY Work Breakdown Schedule (WBS). Furthermore, each deliverable specified in the WBS was traced back to an originating requirement contained within the current version of the SRS, and that the SRS could map each of its requirements to a counterpart deliverable within the WBS. In this manner, the test team verified that each (build) deliverable contained all of the contracted functions specified in the WBS, and that the WBS correctly accounted for all requirements contained within the SRS.

When completed, the test template served as the interim test report for that build. The completed template was forwarded to the program manager and developer for their review and appropriate action. Any defects or errors noted in the template were transferred to the QA Defect Tracking Log to ensure follow up action.

2.2. Test environment. All tests were conducted using a Microsoft Internet Explorer 3.01a browser hosted on a Micron ClientPro PC platform with a Pentium 166 MHz processor using a Microsoft Windows 95 operating system. The test platform was geographically separated from the website host server/network.

3.0. Test Procedures. The following test activities were followed upon developer notification that an incremental build was available for test on the MJY website:

3.1. Preparation

3.1.1. Test team reviewed previous test logs and QA Error Tracking Logs to determine open or reworked items that would need to be retested.

3.1.2. Test team would peruse the site for modifications to site layout, and make appropriate corrections or annotations to the test template.

Using a blank test template, the team would accomplish the following activities:

3.2. Initial Link Test Procedure. Each new web page link or control button, whether it links to an internal portion of the MJY site, or to some other external web site, has been subjected to the following tests:

3.2.1. Connectivity (that is, can I connect to the indicated URL?)

3.2.2. Readability (that is, can I read the page once connection is established?);

3.2.3. Navigation at the linked site (that is, can I access or operate the various functions or capabilities at the "linked" site.)

3.2.4. Download capability. If the site page has document download capability, can I download a readable copy of the document?). Download is marked "N/A" if the site has no download capability (other than saving display as an HTML file).

3.3. Rework Test Procedure. Test team would revisit any open items from previous tests to confirm that the defect or error had been corrected by the development team. Open items would be subjected to the same paragraph 3.2. test procedures, and were counted against the regression testing requirement.

3.4. Regression Test Procedure. After performing the tests outlined in paragraph 3.2. and 3.3., a regression test, using these same procedures, was accomplished for at least ten percent of the links from previous builds. Previous test templates were referenced by the test team so that different links would be subjected to regression tests.

4.0. Test Results Summary.

4.1. Requirements Cross Walk (Figure 1). Using this procedure, test team verified that all requirements enumerated in the requirements matrix were reflected in the SRS. Two requirements, links to other SWSE 625 sites, and web site area for draft project papers, were not enumerated in the WBS, although these were delivered during Build 2. Because the WBS contains the requirements that will be satisfied by each phased deliverable, test template references test of each web site section to its corresponding WBS number, with the two exceptions noted above.

4.2. Interim Builds summary. Test results for each phased software deliverable are located at Appendices A through D. A total of seven errors were uncovered by the test team. With two exceptions, all of the errors occurred at the external site to which our site was linking. Examples include failure of the external site to access downloadable information, links at the external site which had been internally renamed and thus no longer available, or tertiary links (when an external site linked to another external site) which were no longer valid. These errors were not controllable by our development team, and while noted for possible action after final build testing, did not delay the development team from moving into the next build. Two errors involving browser capability was discovered, and will be addressed further in the final build report.

Defects detected during test were of a QA variety, with the predominant proportion involving consistency of version number and date between what was shown on the document cover sheet and what was contained in the document site index. Disposition of errors and defects were documented on the QA Error Tracking Log.

4.3. Final Build report. The final build phase test also served as the customer acceptance test. Paragraph 3.0. build phase test procedures were accomplished for the new bibliography, "Lessons Learned," and "unique materials" (term papers) web sections, as well as the software project management site sectional layout and the new navigation buttons at the top of each page of the web site. For acceptance purposes, all internal and (primary) external web site links were rechecked for correct functionality. (Secondary and tertiary linkages were spot checked; download capabilities were not checked.) In addition, another complete bulletin board functionality check was accomplished. Test results can be found at Appendix D. A summary of acceptance test results follows:

4.3.1. All external links functioned properly, allowing test team to connect and correctly navigate within external sites.

4.3.2. All primary internal site links functioned properly, allowing test team to connect and correctly navigate between site pages.

4.3.3. A previously discovered Build 3 error, concerning the ability to link to an internal web subpage, continued to manifest itself during acceptance test. The problem: when navigating to a subsection of the page, while the correct subsection URL is displayed, the browser takes you to the top of the selected page. The reader can scroll down to the correct section, and all other functionality is not impacted (ie, can download from the section, can navigate to other links, etc). Further testing showed that the problem occurs only with the test browser (MS Internet Explorer V3.01a), and that the problem is intermittent (no particular preconditions have been identified to cause either consistent successful or unsuccessful linkage to the targeted section). (This error was confirmed by a test using the same browser version on a different test platform, thus ruling out possible platform or platform configuration induced errors). This error does not occur with those Netscape browser versions available to the team. Other browsers are not available to determine their sublink navigation behavior.

4.3.4. Bulletin board functionality was satisfactorily demonstrated by posting an initial test message, followed by two nested responses. In each case, the reader could access and read the selected message, and post a response to same; the bulletin board then correctly posted the message or response.

5.0. Unresolved (Open) Items:

5.1. Sublink navigation. Further testing using different browsers is needed to determine whether other browsers are also affected by this glitch. A different web authoring tool should also be tried to rule out the possibility of a tool-induced error. As an interim solution, a warning should be posted on the web site alerting readers that certain browsers may require they scroll down the page to reach the correct section.

5.2. Open document review write ups. Time permitting, recommended corrective action annotated in Quality Assurance Error Tracking Log and Document Review Reports (see QATP FINAL, Appendices A-C) should be accomplished by appropriate development team members. Otherwise, corrections should be accomplished as part of initial software maintenance activities.

6.0 Recommendations. Based on experience gained from this test process, the following recommendations are offerred:

6.1. Web site maintenance: although the internal site links will remain static, an aggressive monitoring effort will have to be made to ensure external site links remain valid. Over this relatively short test period, several tertiary links (those at external sites) were found to be invalid. The customer could experience a great deal of user dissatisfaction (as indicated by steadily decreasing user traffic at the site) should such a link monitoring program not be put in place. Market research to find a tool which provides an automatic monitoring or testing capability of web sites and web site links is recommeneded.

6.2. As an extension of recommendation 1, acquisition of automated web site test tools for the MJY development and test teams is recommended. This would facilitate detection and elimination of any internal site link faults during development as well as speed up the test process, which must now be manually accomplished for each link.