samedi 9 août 2008

Certification Process Checklists

Pass

Fail

Checklist for Testing

Are test plans and procedures created and reviewed?

Are test results recorded?

Are defects or problems recorded, assigned, and tracked to closure?

Is there an adequate test process to ensure that areas impacted by changes are retested?

Pass

Fail

Checklist for Measurement

Is the software validated (tested) as a complete system in an environment as similar as possible to the final operating environment? Is this done prior to delivery and acceptance?

If field-testing is required, are the responsibilities of the supplier and the purchaser defined? Is the user environment restored following the test?

Are product metrics collected and used to manage the testing effort?

Are product defects measured and reported?

Is corrective action taken if metric levels exceed established target levels?

Are improvement goals established in terms of metrics

Are process metrics collected to measure the effectiveness of the testing process in terms of schedule and in terms of fault prevention and detection?

Pass

Fail

Checklist for Tools / Techniques

Are tools and techniques used to help make testing and management processes more effective?

Are the used tools and techniques reviewed, as required, and improved upon?

Pass

Fail

Checklist for Training

Are training needs identified according to a procedure?

Is training conducted for all personnel performing work related to quality?

Are personnel who are performing specific tasks qualified on the basis of appropriate education, training, and/or experience?

Are records kept of personnel training and experience?

Pass

Fail

Checklist for Documentation

Are test plans, requirements, and other documents revision controlled?

Do procedures exist to control document approval and issue?

Are changes to controlled documents reviewed and approved?

Are current versions of test documents identifiable by a master list or document control procedures?

Pass

Fail

Checklist for Configuration Management

Is there a Configuration Management (CM) system that identifies and tracks versions of the software under test, software components, build status, and changes? Does the system control simultaneous updates?

Does the configuration management plan include a list of responsibilities, CM activities, CM tools and techniques, and timing of when items are brought under CM control?

Is there a mechanism and procedure that enables software, hardware, and files to be uniquely identified throughout the entire software development lifecycle?

Is there a documented mechanism to identify, record, review, and authorize changes to software items under configuration management? Is this process always followed?

Are affected personnel notified of software changes?

Is the status of software items and change requests reported?

Defect Removal Efficiency (DRE)


A more powerful metric for test effectiveness (and the one that we recommend) can be created using both of the defect metrics discussed above: defects found during testing and defects found during production. What we really want to know is, "How many bugs did we find out of the set of bugs that we could have found?" This measure is called Defect Removal Efficiency (DRE).
The later we discover a bug, the greater harm it does and the more it costs to fix.

Incident Description

The author of the incident report should include enough information so that the readers of the report will be able to understand and replicate the incident. Sometimes, the test case reference alone will be sufficient, but in other instances, information about the setup, environment, and other variables is useful.

Section Heading Description
4.1 Inputs: Describes the inputs actually used (e.g., files, keystrokes, etc.).
4.2 Expected Results: This comes from the test case that was running when the incident was discovered.
4.3 Actual Results: Actual results are recorded here.
4.4 Anomalies: How the actual results differ from the expected results. Also record other data (if it appears to be significant) such as unusually light or heavy volume on the system, it's the last day of the month, etc.
4.5 Date and Time: The date and time of the occurrence of the incident.
4.6 Procedure Step: The step in which the incident occurred. This is particularly important if you use long, complex test procedures.
4.7 Environment: The environment that was used (e.g., system test environment or acceptance test environment, customer 'A' test environment, beta site, etc.)
4.8 Attempts to Repeat: How many attempts were made to repeat the test?
4.9 Testers: Who ran the test?
4.10 Observers: Who else has knowledge of the situation?

Example of Minor, Major, and Critical Defects
Minor: Misspelled word on the screen.
Major: System degraded, but a workaround is available.
Critical: System crashes.

Repetitive Tasks


Repetitive tasks, such as regression tests, are prime candidates for automation because they're typically executed many times. Smoke, load, and performance tests are other examples of repetitive tasks that are suitable for automation.
If the application being tested is unstable or changing rapidly, automating the test scripts may be difficult.
Regression tests are tests that are run after changes (corrections and editions) are made to the software to ensure that the rest of the system still works correctly.
Timing is everything. Trying to implement a major tool or automation effort in the midst of the biggest software release of all time is not a good strategy.
A test set is a group of test cases that cover a feature or system.
As a rule of thumb, we normally recommend that the regression test set (or at least the smoke test) be run in its entirety early on to flag areas that are obviously problematic.
No matter how good you and your colleagues are at designing test cases, you'll always think of new test cases to write when you begin test execution.
Obviously, the results of each test case must be recorded. If the testing is automated, the tool will record both the input and the results. If the tests are manual, the results can be recorded right on the test case document. In some instances, it may be adequate to merely indicate whether the test case passed or failed. Failed test cases will also result in an incident report being generated. Often, it may be useful to capture screens, copies of reports, or some other output stream.

Black-Box vs. White-Box


Black-box testing or behavioral testing is testing based upon the requirements and, just as the name implies, the system is treated as a "black box." That is, the internal workings of the system are unknown. In black-box testing the system is given a stimulus (input) and if the result (output) is what was expected, then the test passes. No consideration is given to how the process was completed.
In white-box testing, an input must still produce the correct result in order to pass, but now we're also concerned with whether or not the process worked correctly. White-box testing is also called structural testing because it's based upon the object's structure.
The creation and execution of tests is best be done by the people who understand the environment associated with that level of test.
Creating automated test scripts can often take more expertise and time than creating manual tests. Some test groups use the strategy of creating all tests manually, and then automating the ones that will be repeated many times. In some organizations, this automation may even be done by an entirely separate group. If you're working in an environment where it takes longer to write an automated script than a manual one, you should determine how much time is saved in the execution of the automated scripts. Then, you can use this estimate to predict how many times each script will have to be executed to make it worthwhile to automate. This rule of thumb will help you decide which scripts to automate. Unless there is very little cost in automating the script (perhaps using capture-replay, but don't forget the learning curve), it's almost always more efficient to execute the test manually if it's intended to be run only once.

Levels of Test Planning




Levels of Test Planning

Test planning SHOULD be separated from test design.

In addition to the Master Test Plan, it is often necessary to create detailed or level-specific test plans. On a larger or more complex project, it's often worthwhile to create an Acceptance Test Plan, System Test Plan, Integration Test Plan, Unit Test Plan, and other test plans, depending on the scope of your project. Smaller projects, that is, projects with smaller scope, number of participants, and organizations, may find that they only need one test plan, which will cover all levels of test. Deciding the number and scope of test plans required should be one of the first strategy decisions made in test planning. As the complexity of a testing activity increases, the criticality of having a good Master Test Plan increases exponentially

If your test plan is too long, it may be necessary to create a number of plans of reduced scope built around subsystems or functionality.

It's important that an organization have a template for its test plans. If a template doesn't meet your particular requirements, you should feel free to customize it as necessary.

What you test is more important than how much you test.

Regression testing is retesting previously tested features to ensure that a change or bug fix has not introduced new problems.

Confirmation testing is rerunning tests that revealed a bug to ensure that the bug was fully and actually fixed.

To be most effective, test planning must start at the beginning and proceed in parallel with software development. General project information is used to develop the master test plan, while more specific software information is used to develop the detailed test plans. This approach will target testing to the most effective areas, while supporting and enhancing your development process. When fully implemented, test planning will provide a mechanism to identify improvements in all aspects of the system and development process.

Preventive Testing

Preventive testing uses the philosophy that testing can actually improve the quality of the software being tested if it occurs early enough in the lifecycle. Specifically, preventive testing requires the creation of test cases to validate the requirements before the code is written

Testware is any document or product created as part of the testing effort (e.g., test cases, test plans, etc.). Testware is to testing what software is to development.

An added benefit of creating the test cases before the code is that the test cases themselves help document the software.

Software Test Documentation Template for Test Documents


1.

Test Plan

Used for the master test plan and level-specific test plans.

2.

Test Design Specification

Used at each test level to specify the test set architecture and coverage traces.

3.

Test Case Specification

Used as needed to describe test cases or automated scripts.

4.

Test Procedure Specification

Used to specify the steps for executing a set of test cases.

5.

Test Log

Used as needed to record the execution of test procedures.

6.

Test Incident Report

Used to describe anomalies that occur during testing or in production. These anomalies may be in the requirements, design, code, documentation, or the test cases themselves. Incidents may later be classified as defects or enhancements.

7.

Test Summary Report

Used to report completion of testing at a level or a major test objective within a level.

Role

Description of Responsibilities

Manager

Communicate, plan, and coordinate.

Analyst

Plan, inventory, design, and evaluate.

Technician

Implement, execute, and check.

Reviewer

Examine and evaluate.

Test planning is one of the keys to successful software testing, yet it's frequently omitted due to time constraints, lack of training, or cultural bias.

Test planning CAN'T be separated from project planning.

All important test planning issues are also important project planning issues.