Black-Box vs. White-Box

Black-box testing or behavioral testing is testing based upon the requirements and, just as the name implies, the system is treated as a "black box." That is, the internal workings of the system are unknown. In black-box testing the system is given a stimulus (input) and if the result (output) is what was expected, then the test passes. No consideration is given to how the process was completed.
In white-box testing, an input must still produce the correct result in order to pass, but now we're also concerned with whether or not the process worked correctly. White-box testing is also called structural testing because it's based upon the object's structure.
The creation and execution of tests is best be done by the people who understand the environment associated with that level of test.
Creating automated test scripts can often take more expertise and time than creating manual tests. Some test groups use the strategy of creating all tests manually, and then automating the ones that will be repeated many times. In some organizations, this automation may even be done by an entirely separate group. If you're working in an environment where it takes longer to write an automated script than a manual one, you should determine how much time is saved in the execution of the automated scripts. Then, you can use this estimate to predict how many times each script will have to be executed to make it worthwhile to automate. This rule of thumb will help you decide which scripts to automate. Unless there is very little cost in automating the script (perhaps using capture-replay, but don't forget the learning curve), it's almost always more efficient to execute the test manually if it's intended to be run only once.

Levels of Test Planning

Levels of Test Planning

Test planning SHOULD be separated from test design.

In addition to the Master Test Plan, it is often necessary to create detailed or level-specific test plans. On a larger or more complex project, it's often worthwhile to create an Acceptance Test Plan, System Test Plan, Integration Test Plan, Unit Test Plan, and other test plans, depending on the scope of your project. Smaller projects, that is, projects with smaller scope, number of participants, and organizations, may find that they only need one test plan, which will cover all levels of test. Deciding the number and scope of test plans required should be one of the first strategy decisions made in test planning. As the complexity of a testing activity increases, the criticality of having a good Master Test Plan increases exponentially

If your test plan is too long, it may be necessary to create a number of plans of reduced scope built around subsystems or functionality.

It's important that an organization have a template for its test plans. If a template doesn't meet your particular requirements, you should feel free to customize it as necessary.

What you test is more important than how much you test.

Regression testing is retesting previously tested features to ensure that a change or bug fix has not introduced new problems.

Confirmation testing is rerunning tests that revealed a bug to ensure that the bug was fully and actually fixed.

To be most effective, test planning must start at the beginning and proceed in parallel with software development. General project information is used to develop the master test plan, while more specific software information is used to develop the detailed test plans. This approach will target testing to the most effective areas, while supporting and enhancing your development process. When fully implemented, test planning will provide a mechanism to identify improvements in all aspects of the system and development process.

Preventive Testing

Preventive testing uses the philosophy that testing can actually improve the quality of the software being tested if it occurs early enough in the lifecycle. Specifically, preventive testing requires the creation of test cases to validate the requirements before the code is written

Testware is any document or product created as part of the testing effort (e.g., test cases, test plans, etc.). Testware is to testing what software is to development.

An added benefit of creating the test cases before the code is that the test cases themselves help document the software.

Software Test Documentation Template for Test Documents

1.	Test Plan Used for the master test plan and level-specific test plans.
2.	Test Design Specification Used at each test level to specify the test set architecture and coverage traces.
3.	Test Case Specification Used as needed to describe test cases or automated scripts.
4.	Test Procedure Specification Used to specify the steps for executing a set of test cases.
5.	Test Log Used as needed to record the execution of test procedures.
6.	Test Incident Report Used to describe anomalies that occur during testing or in production. These anomalies may be in the requirements, design, code, documentation, or the test cases themselves. Incidents may later be classified as defects or enhancements.
7.	Test Summary Report Used to report completion of testing at a level or a major test objective within a level.

Role	Description of Responsibilities
Manager	Communicate, plan, and coordinate.
Analyst	Plan, inventory, design, and evaluate.
Technician	Implement, execute, and check.
Reviewer	Examine and evaluate.

Test planning is one of the keys to successful software testing, yet it's frequently omitted due to time constraints, lack of training, or cultural bias.

Test planning CAN'T be separated from project planning.

All important test planning issues are also important project planning issues.

Risk Analysis

To mitigate the risk of some event causing damage, you must first estimate the probability that the event will occur. This probability has to be translated into some quantity, usually represented as a percentage-for example, "There is a 50 percent chance that this will happen." Next, you need to determine the severity of such a failure. Severity is usually measured in currency, such as dollars, and loss of life. If the severity is minor, then even a high probability of occurrence may still be judged to cause a trivial problem.

The probability that a thing will or won't occur can be calculated under certain circumstances-especially if you can answer a question like, "What was the outcome last time?" or "Do we know if the platform can really do what the maker claims it can?" If you can't provide a good measured answer to these questions up front, then you will need a strategy for dealing with the events that will occur later in the process. If the probability and severity cannot be measured, then they must be estimated. MITs risk analysis provides a formal method for both estimating up front and dealing with events as they unfold. In this chapter, we look at this formal approach to establishing risk and prioritizing the items on the test inventory.

MITs risk analysis uses both quantitative and qualitative analysis to establish a numeric value for risk based on a number of specific criteria. In the early planning phases of a test effort, this risk number is used to focus test resources to size the test effort. As the inventory evolves, the risk ranking plays an important part in actual test selection and optimal test coverage determination.

Metrics to Measure Performance

Was It Worth It?

Test Effectiveness

How Good Were the Tests?

Test Coverage

How Much of It Was Tested?