31.7.08

The Benefits of Automation and Tools


A project-level pie chart shows how many bugs were found in each major functional area of the software.

If you're testing a particular feature that means you may need to run your tests not once, but potentially dozens of times. You'll check that the bugs you found in previous test runs were indeed fixed and that no new bugs were introduced. This process of rerunning your tests is known as regression testing.

If a small software project had several thousand test cases to run, there might be barely enough time to execute them just once. Running them numerous times might be impossible, let alone monotonous. Software test tools and automation can help solve this problem by providing a more efficient means to run your tests than by manual testing.

The principal attributes of tools and automation are:

· Speed. Think about how long it would take you to manually try a few thousand test cases for the Windows Calculator. You might average a test case every five seconds or so. Automation might be able to run 10, 100, even 1000 times that fast.

· Efficiency. While you're busy running test cases, you can't be doing anything else. If you have a test tool that reduces the time it takes for you to run your tests, you have more time for test planning and thinking up new tests.

· Accuracy and Precision. After trying a few hundred cases, your attention span will wane and you'll start to make mistakes. A test tool will perform the same test and check the results perfectly, each and every time.

· Resource Reduction. Sometimes it can be physically impossible to perform a certain test case. The number of people or the amount of equipment required to create the test condition could be prohibitive. A test tool can be used to simulate the real world and greatly reduce the physical resources necessary to perform the testing.

· Simulation and Emulation. Test tools are often used to replace hardware or software that would normally interface to your product. This "fake" device or application can then be used to drive or respond to your software in ways that you choose and ways that might otherwise be difficult to achieve.

· Relentlessness. Test tools and automation never tire or give up. They're like that battery-operated bunny of the TV commercials they can keep going and going and…

All this probably sounds like great news. You could have test tools do all the work for you turn them loose and wait for the results. Unfortunately, it's not that easy. Houses aren't built automatically, even though carpenters have power saws and nail guns. The tools just make it easier for them to do their work and for the resulting work to be of higher quality. Software test tools operate the same way.

NOTE:

Software test tools aren't a substitute for software testers they just help software testers perform their jobs better.

It's important to note that using test tools isn't always the right answer. Sometimes there's no substitute for manual testing. For now, take in the information about what tests tools can do and how they work. Think about how you might use them to complement your testing tasks. At the end of this chapter, you'll learn about a few limitations and cautions to consider before you embark on using tools on your projects.


The fundamentals of software testing

· Static black-box testing involves examining the specification and looking for problems before they get written into the software.

· Dynamic black-box testing involves testing the software without knowing how it works.

· Static white-box testing involves examining the details of the written code through formal reviews and inspections.

· Dynamic white-box testing involves testing the software when you can see how it works and basing your tests on that information.

Here's a list of seven important traits common to a good UI. It doesn't matter if the UI is on a digital watch or is the Mac OS X interface, they all still apply.

· Follows standards and guidelines

· Intuitive

· Consistent

· Flexible

· Comfortable

· Correct

· Useful

When you're testing a user interface, consider the following things and how they might apply to gauging how intuitive your software is:

· Is the user interface clean, unobtrusive, not busy? The UI shouldn't get in the way of what you want to do. The functions you need or the response you're looking for should be obvious and be there when you expect them.

· Is the UI organized and laid out well? Does it allow you to easily get from one function to another? Is what to do next obvious? At any point can you decide to do nothing or even back up or back out? Are your inputs acknowledged? Do the menus or windows go too deep?

· Is there excessive functionality? Does the software attempt to do too much, either as a whole or in part? Do too many features complicate your work? Do you feel like you're getting information overload?

· If all else fails, does the help system really help you?

Follow Experience, Intuition, and Hunches

There's no better way to improve as a software tester than to gain experience. There's no better learning tool than just doing it, and there's no better lesson than getting that first phone call from a customer who found a bug in the software you just finished testing.

Experience and intuition can't be taught. They must be gained over time. You can apply all the techniques you've learned so far and still miss important bugs. It's the nature of the business. As you progress through your career, learning to test different types and sizes of products, you'll pick up little tips and tricks that steer you toward those tough-to-find bugs. You'll be able to start testing a new piece of software and quickly find bugs that your peers would have missed.

Take notes of what works and what doesn't. Try different approaches. If you think something looks suspicious, take a closer look. Go with your hunches until you prove them false.

Think like a Hacker

No software is 100% secure. The hackers of the world know this and will seek to find vulnerabilities in your software and exploit them. As a tester, you'll need to be devious and conniving. Try to think of what things of value your software contains, why someone might want to gain access to them, and what the methods of entry might be for them to get in. Don't be gentle. They won't be.

Look for Bugs Where You've Already Found Them

There are two reasons to look for bugs in the areas where you've already found them:

· The more bugs you find, the more bugs there are. If you discover that you're finding lots of bugs at the upper boundary conditions across various features, it would be wise to emphasize testing these upper boundaries on all features. Of course you're going to test these anyway, but you might want to throw in a few special cases to make sure the problem isn't pervasive.

· Many programmers tend to fix only the specific bug you report. No more, no less. If you report a bug that starting, stopping, and restarting a program 255 times results in a crash, that's what the programmer will fix. There may have been a memory leak that caused the problem and the programmer found and fixed it. When you get the software back to retest, make sure you rerun the same test for 256 times and beyond. There could very well be yet another memory leak somewhere out there.

Repetition, Stress, and Load

Three other test-to-fail state tests are repetition, stress, and load. These tests target state handling problems where the programmer didn't consider what might happen in the worst-case scenarios.

Repetition testing involves doing the same operation over and over. This could be as simple as starting up and shutting down the program over and over. It could also mean repeatedly saving and loading data or repeatedly selecting the same operation. You might find a bug after only a couple repetitions or it might take thousands of attempts to reveal a problem.

The main reason for doing repetition testing is to look for memory leaks. A common software problem happens when computer memory is allocated to perform a certain operation but isn't completely freed when the operation completes. The result is that eventually the program uses up memory that it depends on to work reliably. If you've ever used a program that works fine when you first start it up, but then becomes slower and slower or starts to behave erratically over time, it's likely due to a memory leak bug. Repetition testing will flush these problems out.

Stress testing is running the software under less-than-ideal conditions: slow memory, low disk space, slow CPUs, slow modems, and so on. Look at your software and determine what external resources and dependencies it has. Stress testing is simply limiting them to their bare minimum. Your goal is to starve the software. Does this sound like boundary condition testing? It is.

Load testing is the opposite of stress testing. With stress testing, you starve the software; with load testing, you feed it all that it can handle. Operate the software with the largest possible data files. If the software operates on peripherals such as printers or communications ports, connect as many as you can. If you're testing an Internet server that can handle thousands of simultaneous connections, do it. Max out the software's capabilities. Load it down.

Don't forget about time as a load testing variable. With most software, it's important for it to run over long periods. Some software should be able to run forever without being restarted.

NOTE:

There's no reason that you can't combine repetition, stress, and load, running all the tests at the same time. This is a sure way to expose severe bugs that might otherwise be difficult to find.

What Makes a Good Software Tester?

Here's a list of traits that most software testers should have:

· They are explorers. Software testers aren't afraid to venture into unknown situations. They love to get a new piece of software, install it on their PC, and see what happens.

· They are troubleshooters. Software testers are good at figuring out why something doesn't work. They love puzzles.

· They are relentless. Software testers keep trying. They may see a bug that quickly vanishes or is difficult to re-create. Rather than dismiss it as a fluke, they will try every way possible to find it.

· They are creative. Testing the obvious isn't sufficient for software testers. Their job is to think up creative and even off-the-wall approaches to find bugs.

· They are (mellowed) perfectionists. They strive for perfection, but they know when it becomes unattainable and they're okay with getting as close as they can.

· They exercise good judgment. Software testers need to make decisions about what they will test, how long it will take, and if the problem they're looking at is really a bug.

· They are tactful and diplomatic. Software testers are always the bearers of bad news. They have to tell the programmers that their baby is ugly. Good software testers know how to do so tactfully and professionally and know how to work with programmers who aren't always tactful and diplomatic.

· They are persuasive. Bugs that testers find won't always be viewed as severe enough to be fixed. Testers need to be good at making their points clear, demonstrating why the bug does indeed need to be fixed, and following through on making it happen.

Software testing is a critical job. With the size and complexity of today's software, it's imperative that software testing be performed professionally and effectively. Too much is at risk. We don't need more defective computer chips, crashed systems, or stolen credit card numbers.

Look at a few examples so you can start thinking about all the boundary possibilities:

· If a text entry field allows 1 to 255 characters, try entering 1 character and 255 characters as the valid partition. You might also try 254 characters as a valid choice. Enter 0 and 256 characters as the invalid partitions.

· If a program reads and writes to a CD-R, try saving a file that's very small, maybe with one entry. Save a file that's very largejust at the limit for what the disc holds. Also try saving an empty file and a file that's too large to fit on the disc.

· If a program allows you to print multiple pages onto a single page, try printing just one (the standard case) and try printing the most pages that it allows. If you can, try printing zero pages and one more than it allows.

· Maybe the software has a data-entry field for a 9-digit ZIP code. Try 00000-0000, the simplest and smallest. Try entering 99999-9999 as the largest. Try entering one more or one less digit than what's allowed.

· If you're testing a flight simulator, try flying right at ground level and at the maximum allowed height for your plane. Try flying below ground level and below sea level as well as into outer space.

Since you can't test everything, performing equivalence partitioning around boundary conditions, such as in these examples, to create your test cases is critical. It's the most effective way to reduce the amount of testing you need to perform.

NOTE:

It's vitally important that you continually look for boundaries in every piece of software you work with. The more you look, the more boundaries you'll discover, and the more bugs you'll find.

What Exactly Does a Software Tester Do?

The goal of a software tester is to find bugs, find them as early as possible, and make sure they get fixed.

30.7.08

The Cost of Bugs

The costs are logarithmic that is, they increase tenfold as time increases. A bug found and fixed during the early stages when the specification is being written might cost next to nothing, or $1 in our example. The same bug, if not found until the software is coded and tested, might cost $10 to $100. If a customer finds it, the cost could easily be thousands or even millions of dollars.

Why Do Bugs Occur?

Bugs are caused for numerous reasons, but, in this sample project analysis, the main cause can be traced to the specification.

What you'll be surprised to find out is that most of them aren't caused by programming errors. Numerous studies have been performed on very small to extremely large projects and the results are always the same. The number one cause of software bugs is the specification.

What Is a Bug?

Terms for Software Failures

Depending on where you're employed as a software tester, you will use different terms to describe what happens when software fails. Here are a few:

Defect

Variance

Fault

Failure

Problem

Inconsistency

Error

Feature

Incident

Bug

Anomaly


(There's also a list of unmentionable terms, but they're most often used privately among programmers.)

You might be amazed that so many names could be used to describe a software failure. Why so many? It's all really based on the company's culture and the process the company uses to develop its software. If you look up these words in the dictionary, you'll find that they all have slightly different meanings. They also have inferred meanings by how they're used in day-to-day conversation.

For example, fault, failure, and defect tend to imply a condition that's really severe, maybe even dangerous. It doesn't sound right to call an incorrectly colored icon a fault. These words also tend to imply blame: "It's his fault that the software failed."

Anomaly, incident, and variance don't sound quite so negative and are often used to infer unintended operation rather than all-out failure. "The president stated that it was a software anomaly that caused the missile to go off course."

Problem, error, and bug are probably the most generic terms used.

For the purposes of this book and much of the software industry, a software bug occurs when one or more of the following five rules is true:

1. The software doesn't do something that the product specification says it should do.

2. The software does something that the product specification says it shouldn't do.

3. The software does something that the product specification doesn't mention.

4. The software doesn't do something that the product specification doesn't mention but should.

5. The software is difficult to understand, hard to use, slow, orin the software tester's eyeswill be viewed by the end user as just plain not right.