Carnegie Quality

Looking to build your testing library or just adding one or two books? Consider the following.

General Testing

How to Break Software: A Practical Guide to Testing By James A. Whittaker
A quick read (just 174 pages with the appendices), but lots of good info. If you’re looking for something to help you develop a tester’s mentality, this is it. Not everyone can be a good tester – you need the right kind of perspective. Most developers when they do unit testing make sure that their code is working. Most good testers I know want to find out how they can break the code. The book is mostly focused on PC applications – there are sections on UI testing and system interface testing. The book also comes with some software, which to be honest I’ve never tried.

Effective Software Testing: 50 Specific Ways to Improve Your Testing By Elfriede Dustin
This is a reference that is essentially 50 fully documented suggestions/best practices. While some of them are obvious (like #2 “Verify the Requirements”) the author does a good job at explaining the whys and hows behind each tip. The suggestions are grouped together logically (there’s a section on the requirements phase, test planning, test design, test management, etc…), so it’s easy to pick up this book over the life of your project and get specific hints tailored for where you are in the project life-cycle.

Software Testing By Ron Patton
There’s a second edition out now, though I haven’t yet picked it up. This is a great introductory book to software testing. It covers the different forms of testing, how to review documentation and code, test planning, and test case writing. One thing that Ron Patton really focuses on is the role of the tester in software quality: “The goal of a software tester is to find bugs, find them as early as possible, and make sure they get fixed.” Couldn’t say it better myself!

Test Management

Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing By Rex Black
Second Edition. Chapters include planning, mapping out the test coverage, defect tracking, managing test cases, staffing, and organizational considerations. I enjoy Rex Black’s writing style – it is very down-to-earth with a focus on practical advice. Each chapter ends with a real-life case study and exercise questions. While I think there’s good material for any test manager in this book, it would be particularly helpful for those either new to test management or are setting up a new test system/organization.

Critical Testing Processes: Plan, Prepare, Perform, Perfect By Rex Black
Builds on Black’s previous book. As the title suggests, the book’s focus is on testing processes. You can think of this book as a method for implementing continuous improvement in your testing process. Like his previous book, this one is full of practical advice and tips. I particularly like the planning chapters, and the chapters on reporting defects and producing test metrics. Throughout the book the author illuminates many of the chapters with a hypothetical case study – so you can see how to apply suggestions from the start of test planning all the way through post-test execution process improvement. Very highly recommended.

Quality Software Project Management By Robert T. Futrell, Donald F. Shafer, and Linda I. Shafer
The main audience for this book is Project Managers. Each chapter is well documented with footnotes and has it’s own bibliography, which is quite handy. There are a large number of templates and checklists. The book ties in very well with PMI standards, but is probably a bit more practical than your average PMI project management book (and since it’s on just software, more relevant to the software PM). The book’s strength is tying basic software engineering and software quality assurance concepts in with the process of planning, executing, and controlling a project. The book weighs in at over 1,500 pages, so it may not be exactly what you need – but if your PM doesn’t have a copy, they should.

Quality Assurance and Certification

Software Quality Assurance : From Theory to Implementation By Daniel Galin
This is a college textbook, but has a lot of very practical information. The book does a good job at laying a foundation about quality testing, but I think it’s stronger points are the sections on quality assurance. Looking at establishing programs for quality improvement, software/quality metrics, certification, or standards? Then this is a great addition to your library. There is also a section on organizational alignment for quality assurance.

The Certified Quality Manager Handbook Edited By Duke Okes and Russell T. Westcott
This book is a must if you’re getting certified by ASQ in Quality Management. It’s a handbook, so don’t expect in depth treatment on any single topic, but it’s a great overview of quality management. Use this as a reference to help identify areas that you want to research/learn more about.

General Skills

Flawless Consulting: A Guide to Getting Your Expertise Used By Peter Block
So what’s this book doing on a test manager’s bookshelf, you may ask. Testing is a funny activity in software development, if you think about. We don’t produce the product and in many cases we don’t have any real say in whether the software goes out the door or not. We should be honest, impartial, and knowledgeable producers of information that others (such as project management and sponsors) utilize – in other words, we are consultants. This book, more than any other on the subject IMO, will help you hone skills so that the information you produce is valuable, pertinent, and used by others.

Control charts were developed by Dr. Walter A. Shewhart of Bell Laboratories back in 1924. Dr. Shewhart was studying variation in manufactured products and wanted a way to visually display data and know when a manufacturing process was not performing as expected. Manufacturing is still the primary discipline that utilizes control charts, but they are also helpful in software testing.

Before we get into the use of control charts in software quality control, let’s first explain control charts. Processes that repeat over time (such as manufacturing ball-bearings or testing software) have two types of variation:

• Natural variation, or randomness
• Assignable variation, for which there is an underlying cause

Think of the manufacturing of ball-bearings. Ball-bearings are sold by size and within each size there are upper and lower tolerance levels that the bearings have to be within. So periodically samples are taken from the manufacturing process and measured. Now there is some normal variance, or randomness, in the samples – as long as they are within the tolerances levels everything’s ok. This describes natural variance. But what if during one measurement a large portion was outside of those tolerances? At that point we have something in the manufacturing process (or possibly the measurement process) that is causing the variation.

Control charts help you: 1) define the normal process, and 2) alert you when the process is being influenced by special causes (as known as “being out of control”).

Making a Control Chart

As you can probably guess, since we are dealing with samples, we’ll need some numbers. You can make control charts on lots of different testing variables – here are some possibilities:

• Change in Oustanding Defects
• Change in Resolved Defects
• New Defects
• Re-Opened Defects

If you are following my advice on tracking test statistics you already have the raw numbers you need. We’ll need to calculate four numbers:

Average

Since we’re doing a chart on an individual, and not a group (i.e. we’re not collecting samples from a manufacturing process) we just need the average over time. The technical term for the chart we’re going to make is an X Chart. Let’s assume we’re monitoring the number of new defects. We need enough data to calculate a meaningful average – roughly 20 data points.

Standard Deviation

With those 20 or so data points, calculate the standard deviation. In spreadsheets this is typically calculated by entering STDEV(LIST RANGE).

Upper Limit

This is the upper limit for the in control range. It is typically defined by: Average + 3*(Standard Deviation), though you can tighten the control range by using a smaller number than 3 if you wish.

Lower Limit

Just like the upper limit – the calculation is: Average – 3*(Standard Deviation).

Making the Chart

Once you have your control limits and data points, making the chart is easy. Plot your data points as a line chart, then place horizontal bars on your chart for the lower limit, average, and upper limit.

Let’s take a look at an example before we discuss how to use control charts during testing. In this example we’re looking at the daily change in resolved defects (defects closed and rejected/withdrawn). We have an average daily increase of 4.7, a standard deviation of 3.8, an upper control limit of 16.2, and a lower control limit of -6.8. Note that you probably won’t see negative numbers on this chart – that’s ok. You’ll notice that there are two areas that are out of control – a data point of 18 and 13 data points below the average. Let’s do a quick discussion on the definition of out of control.

When is a Process Out of Control?

There are two instances when a process is out of control:

• Randomness is gone (rule of 7)
• Process goes beyond control limits

The rule of 7 says that if there are 7 or more consecutive data points above or below the average, then the process is no longer random – that is, something non-random is influencing the process. When this happens, or when data points go beyond the control limit, we say that there is “assignable cause” and it’s your job as a Test Manager to find out what is happening.

In the example there are 13 consecutive data points below the average. In this case, the development team was re-working a significant design item and there were few bug fixes released to test. When presenting data like this it’s important to show the why in addition to the what – so on the example chart you’ll see call-out boxes that offer the viewer additional insight into the chart.

When to Utilize Control Charts

Control charts are not for every situation. You shouldn’t use control charts when you don’t have applicable historical data to base your control limits on. Examples of this might include:

• The first phase of a brand new project with no comparable project.
• A new set of developers and/or testers (even if the project is similar to another one, if you have a completely new set of people, that’s
  enough of a difference to make comparisons suspect).
• A short test phase (if your test schedule is two weeks in duration, the use of control charts is probably not going to gain you anything).

The real power in utilizing control charts is in larger projects with multiple test streams and/or with multi-phase development projects. In these cases you have lots of historical data, which makes your control limits accurate, and makes spotting trouble simple!

Next Steps

So you’ve gathered your data, set your control limits, and you’re plotting your data points as you execute test. What next? Control charts are a tool for process improvement, and should follow the typical plan-do-check-act process improvement cycle. Make sure you’re performing these steps:

• Inform project personnel on what control charts are and how they can help the project – provide regular updates to all project members on how things are progressing.
• Periodically check the control limits and see if they need to be updated.
• Assess the effectiveness of the control charts – are they providing meaningful value? If not, it may be time to shelve them.

Feel free to contact me with questions or suggestions that you have!

Have you ever been in this situation? You’re 6 weeks into a 4 week test cycle. The defect count is not going down, you’ve got over half of your test scenarios left to run, and everyone keeps asking – when is testing going to complete?

Sound familiar? Unfortunately, we’ve all been there before. Of course, the real answer is that you shouldn’t be in test – you’re probably stuck in a code and fix mode, which is very difficult to get out of. Better to do some analysis and find out why test is faltering (break out the Ishikawa diagram or your favorite problem solving technique), and go back to another build cycle. Of course the reality is that project management and the business sponsors don’t want to hear that the project needs to go “backwards”. So you’re stuck in test and need to provide an estimate.

Even if development cannot provide an accurate estimate, the test team probably can. Here’s why – you’re several weeks into a test cycle. If you’ve been doing things properly you have some great test statistics that can answer this question.

Read more for details and a template that can help.

History’s Worst Software Bugs, an article from Wired News, is making a lot of people’s blogs. If you haven’t yet read it, I highly recommend it.

I’ve never had an experience with someone dying from a project I was on, but here’s my story. Back in the mid-90′s I was working at a telecommunications firm managing the development of a custom client-server VB application. It was an order-entry system for ISDN that interfaced to the telco’s Service Order mainframe. We had just gone live when the mainframe had an abend. If you haven’t worked with mainframes before, an abend is bad news – think blue screen of death, only much, much worse. After some investigation it was determined that our application sent over a carriage return, which crashed the mainframe. We then did some investigation and found that a user had opened up Notepad to write some comments and then pasted the comments, with carriage returns, into the comments field in our application, which we merrily sent along to the mainframe. Needless to say we both modified code to strip out carriage returns, but it goes to show that if there’s a way to break your software, you can be sure users will find it!