Carnegie Quality

Control charts were developed by Dr. Walter A. Shewhart of Bell Laboratories back in 1924. Dr. Shewhart was studying variation in manufactured products and wanted a way to visually display data and know when a manufacturing process was not performing as expected. Manufacturing is still the primary discipline that utilizes control charts, but they are also helpful in software testing.

Before we get into the use of control charts in software quality control, let’s first explain control charts. Processes that repeat over time (such as manufacturing ball-bearings or testing software) have two types of variation:

• Natural variation, or randomness
• Assignable variation, for which there is an underlying cause

Think of the manufacturing of ball-bearings. Ball-bearings are sold by size and within each size there are upper and lower tolerance levels that the bearings have to be within. So periodically samples are taken from the manufacturing process and measured. Now there is some normal variance, or randomness, in the samples – as long as they are within the tolerances levels everything’s ok. This describes natural variance. But what if during one measurement a large portion was outside of those tolerances? At that point we have something in the manufacturing process (or possibly the measurement process) that is causing the variation.

Control charts help you: 1) define the normal process, and 2) alert you when the process is being influenced by special causes (as known as “being out of control”).

Making a Control Chart

As you can probably guess, since we are dealing with samples, we’ll need some numbers. You can make control charts on lots of different testing variables – here are some possibilities:

• Change in Oustanding Defects
• Change in Resolved Defects
• New Defects
• Re-Opened Defects

If you are following my advice on tracking test statistics you already have the raw numbers you need. We’ll need to calculate four numbers:

Average

Since we’re doing a chart on an individual, and not a group (i.e. we’re not collecting samples from a manufacturing process) we just need the average over time. The technical term for the chart we’re going to make is an X Chart. Let’s assume we’re monitoring the number of new defects. We need enough data to calculate a meaningful average – roughly 20 data points.

Standard Deviation

With those 20 or so data points, calculate the standard deviation. In spreadsheets this is typically calculated by entering STDEV(LIST RANGE).

Upper Limit

This is the upper limit for the in control range. It is typically defined by: Average + 3*(Standard Deviation), though you can tighten the control range by using a smaller number than 3 if you wish.

Lower Limit

Just like the upper limit – the calculation is: Average – 3*(Standard Deviation).

Making the Chart

Once you have your control limits and data points, making the chart is easy. Plot your data points as a line chart, then place horizontal bars on your chart for the lower limit, average, and upper limit.

Let’s take a look at an example before we discuss how to use control charts during testing. In this example we’re looking at the daily change in resolved defects (defects closed and rejected/withdrawn). We have an average daily increase of 4.7, a standard deviation of 3.8, an upper control limit of 16.2, and a lower control limit of -6.8. Note that you probably won’t see negative numbers on this chart – that’s ok. You’ll notice that there are two areas that are out of control – a data point of 18 and 13 data points below the average. Let’s do a quick discussion on the definition of out of control.

When is a Process Out of Control?

There are two instances when a process is out of control:

• Randomness is gone (rule of 7)
• Process goes beyond control limits

The rule of 7 says that if there are 7 or more consecutive data points above or below the average, then the process is no longer random – that is, something non-random is influencing the process. When this happens, or when data points go beyond the control limit, we say that there is “assignable cause” and it’s your job as a Test Manager to find out what is happening.

In the example there are 13 consecutive data points below the average. In this case, the development team was re-working a significant design item and there were few bug fixes released to test. When presenting data like this it’s important to show the why in addition to the what – so on the example chart you’ll see call-out boxes that offer the viewer additional insight into the chart.

When to Utilize Control Charts

Control charts are not for every situation. You shouldn’t use control charts when you don’t have applicable historical data to base your control limits on. Examples of this might include:

• The first phase of a brand new project with no comparable project.
• A new set of developers and/or testers (even if the project is similar to another one, if you have a completely new set of people, that’s
  enough of a difference to make comparisons suspect).
• A short test phase (if your test schedule is two weeks in duration, the use of control charts is probably not going to gain you anything).

The real power in utilizing control charts is in larger projects with multiple test streams and/or with multi-phase development projects. In these cases you have lots of historical data, which makes your control limits accurate, and makes spotting trouble simple!

Next Steps

So you’ve gathered your data, set your control limits, and you’re plotting your data points as you execute test. What next? Control charts are a tool for process improvement, and should follow the typical plan-do-check-act process improvement cycle. Make sure you’re performing these steps:

• Inform project personnel on what control charts are and how they can help the project – provide regular updates to all project members on how things are progressing.
• Periodically check the control limits and see if they need to be updated.
• Assess the effectiveness of the control charts – are they providing meaningful value? If not, it may be time to shelve them.

Feel free to contact me with questions or suggestions that you have!

This work, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 License.