Defect Modeling Template

Have you ever been in this situation? You’re 6 weeks into a 4 week test cycle. The defect count is not going down, you’ve got over half of your test scenarios left to run, and everyone keeps asking - when is testing going to complete?

Sound familiar? Unfortunately, we’ve all been there before. Of course, the real answer is that you shouldn’t be in test - you’re probably stuck in a code and fix mode, which is very difficult to get out of. Better to do some analysis and find out why test is faltering (break out the Ishikawa diagram or your favorite problem solving technique), and go back to another build cycle. Of course the reality is that project management and the business sponsors don’t want to hear that the project needs to go “backwards”. So you’re stuck in test and need to provide an estimate.

Even if development cannot provide an accurate estimate, the test team probably can. Here’s why - you’re several weeks into a test cycle. If you’ve been doing things properly you have some great test statistics that can answer this question. This is what you need:

  • Average number of defects found each day of testing
  • Average number of defects delivered to test each day by development
  • Average defect pass rate

Let’s review each statistic in turn.

Average Defects Found Each Day

Our model essentially says you start with X, you add Y each day, and you remove Z each day, which leaves you with your new X for the next day. Average defects found each day is the Y in the equation. If you’ve been collecting statistics for awhile, you should have a good handle on what this number is. The model has a constant that will scale this average back over time - the theory being that you should identify fewer defects later in the test process.

Average Defects Delivered to Test Each Day

This is part of Z in the equation, along with the average defect pass rate. Base this on historical data just like the average defects found each day. Again, the model has a constant that will scale this back over time - as the overall defect count falls, so will the fixed defects.

Average Defect Pass Rate

This is the other part of Z in the equation. This statistic represents the percentage of defects fixed by development and given to test that pass examination. The best way to understand this is through an example - say that development turns over 10 defects a day. If the pass rate is 70%, then you will close 7 out of the 10, and turn the other 3 back to development for re-fixing.

The Model and Template Explained

First thing - right click and grab a copy of the template in OpenOffice format, or in Excel format.
Open up the model - the first worksheet contains some quick info on some of the main cells, the second worksheet is a blank model ready for you to add data to, and the third worksheet is a sample. I’ll go through each column in the model.

Test Day

This is simply the testing dates - you could change the format and just sequently number the rows, but I’ve found the ability to reference back an actual day to be helpful in discussions with the broader team.

Outstanding (BOD)

BOD stands for Beginning of Day - this the number of outstanding defects prior to that day’s activity. It is equal to the outstanding defects at the end of the previous day. A quick note on outstanding - I define a defect as outstanding if it is either new, actively being worked on by development, fixed by development, or being tested - in other words, defects are outstanding until they are closed, and you’ll see why in a moment.

New

Number of new defects identified.

Closed

Number of defects successfully tested and closed by the test team.

Rejected

Number of logged defects that are mutually rejected by test, development, and project management. The model does not build-in a projected number of rejected defects each day (though of course you could change it to do so), but does let you capture that data as you note actual performance.

Re-Opened

Number of defects tested and given back to the development team for re-work. These defects are the reason why we don’t consider defects to be resolved until the test team closes the defect.

Delivered to Test

Number of fixed defects turned over to the test team by the development team.

Outstanding (EOD)

As you can guess, EOD stands for End of Day - this number is equal to Outstanding (BOD) + New - Closed - Rejected.

Failure Rate (Actual)

This is a calculated field and you should not enter data into this column. The Failure Rate denotes the % of fixed defects that do not pass test and is equal to Re-Opened/Delivered to Test.

Actual Data

This field is used by some of the calculated fields - as you use the model and put in the day’s actual test results, change this value to “yes”.

Comments

Use this to capture aspects of the day’s testing as applicable.

Constants

The template uses constants to predict what will happen over time based on certain assumptions you make. You can tweak the constants and see what impact they make. I’ll explain each constant.

Defects Found per Day

This constant models the New column.

Defects Found Scale Factor

This constant scales back the number of new defects found over time, the assumption being that you will find more defects early in testing, and fewer later in testing. The model has a fairly simplistic straight-line factor - if you want, you can change the values in column M to produce different results.

Test Failure Rate

The test failure rate depends on how well defects are documented, requirements are understood, development is disciplined in unit testing, and a host of other variables. This number represents the percentage of defects that do not pass testing after being turned over by the development team.

Fixed Defects Delivered to Test per Day

This constant represents the number of defects fixed by the development team and turned over to the test team each day.

Fixed Defects Delivered to Test Scale Factor

Just like the defects found scale, the defects delivered to test is scaled back over time, assuming that as the total count of defects drops, the number that can be fixed falls (also the more difficult defects to fix are typically fixed towards the end of the test phase).

Upper & Lower Limits

The template uses conditional formatting to color in cells based on values put in here, supplying you with a visual clue on how things are proceeding. This is optional and you can turn the formatting off if you wish.

Averages

This data is based on actuals that you enter into the model and will only be calculated if you select “yes” in the Actual Data column.

Using the Model & Template

Let’s look at the example first. You can see that I’ve started with 16 defects (the model works best when you’ve collected some real data on averages and trends). You’ll also notice that the first 7 rows of data are actual data. The average defects found per day (5.9) is nearly identical to the modeled constant of 6. But the defects delivered to test average is 6.1 while the constant is 10 (you can also see a wide discrepancy in the test failure rate). The implications of this is that unless the speed and quality of fixed defects is improved, the test phase duration will likely increase.
Play around with the example to see how the numbers change. One advantage to using this model is that you can clearly demonstrate that the level of outstanding defects is influenced by the number found, the rate at which they are fixed, and the quality of the fixes.

What Next?

While I find this model very helpful, I will readily admit that it is fairly simplistic:

  • It assumes that defects are closed the same day they are delivered to test.
  • It assumes that the chief goal of testing is hitting some level of outstanding defects (whether it is zero, 5, or some other number). The reality is that managers make decisions on test completion based on a host of factors, such as the severity of outstanding defects, the likelihood of a given defect being uncovered in production, the cost of delaying production, etc…
  • It doesn’t differentiate between defect severities - essentially defects are treated as if they are equally important, and that is rarely the case in real-life.
  • It assumes that non-defect testing (executing test cases, regression testing, etc…) happens in parallel and is not significantly impacted by the volume of defects.

You could adapt the model to account for these things. Remember, this model is not going to provide a magic answer - it’s one input that can aid in test management, but by no means is it the only one.
Contact me with questions or suggestions that you have!