Carnegie Quality

I’ve posted a template for producing test statistics. The template is a follow-up on my previous on 10 Tips for Test Statistics. Drop me a comment and let me know what you think.

In order to effectively and efficiently manage the testing process, you must be able to generate meaningful and accurate test statistics. You can’t possibly really know how testing is going without the proper metrics. But generating those metrics takes some planning and work.

Here are some tips for generating and managing test statistics.

Use a Robust Bug Tracking System

This may seem like a given, but don’t leave the selection and installation of a bug tracking system/database to the last minute. If you don’t have one in place already, you’ll need time (and possibly other resources within your organization) to install and setup the system, plus time to tailor it to your specific needs and train the project team. I’ve used numerous tools and don’t really have a preference – there are a ton of places you can go and get people’s opinions on which one to use. Just make sure it supports:

• Multiple users (preferably with support for different access groups)
• Modification to your unique needs (i.e. the ability to add/remove fields and customize values in drop-down fields)
• Attachments

There are some free systems out there (like Bugzilla) and tons of commercial products. My firm is currently using IBM’s Rational Portfolio Manager, and while I’m not wild about the product (it is more of a Project Manager tool), it gets the job done.

Plan Your Metrics Ahead of Time

If you want to produce a metric on mean time to resolution, then you need to capture an open date and a resolved date for each defect. Think about the metrics you want to produce and make sure that the fields and values you are capturing support the metrics you want to generate. Make modifications to your bug tracking system before you begin test – it’s not something you want to be doing during test execution.

Think About Future Projects, Not Just Your Current One

In addition to supporting your current project, test metrics should also assist in estimating future projects and continuous improvement. If you identified X number of defects in project A, and project B has similar characteristics, then it makes sense that you will identify some number near X when testing project B. Information from previous projects should give you some guidance in future projects, such as expected problem areas, expected volume of defects, expected resolution times, etc… They should also support continuous improvement – if you’ve identified problem areas with the application, for example, then more attention should be paid to those areas in the future (e.g. increased design reviews, peer code reviews, additional unit testing, etc…).

One suggested approach is to capture and track the phase the defect was injected. To do this you need to list out the phases of your project (e.g. Requirements, Design, Development, Unit Test, etc…) and for each defect determine when the defect was introduced (injected) into the application. This takes some work, and developers typically won’t want to go to this level of effort (since they are normally the ones that have to make the determination), but it has real value long term. If you determine, for example, that 25% of your defects come from the Requirements phase, then you have real ammo to go back to Project Management and suggest that that particular phase needs more attention in future releases. It is a great tool for quality improvement over the long term.

Consider Capturing Additional Information

There’s certain data that every system captures (e.g. id, description, status, creation date, resolution date, severity), but it’s worth spending a bit more time capturing additional information. Note that some defect tracking databases will already have some of these fields present. Here are some candidates worth considering:

• Subsystem/Component – Tracking the subsystem can help you identify issues in certain aspects of the application that may be caused by inadequate requirements, sub-standard design, immature technology, under-performing developers, etc…
• Root Cause – Like the phase injected field described above, capturing a root cause is a great way to improve the quality of your projects over time.
• Re-Open Flag – Tracking whether a defect was re-opened (or failed test after being fixed) allows you to measure the effectiveness of the bug fix process. If you notice a large percentage of defects being re-opened, then you need to find out why (e.g. developers are not properly unit testing their fixes or the testers are not properly documenting the defects).

Have a Defined System for Producing Metrics

If you’re lucky, the bug tracking database/system you’re using will produce the metrics you want. If not, you’ll need some kind of method for cranking those numbers out. Whatever method you use, you need to figure it out prior to the start of testing and work out any kinks in the system. Don’t leave this until the day you need to produce your first status report!

I’m generally never happy with the metrics and graphs that come out of bug tracking databases. I usually setup an export out of the database (check your tool’s documentation – most systems have the capability to export a list of defects) and import it into a spreadsheet, which I use to produce metrics and graphs. Once I get my system working, it’s relatively painless to crank out the daily information I need. I’m working on a generic template which I’ll post here soon. Please keep in mind that I am not advocating using a spreadsheet to manage the defects themselves – unless you’re a team of one, this is impossible.

Use Trend Data

Snapshots in time are fine, but to truly understand what’s occurring you need to examine how the data is changing over time. Trends help you analyze data against the broader context of testing. Trends can assist in many ways, including:

• Identifying what’s happening to the defect backlog
• Estimating volume (and severity) of un-found defects
• Spotting problems (see my previous article on Using Control Charts in Testing)

Educate the Project Team on Using Metrics

Not just the immediate project team, but anyone who will be reading the information you publish (e.g. management, project sponsors). Data without guidance on how to interpret it can be dangerous. Let me give you a couple of examples:

• You’re half way through a test phase. The number of defects logged each day hasn’t dropped significantly. Should you be concerned? Well, what’s happening with the resolved count? And what is the severity of the defects being logged? If the resolved count is steadily growing (i.e. the team is getting through the backlog of defects) and the severity of defects is dropping, that’s probably not a bad position to be in half-way through the test phase.
• The resolved defects count continues to climb, and the average time to resolution has been dropping recently. Things are going great, right? Well, what if the average days outstanding count is climbing at the same time. What does that tell you? It means that the backlog of defects is increasing – you may want to look at the severity of the defects being resolved. Developers may be cherry-picking easy, lower severity defects to fix (which would decrease the mean time to resolution since these are quick to fix), but at the expense of more difficult, higher severity defects (which results in an increased backlog).

Use a Dashboard

A dashboard is simply a summary (usually a page or two) that gives the reader a quick view into the most important aspects of the project. A testing dashboard should include statistics and trends about defect counts, but also:

• Progress of test scripts/cases
• Schedule and resources used vs. planned
• Top level risks and issues

Qualitatively Describe Metrics

Sometimes numbers speak for themselves, but sometimes they don’t. While you may know why a specific metric or trend significantly changed, everyone reading your status report or looking at the metrics may not. Think about adding callout boxes to graphs and a sentence or two to status reports described what’s impacting the numbers.

Get Organized!

To sum this all up – get organized. Find a system that works for you and stick to it. Here are some things I do that I find helpful:

• Carve out a set time each day to do metric production. For me, I like to get in early in the morning and run the reports from the previous day.
• Make sure you have a backup system in place and operating in case something happens to the defect database. I also do a daily print of the outstanding defect list and keep it in a binder (this has the added advantage of providing a historical snapshot if your defect tracking system can’t provide one).
• Develop a naming convention for spreadsheets, documents, etc… and a central repository so people can quickly find metrics, summaries, and dashboards.