Published by Brad Kuhn on 18 May 2007
I’ve just posted a template for test case planning and execution. If you don’t have access to a testing tool, you’ll find this helpful in planning and tracking test cases.
Published by Brad Kuhn on 12 May 2006
Ok, it’s been awhile since I’ve posted anything. I’ve been swamped with work and life and all that – but I still have plenty of items that I want to get posted here. So let me get back to the swing of things with a post today on managing software defects. As I discussed on a previous post, planning how you are going to manage the testing process ahead of time is critical. One of the key processes you need to have worked out prior to testing is the defect life-cycle you’re going to use.
The defect life-cycle defines what states (or statuses) your defects can reach, and how they move from one state to another. In this article I’ll be demonstrating a fairly standard approach – but it’s an approach I’ve been using for years with a lot of success. I’ll also describe some details that are specific to the software I’m currently using – TestTrack Pro by Seapine Software. Most of what I’ll describe can be leveraged over to the software you’re using, depending on it’s capabities.
First some definitions:
Grab this picture of my sample workflow – this was created in TestTrack Pro, so you may need to tweak it to work with whatever software you are using.
The first thing you notice is a lot of boxes/rectangles with arrows going all over the place. The boxes are States and the arrows are Events. I’ll describe each one below.
Initial state for defects. Valid actions for Open defects are Fix and Force Close. You would Force Close an item that didn’t require a normal fix, release, and test cycle – e.g. an entry wasn’t actually a defect or an entry resulted in an enhancement request.
Once the Development team has fixed, unit tested, and made appropriate documentation updates, the entry should be moved to the Fixed state. Note that Fixed does not mean that the Test team is ready to look at the defect yet.
Normally during testing there is a process for promoting fixed code into the Test environment and most defects (but not normally all) require this code promotion before the Test team can re-test. This is critical part of the testing process – make sure that the Development Team Lead and you have a very clear understanding on how this promotion occurs, and who moves defect entries from Fixed to Ready To Test once the code promotion occurs. The Event in the sample workflow is called Release To Test – you’ll also notice an arrow from Open leading to the Ready To Test State – not all fixes require a code promotion.
The Development Team has fixed the defect and the Test Team has verified the fix.
Sometimes fixes don’t fix the problem, and that’s what this State captures. Many people don’t use this State and instead cycle back to Open, but I prefer to track it in a separate status. I’ve argued in a previous post that the defect pass rate is an important factor in estimating test completion (as well as overall software quality). Maintaining a separate State makes calculating this ratio much easier.
This State is necessary, but should be used sparingly. Sometimes defects do require a fix, but don’t require a release and re-test cycle. If your software doesn’t have a good audit capability, you may want to consider not having this state – it can be abused.
Just what it says. Some test managers prefer to start over again with a new defect – it’s your call.
So it should be obvious how you want to utilize the different States and what kind of metrics and reports you’ll need. Here’s a list to start with:
Published by Brad Kuhn on 13 Dec 2005
I’ve posted a template for producing test statistics. The template is a follow-up on my previous on 10 Tips for Test Statistics. Drop me a comment and let me know what you think.
Published by Brad Kuhn on 02 Dec 2005
In order to effectively and efficiently manage the testing process, you must be able to generate meaningful and accurate test statistics. You can’t possibly really know how testing is going without the proper metrics. But generating those metrics takes some planning and work.
Here are some tips for generating and managing test statistics.
This may seem like a given, but don’t leave the selection and installation of a bug tracking system/database to the last minute. If you don’t have one in place already, you’ll need time (and possibly other resources within your organization) to install and setup the system, plus time to tailor it to your specific needs and train the project team. I’ve used numerous tools and don’t really have a preference – there are a ton of places you can go and get people’s opinions on which one to use. Just make sure it supports:
There are some free systems out there (like Bugzilla) and tons of commercial products. My firm is currently using IBM’s Rational Portfolio Manager, and while I’m not wild about the product (it is more of a Project Manager tool), it gets the job done.
If you want to produce a metric on mean time to resolution, then you need to capture an open date and a resolved date for each defect. Think about the metrics you want to produce and make sure that the fields and values you are capturing support the metrics you want to generate. Make modifications to your bug tracking system before you begin test – it’s not something you want to be doing during test execution.
In addition to supporting your current project, test metrics should also assist in estimating future projects and continuous improvement. If you identified X number of defects in project A, and project B has similar characteristics, then it makes sense that you will identify some number near X when testing project B. Information from previous projects should give you some guidance in future projects, such as expected problem areas, expected volume of defects, expected resolution times, etc… They should also support continuous improvement – if you’ve identified problem areas with the application, for example, then more attention should be paid to those areas in the future (e.g. increased design reviews, peer code reviews, additional unit testing, etc…).
One suggested approach is to capture and track the phase the defect was injected. To do this you need to list out the phases of your project (e.g. Requirements, Design, Development, Unit Test, etc…) and for each defect determine when the defect was introduced (injected) into the application. This takes some work, and developers typically won’t want to go to this level of effort (since they are normally the ones that have to make the determination), but it has real value long term. If you determine, for example, that 25% of your defects come from the Requirements phase, then you have real ammo to go back to Project Management and suggest that that particular phase needs more attention in future releases. It is a great tool for quality improvement over the long term.
There’s certain data that every system captures (e.g. id, description, status, creation date, resolution date, severity), but it’s worth spending a bit more time capturing additional information. Note that some defect tracking databases will already have some of these fields present. Here are some candidates worth considering:
If you’re lucky, the bug tracking database/system you’re using will produce the metrics you want. If not, you’ll need some kind of method for cranking those numbers out. Whatever method you use, you need to figure it out prior to the start of testing and work out any kinks in the system. Don’t leave this until the day you need to produce your first status report!
I’m generally never happy with the metrics and graphs that come out of bug tracking databases. I usually setup an export out of the database (check your tool’s documentation – most systems have the capability to export a list of defects) and import it into a spreadsheet, which I use to produce metrics and graphs. Once I get my system working, it’s relatively painless to crank out the daily information I need. I’m working on a generic template which I’ll post here soon. Please keep in mind that I am not advocating using a spreadsheet to manage the defects themselves – unless you’re a team of one, this is impossible.
Snapshots in time are fine, but to truly understand what’s occurring you need to examine how the data is changing over time. Trends help you analyze data against the broader context of testing. Trends can assist in many ways, including:
Not just the immediate project team, but anyone who will be reading the information you publish (e.g. management, project sponsors). Data without guidance on how to interpret it can be dangerous. Let me give you a couple of examples:
A dashboard is simply a summary (usually a page or two) that gives the reader a quick view into the most important aspects of the project. A testing dashboard should include statistics and trends about defect counts, but also:
Sometimes numbers speak for themselves, but sometimes they don’t. While you may know why a specific metric or trend significantly changed, everyone reading your status report or looking at the metrics may not. Think about adding callout boxes to graphs and a sentence or two to status reports described what’s impacting the numbers.
To sum this all up – get organized. Find a system that works for you and stick to it. Here are some things I do that I find helpful:
Published by Brad Kuhn on 28 Nov 2005
Looking to build your testing library or just adding one or two books? Consider the following.
How to Break Software: A Practical Guide to Testing By James A. Whittaker
A quick read (just 174 pages with the appendices), but lots of good info. If you’re looking for something to help you develop a tester’s mentality, this is it. Not everyone can be a good tester – you need the right kind of perspective. Most developers when they do unit testing make sure that their code is working. Most good testers I know want to find out how they can break the code. The book is mostly focused on PC applications – there are sections on UI testing and system interface testing. The book also comes with some software, which to be honest I’ve never tried.
Effective Software Testing: 50 Specific Ways to Improve Your Testing By Elfriede Dustin
This is a reference that is essentially 50 fully documented suggestions/best practices. While some of them are obvious (like #2 “Verify the Requirements”) the author does a good job at explaining the whys and hows behind each tip. The suggestions are grouped together logically (there’s a section on the requirements phase, test planning, test design, test management, etc…), so it’s easy to pick up this book over the life of your project and get specific hints tailored for where you are in the project life-cycle.
Software Testing By Ron Patton
There’s a second edition out now, though I haven’t yet picked it up. This is a great introductory book to software testing. It covers the different forms of testing, how to review documentation and code, test planning, and test case writing. One thing that Ron Patton really focuses on is the role of the tester in software quality: “The goal of a software tester is to find bugs, find them as early as possible, and make sure they get fixed.” Couldn’t say it better myself!
Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing By Rex Black
Second Edition. Chapters include planning, mapping out the test coverage, defect tracking, managing test cases, staffing, and organizational considerations. I enjoy Rex Black’s writing style – it is very down-to-earth with a focus on practical advice. Each chapter ends with a real-life case study and exercise questions. While I think there’s good material for any test manager in this book, it would be particularly helpful for those either new to test management or are setting up a new test system/organization.
Critical Testing Processes: Plan, Prepare, Perform, Perfect By Rex Black
Builds on Black’s previous book. As the title suggests, the book’s focus is on testing processes. You can think of this book as a method for implementing continuous improvement in your testing process. Like his previous book, this one is full of practical advice and tips. I particularly like the planning chapters, and the chapters on reporting defects and producing test metrics. Throughout the book the author illuminates many of the chapters with a hypothetical case study – so you can see how to apply suggestions from the start of test planning all the way through post-test execution process improvement. Very highly recommended.
Quality Software Project Management By Robert T. Futrell, Donald F. Shafer, and Linda I. Shafer
The main audience for this book is Project Managers. Each chapter is well documented with footnotes and has it’s own bibliography, which is quite handy. There are a large number of templates and checklists. The book ties in very well with PMI standards, but is probably a bit more practical than your average PMI project management book (and since it’s on just software, more relevant to the software PM). The book’s strength is tying basic software engineering and software quality assurance concepts in with the process of planning, executing, and controlling a project. The book weighs in at over 1,500 pages, so it may not be exactly what you need – but if your PM doesn’t have a copy, they should.
Software Quality Assurance : From Theory to Implementation By Daniel Galin
This is a college textbook, but has a lot of very practical information. The book does a good job at laying a foundation about quality testing, but I think it’s stronger points are the sections on quality assurance. Looking at establishing programs for quality improvement, software/quality metrics, certification, or standards? Then this is a great addition to your library. There is also a section on organizational alignment for quality assurance.
The Certified Quality Manager Handbook Edited By Duke Okes and Russell T. Westcott
This book is a must if you’re getting certified by ASQ in Quality Management. It’s a handbook, so don’t expect in depth treatment on any single topic, but it’s a great overview of quality management. Use this as a reference to help identify areas that you want to research/learn more about.
Flawless Consulting: A Guide to Getting Your Expertise Used By Peter Block
So what’s this book doing on a test manager’s bookshelf, you may ask. Testing is a funny activity in software development, if you think about. We don’t produce the product and in many cases we don’t have any real say in whether the software goes out the door or not. We should be honest, impartial, and knowledgeable producers of information that others (such as project management and sponsors) utilize – in other words, we are consultants. This book, more than any other on the subject IMO, will help you hone skills so that the information you produce is valuable, pertinent, and used by others.
