Carnegie Quality

The company I work for recently finished a software project. I was hired right before testing began and we had engaged a systems integrator to do the analysis, design, and development. For reasons still unknown to me, the SI did not document any software requirements. It should come as no surprise that testing didn’t go very well – we often had difficulty validating pieces of functionality (what is this function supposed to do?) and there was a lot of re-work as the project identified missing or mis-implemented requirements.

It goes without saying that the project would have gone a lot smoother, completed closer to the planned delivery date, and produced a higher quality product if requirements had been done. But requirements should be done correctly. A best practice within requirements analysis is the use of the Requirements Traceability Matrix (RTM). In this post, I’ll briefly describe the RTM and explain why you as a tester should be leveraging it.

The Requirements Traceability Matrix

In the broadest terms the RTM is simply a matrix of requirements showing each requirement’s relation to something. It could be the source for the requirement or the design component that describes the implementation of the requirement. Or it could be the test case(s) that covers the requirement. Just think of a spreadsheet with a list of requirements, a unique ID for each requirement, and the corresponding relationship.

Advantages of Using the RTM

The tool itself is very straight forward – it’s the proper use of the tool that provides benefit to your project. The RTM is not something that is used solely by the analysts or testers – it is something that the entire team utilizes.

Ensure Complete Test Coverage

How do you approach defining your test cases? More importantly, how do you know when you’re done? If you use a RTM, you always know the answer to this. As you plan out each test case, decide which requirements each test case will cover – then map those in the RTM. When all requirements are matched against a test case, you’re done (though you’ll probably still want to look at special cases, alternative tests, etc…).

Help Focus Testing

Along with knowing when your test coverage is complete, a RTM can help reduce unnecessary focus on the same set of requirements. You can quickly see in the matrix when a requirement is covered and avoid continuing to assign more test cases to it.

Validation of Requirements Consistency

I’m not suggesting that testers are responsible for validating requirements – we’re not. This should be done during requirements reviews and walkthroughs, and if problems are not caught there, they should certainly be found during technical design. But the reality of life is that these things are often not done as well as they should be, and requirements sometimes conflict with each other. You may have one requirement state that all numeric input fields are integer, and then another one that states that input field #5 on screen #2 is a numeric field with precision to 2 decimals. Using a RTM can help find these kind of problems.

Permit Prioritization

Another question for you – you’ve just been given the system by the development team and have been asked by management to do a quick assessment of quality. How would you approach this? There are probably numerous ways to attack this, but one way is to look at the RTM and see which set of test cases would cover the largest set of requirements and do those first.

Provide Proof of Testing

Your sponsor asks you “How do you know you’re done with testing? Did you test everything?” With a RTM it’s easy to provide the proof.

Help With Future Projects

Finally, if you have a system to collect the right kind of data, you can provide valuable insight to management for future work. Which requirements resulted in the most defects? Which requirements had the most Change Requests performed? Data like this allows management to assess high risk areas of the application and plan against those risks in the future.

For More Information

I hope you’ve found this introduction to the RTM helpful. Most good books on testing will have a word or two to say about the RTM. There is also a good collection of links at this site.

 

I’ve posted a template for documenting test scenarios/scripts. Drop me a comment/email with any questions or feedback you may have.

I’ve posted a template for producing test statistics. The template is a follow-up on my previous on 10 Tips for Test Statistics. Drop me a comment and let me know what you think.

In order to effectively and efficiently manage the testing process, you must be able to generate meaningful and accurate test statistics. You can’t possibly really know how testing is going without the proper metrics. But generating those metrics takes some planning and work.

Here are some tips for generating and managing test statistics.

Use a Robust Bug Tracking System

This may seem like a given, but don’t leave the selection and installation of a bug tracking system/database to the last minute. If you don’t have one in place already, you’ll need time (and possibly other resources within your organization) to install and setup the system, plus time to tailor it to your specific needs and train the project team. I’ve used numerous tools and don’t really have a preference – there are a ton of places you can go and get people’s opinions on which one to use. Just make sure it supports:

• Multiple users (preferably with support for different access groups)
• Modification to your unique needs (i.e. the ability to add/remove fields and customize values in drop-down fields)
• Attachments

There are some free systems out there (like Bugzilla) and tons of commercial products. My firm is currently using IBM’s Rational Portfolio Manager, and while I’m not wild about the product (it is more of a Project Manager tool), it gets the job done.

Plan Your Metrics Ahead of Time

If you want to produce a metric on mean time to resolution, then you need to capture an open date and a resolved date for each defect. Think about the metrics you want to produce and make sure that the fields and values you are capturing support the metrics you want to generate. Make modifications to your bug tracking system before you begin test – it’s not something you want to be doing during test execution.

Think About Future Projects, Not Just Your Current One

In addition to supporting your current project, test metrics should also assist in estimating future projects and continuous improvement. If you identified X number of defects in project A, and project B has similar characteristics, then it makes sense that you will identify some number near X when testing project B. Information from previous projects should give you some guidance in future projects, such as expected problem areas, expected volume of defects, expected resolution times, etc… They should also support continuous improvement – if you’ve identified problem areas with the application, for example, then more attention should be paid to those areas in the future (e.g. increased design reviews, peer code reviews, additional unit testing, etc…).

One suggested approach is to capture and track the phase the defect was injected. To do this you need to list out the phases of your project (e.g. Requirements, Design, Development, Unit Test, etc…) and for each defect determine when the defect was introduced (injected) into the application. This takes some work, and developers typically won’t want to go to this level of effort (since they are normally the ones that have to make the determination), but it has real value long term. If you determine, for example, that 25% of your defects come from the Requirements phase, then you have real ammo to go back to Project Management and suggest that that particular phase needs more attention in future releases. It is a great tool for quality improvement over the long term.

Consider Capturing Additional Information

There’s certain data that every system captures (e.g. id, description, status, creation date, resolution date, severity), but it’s worth spending a bit more time capturing additional information. Note that some defect tracking databases will already have some of these fields present. Here are some candidates worth considering:

• Subsystem/Component – Tracking the subsystem can help you identify issues in certain aspects of the application that may be caused by inadequate requirements, sub-standard design, immature technology, under-performing developers, etc…
• Root Cause – Like the phase injected field described above, capturing a root cause is a great way to improve the quality of your projects over time.
• Re-Open Flag – Tracking whether a defect was re-opened (or failed test after being fixed) allows you to measure the effectiveness of the bug fix process. If you notice a large percentage of defects being re-opened, then you need to find out why (e.g. developers are not properly unit testing their fixes or the testers are not properly documenting the defects).

Have a Defined System for Producing Metrics

If you’re lucky, the bug tracking database/system you’re using will produce the metrics you want. If not, you’ll need some kind of method for cranking those numbers out. Whatever method you use, you need to figure it out prior to the start of testing and work out any kinks in the system. Don’t leave this until the day you need to produce your first status report!

I’m generally never happy with the metrics and graphs that come out of bug tracking databases. I usually setup an export out of the database (check your tool’s documentation – most systems have the capability to export a list of defects) and import it into a spreadsheet, which I use to produce metrics and graphs. Once I get my system working, it’s relatively painless to crank out the daily information I need. I’m working on a generic template which I’ll post here soon. Please keep in mind that I am not advocating using a spreadsheet to manage the defects themselves – unless you’re a team of one, this is impossible.

Use Trend Data

Snapshots in time are fine, but to truly understand what’s occurring you need to examine how the data is changing over time. Trends help you analyze data against the broader context of testing. Trends can assist in many ways, including:

• Identifying what’s happening to the defect backlog
• Estimating volume (and severity) of un-found defects
• Spotting problems (see my previous article on Using Control Charts in Testing)

Educate the Project Team on Using Metrics

Not just the immediate project team, but anyone who will be reading the information you publish (e.g. management, project sponsors). Data without guidance on how to interpret it can be dangerous. Let me give you a couple of examples:

• You’re half way through a test phase. The number of defects logged each day hasn’t dropped significantly. Should you be concerned? Well, what’s happening with the resolved count? And what is the severity of the defects being logged? If the resolved count is steadily growing (i.e. the team is getting through the backlog of defects) and the severity of defects is dropping, that’s probably not a bad position to be in half-way through the test phase.
• The resolved defects count continues to climb, and the average time to resolution has been dropping recently. Things are going great, right? Well, what if the average days outstanding count is climbing at the same time. What does that tell you? It means that the backlog of defects is increasing – you may want to look at the severity of the defects being resolved. Developers may be cherry-picking easy, lower severity defects to fix (which would decrease the mean time to resolution since these are quick to fix), but at the expense of more difficult, higher severity defects (which results in an increased backlog).

Use a Dashboard

A dashboard is simply a summary (usually a page or two) that gives the reader a quick view into the most important aspects of the project. A testing dashboard should include statistics and trends about defect counts, but also:

• Progress of test scripts/cases
• Schedule and resources used vs. planned
• Top level risks and issues

Qualitatively Describe Metrics

Sometimes numbers speak for themselves, but sometimes they don’t. While you may know why a specific metric or trend significantly changed, everyone reading your status report or looking at the metrics may not. Think about adding callout boxes to graphs and a sentence or two to status reports described what’s impacting the numbers.

Get Organized!

To sum this all up – get organized. Find a system that works for you and stick to it. Here are some things I do that I find helpful:

• Carve out a set time each day to do metric production. For me, I like to get in early in the morning and run the reports from the previous day.
• Make sure you have a backup system in place and operating in case something happens to the defect database. I also do a daily print of the outstanding defect list and keep it in a binder (this has the added advantage of providing a historical snapshot if your defect tracking system can’t provide one).
• Develop a naming convention for spreadsheets, documents, etc… and a central repository so people can quickly find metrics, summaries, and dashboards.

Looking to build your testing library or just adding one or two books? Consider the following.

General Testing

How to Break Software: A Practical Guide to Testing By James A. Whittaker
A quick read (just 174 pages with the appendices), but lots of good info. If you’re looking for something to help you develop a tester’s mentality, this is it. Not everyone can be a good tester – you need the right kind of perspective. Most developers when they do unit testing make sure that their code is working. Most good testers I know want to find out how they can break the code. The book is mostly focused on PC applications – there are sections on UI testing and system interface testing. The book also comes with some software, which to be honest I’ve never tried.

Effective Software Testing: 50 Specific Ways to Improve Your Testing By Elfriede Dustin
This is a reference that is essentially 50 fully documented suggestions/best practices. While some of them are obvious (like #2 “Verify the Requirements”) the author does a good job at explaining the whys and hows behind each tip. The suggestions are grouped together logically (there’s a section on the requirements phase, test planning, test design, test management, etc…), so it’s easy to pick up this book over the life of your project and get specific hints tailored for where you are in the project life-cycle.

Software Testing By Ron Patton
There’s a second edition out now, though I haven’t yet picked it up. This is a great introductory book to software testing. It covers the different forms of testing, how to review documentation and code, test planning, and test case writing. One thing that Ron Patton really focuses on is the role of the tester in software quality: “The goal of a software tester is to find bugs, find them as early as possible, and make sure they get fixed.” Couldn’t say it better myself!

Test Management

Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing By Rex Black
Second Edition. Chapters include planning, mapping out the test coverage, defect tracking, managing test cases, staffing, and organizational considerations. I enjoy Rex Black’s writing style – it is very down-to-earth with a focus on practical advice. Each chapter ends with a real-life case study and exercise questions. While I think there’s good material for any test manager in this book, it would be particularly helpful for those either new to test management or are setting up a new test system/organization.

Critical Testing Processes: Plan, Prepare, Perform, Perfect By Rex Black
Builds on Black’s previous book. As the title suggests, the book’s focus is on testing processes. You can think of this book as a method for implementing continuous improvement in your testing process. Like his previous book, this one is full of practical advice and tips. I particularly like the planning chapters, and the chapters on reporting defects and producing test metrics. Throughout the book the author illuminates many of the chapters with a hypothetical case study – so you can see how to apply suggestions from the start of test planning all the way through post-test execution process improvement. Very highly recommended.

Quality Software Project Management By Robert T. Futrell, Donald F. Shafer, and Linda I. Shafer
The main audience for this book is Project Managers. Each chapter is well documented with footnotes and has it’s own bibliography, which is quite handy. There are a large number of templates and checklists. The book ties in very well with PMI standards, but is probably a bit more practical than your average PMI project management book (and since it’s on just software, more relevant to the software PM). The book’s strength is tying basic software engineering and software quality assurance concepts in with the process of planning, executing, and controlling a project. The book weighs in at over 1,500 pages, so it may not be exactly what you need – but if your PM doesn’t have a copy, they should.

Quality Assurance and Certification

Software Quality Assurance : From Theory to Implementation By Daniel Galin
This is a college textbook, but has a lot of very practical information. The book does a good job at laying a foundation about quality testing, but I think it’s stronger points are the sections on quality assurance. Looking at establishing programs for quality improvement, software/quality metrics, certification, or standards? Then this is a great addition to your library. There is also a section on organizational alignment for quality assurance.

The Certified Quality Manager Handbook Edited By Duke Okes and Russell T. Westcott
This book is a must if you’re getting certified by ASQ in Quality Management. It’s a handbook, so don’t expect in depth treatment on any single topic, but it’s a great overview of quality management. Use this as a reference to help identify areas that you want to research/learn more about.

General Skills

Flawless Consulting: A Guide to Getting Your Expertise Used By Peter Block
So what’s this book doing on a test manager’s bookshelf, you may ask. Testing is a funny activity in software development, if you think about. We don’t produce the product and in many cases we don’t have any real say in whether the software goes out the door or not. We should be honest, impartial, and knowledgeable producers of information that others (such as project management and sponsors) utilize – in other words, we are consultants. This book, more than any other on the subject IMO, will help you hone skills so that the information you produce is valuable, pertinent, and used by others.