Communication

As a test manager, you communicate – a lot. You need to be aware of the different consumers of your information (such as business stakeholders, developers, project managers, etc…) and their different needs. You need to be able to produce both written and verbal communication at different levels. For example, a status report needs to have some sort of executive summary so the readers can quickly understand the testing status, but also have sufficient detail included for those who need it. During status meetings, you’ll be asked to briefly summarize testing activities, but you’ll also need to be prepared to drill down on specifics. One of the key pieces of value the test manager brings to the project is data – and people will quickly assess your ability to provide that data in the form that they need.

Political Agility (Leading Without Authority)

I almost placed this one at the top of the list – it’s that important. Imagine you just got the first release of code from Development and it sucks – completely. Or you’re need the end of a planned test phase but due to quality problems you’re only 50% complete and you run into the key business stakeholder and they ask you how testing is going. Or you tell the project manager that you need an additional week to complete regression testing and they tell you that you can’t have it. As test managers, you don’t produce the code, you don’t write the requirements, you don’t control the schedule – but you rely on others to do these things so you can get your job done.
Your ability to respond to and influence others in a politically astute manner is key to being an effective test manager. For example, simply telling the Development Manager that his/her team’s code sucks and wasn’t unit tested is not going to win you any friends. The manner in which you deliver these kind of messages is important – you need to be tactful, unemotional, and stick to facts. To affect change in other groups without true authority can be a difficult skill to master. I’d recommend reading Dale Carnegie’s How to Win Friends and Influence Others.

Time Management

This is an obvious one, but true nonetheless – especially during test execution. You must be able to use your time wisely enough so you can effectively respond to the critical test system failure, last minute modification to the test iteration plan, the emergency staffing issue, and the weekly status report (all happening simultaneously). Basic time management skills (like delegation, prioritizing, developing processes to deal with routine tasks, etc…) are really essential over the long run. Testing is one of those things that is always stressful, pressed for time, and hectic – and if you can’t manage this over the long run, you’re headed for burn-out.

Technical Ability

A lot of experienced testers came up through the programming ranks and like to dissect code and write testing stubs. Personally, I didn’t start as a programmer – but much of my testing experience has been with packaged software, so extensive programming skills were not really a requirement. Whatever your background, an effective test manager needs enough technical ability to be credible with the test engineers and developers.

Attention to Detail

You can’t be a “big-picture” person and be a great test manager, IMO. Good test managers are expected to know the overall status of testing, details around why planned test cases haven’t been executed, the exact status of major defects, and be able to drill down on any given defect and talk about it. You also need to know enough detail about the requirements and test cases to have confidence that your test team will have the proper test coverage.

Analytical Ability

I’ve written about test metrics before. You know the old saying – you can make a set of statistics look like anything you want it to. Well, to an extent that’s true. You have to be able to dig and really understand what’s driving your numbers. For example, if the open defect count is going down – that’s good, right? Well, maybe – but what if that is combined without a corresponding increase in passed test cases? A possible explanation for that scenario might be the Development team clearing a lot of lower priority defects while leaving high priority/blocking defects open. You’d want to examine the priority level of recently closed defects and the reason test cases are not getting passed.
The ability to truly understand what the numbers are telling, and be able to educate others on the team is a must.

Testing Discipline

The last trait for successful test managers is testing ability. In order to effectively manage and lead your team, you must be intimately familiar with the testing discipline. This includes how to develop a test strategy/plan, how to plan and write test cases, how to use testing tools effectively, how to conduct exploratory testing, and many more – and be able to train your staff.

In this post, I’ll discuss some common techniques for analyzing and identifying solutions – and also provide some links to additional information.

Assumption Analysis

List out all of your assumptions about the problem, and then analyze each of them. Are they all necessary? If it were possible to change one or more assumptions, would a solution to the problem exist? Check out Virtual Salt for more details on this approach.

Cause and Effect Diagrams

Cause and effect diagrams – also called Ishikawa diagrams (after Kaoru Ishikawa) or fishbone diagrams (due to their appearance) – allow you to analyze the causes of a particular problem. You begin by drawing a horizontal line that represents the problem. Then you draw a “spine”, or diagonal line to the left off from the main problem line, that represent factors of the problem. These factors might be systems, people, processes, policies, materials, etc… Then for each identified factor, drill down and draw a vertical line for each possible cause of that factor. Asking “why” is a great technique to uncover possible causes. You can then use additional techniques, like a Pareto chart, to get additional insight into the problem. There’s a good explanation and example at MindTools.

SWOT

SWOT analysis is a strategic analysis approach that analyzes strengths, weaknesses, opportunities, and threats. First – a business objective/desired end state is defined. Then, key strengths (assets/characteristics that can help meet the objective), weaknesses (attributes of the situation that hurt the objective), opportunities (external conditions that can help meet the objective), and threats (external conditions that can hurt the objective) are identified. These are usually visually represented in four quadrants on a single page. The key to using SWOT analysis effectively is to start with a business objective, and then make sure you properly segment out internal conditions (strengths and weaknesses) from external ones (opportunities and threats).
From a testing perspective, SWOT analysis can assist when making strategic decisions, such as whether or not to outsource a portion of the testing effort. There’s a very good article at Wikipedia and some templates over at Businessballs.com.

Affinity Diagrams

Affinity diagrams were developed by Jiro Kawakita and are also known by the KJ method. Affinity diagrams are a good method for organizing the problem space. This method is also good for driving consensus. The process involves writing down ideas on sticky notes during a brainstorming session. Then, without anyone talking, the group organizes all of the notes into related areas. After that, the group can talk again and should discuss the patterns that emerged and develop headings for each collection of notes. Essentially, affinity diagrams help you “chunk” the problem into more manageable pieces.
There is a very good description at ASQ and an Excel add-on available (for purchase) at BaRaN Systems (I haven’t tried this, but it looks interesting).

Risk Analysis

I’ve previously posted about risk identification from a testing perspective and I also have a template available for developing a risk management plan. Risk analysis is a great tool when trying to spend limited resources, since it forces you to identify, prioritize, and plan for risks. You can use this approach in conjunction with decision trees to help, for example, predict how long a test phase might be.
Check out MindTools for more information.

Brainstorming

No doubt you’ve heard of brainstorming – a technique where a group generates a large number of ideas to solve a problem. There are lots of different ways to do brainstorming, but most of the methods share four common traits:

1. A focus on quantity
2. No criticism
3. Encouragement of “out of the box”/unusual ideas
4. Combining and improving on generated ideas

I should note that many people argue that brainstorming generates lower quality ideas that a group working independently. You can head over to the University of North Carolina or Wikipedia for more reading.

Decision Matrix

A decision matrix can come in handy when trying to select from two or more alternative solutions. First – come up with dimensions that you want to use to assess the solution (e.g. cost, ease of implementation, effectiveness, etc…). Then weight each factor – it’s easier if you start off with a given total of points to use (say 100), and then allocate those across each factor based on their importance. Next, assess each solution against each factor – use a numeric scale (such as 1-10). Finally, produce a weighted score (solution’s score for a given factor * that factor’s weight) and sum the results. The solution with the higher score is the better solution. Look at this article on the ASQ website for more details.

There are plenty of other techniques, but hopefully this list will get you started. Check out MindTools and the American Society for Quality for more.

But first let me ask a question about why we do risk identification. Is it possible to release a defect-free product? I suppose it depends on your point of view, but speaking practically, the answer is no. In the real-world we have constraints – limited testers, limited schedule, not enough tools, etc… So it is with this realization – no software can be perfect – that risk identification becomes so critical.

Identifying risks on a project lets us answer the question “where do I spend my limited test resources?” I’d like to suggest the following as sources to use when finding risk potentials.

Previous Defect Reports

If you are on a project that has an associated history (e.g. a previous phase), then perhaps the best place to look is the defect history. Which components had the largest number of defects? Which developers had the biggest share of defects? Is there a previous history of defect fixes causing related new defects (if so, then heavy regression testing is called for)?

Extended Team Members

Talk with the team and get their input. Ask the developers (not just the lead) and get their perspective. Talk with the requirements team, with the PM, with the stakeholders. What are their areas of concern?

Change Requests

Change requests are another good source, particularly ones that occur later in the development process.

Project Assumptions

Project assumptions are a great source for not only identifying risks, but also for designing test scripts. What happens when an assumption doesn’t hold true? Does the application handle it gracefully, or just fail?

Status Reports

Do the status reports reveal certain areas of the application that design or development has had issues with?

Requirements

Requirements volatility is a measure of how frequently requirements change over time. Sometimes, this reflects poorly stated requirements that should probably demand more attention during testing (of course, changes could also be the result of stakeholders not knowing what they want). If your team is using a requirements management tool, then getting a report on volatility is fairly straightforward. Even if you don’t have a tool, talk with the requirements engineers and get their input.

Audits

If you are part of an organization that has regular process audits, then their findings are often valuable.

Lessons Learned

Last, but certainly not least, are lessons learned from previous testing efforts. Best practices calls for these to be collected and documented after each project. If you are not doing this (you should be), you can still talk with people to get historical input.

Not all of us have managed a test phase or team before. I would summarize preparing and running a test project in five groups of activities:

1. Define the Test Strategy
2. Document the Test Plan
3. Prepare the Test Scripts
4. Execute the Test Plan
5. Deliver Metrics and Lessons Learned

Test Strategy

You should start with the test strategy. The strategy document defines:

• Any applicable quality assurance guidelines to be followed
• Overall approach to testing (e.g. use of automated test tools, need for regression testing, etc…)
• Overall responsibilities for testing
• The test phases that will be executed and various information about each phase (e.g. description, timing, roles and responsibilities, etc…)

Some people don’t see the need for an overall test strategy and write test plans instead, but I prefer a summary level document like this one. One advantage to this approach is that you can develop your test strategy fairly early in the project – but test plans generally require that you’re further along in the project life cycle.

The Test Plan

Don’t make the mistake of leaping immediately into the definition of test scripts after you’ve defined your test strategy – you need to flesh out a test plan for each test phase of the project. This includes unit testing (Development should develop this plan). I’ll be publishing a test plan template shortly, so I’ll provide more information in that post, but here’s the type of information the test plan should have (at a minimum):

• Scope (including what won’t be tested)
• Quality risks
• Schedule (including planned test iterations)
• Entry, exit, and stopping criteria
• Test configuration(s)/environment(s)
• Roles and responsibilities
• Defect reporting (including references to documentation on how to use the defect tracking tool)
• Release management
• Test case status and metrics reporting

Test Scripts

Here’s a couple of posts that will help in planning and writing test scripts – the first one is on test case planning and execution, and the second one describes a test script template.

Test Execution

If you’ve put together a good test plan and developed a thorough set of test scripts, test execution is a snap! Well, not really – but a large amount of your total test effort should be spent in preparing for test execution. That way you won’t be swamped when all of the unforeseen emergencies land (and they will).

Test Metrics and Lessons Learned

You should regularly deliver test metrics while you execute each test phase. I’ve written a previous post on test statistics, and I’d suggest you start there for some ideas. Also, at the end of testing don’t forget to put together a lessons learned, which should include what went well/could be improved for the next time. You don’t want to constantly hit the same difficulties over and over, do you?

The defect life-cycle defines what states (or statuses) your defects can reach, and how they move from one state to another. In this article I’ll be demonstrating a fairly standard approach – but it’s an approach I’ve been using for years with a lot of success. I’ll also describe some details that are specific to the software I’m currently using – TestTrack Pro by Seapine Software. Most of what I’ll describe can be leveraged over to the software you’re using, depending on it’s capabities.

First some definitions:

• State – Status of the defect
• Workflow – Definition of how defects move from one state to another – this is usually embedded within your testing software
• Event – Actions that happen with a defect

Grab this picture of my sample workflow – this was created in TestTrack Pro, so you may need to tweak it to work with whatever software you are using.

The first thing you notice is a lot of boxes/rectangles with arrows going all over the place. The boxes are States and the arrows are Events. I’ll describe each one below.

Open

Initial state for defects. Valid actions for Open defects are Fix and Force Close. You would Force Close an item that didn’t require a normal fix, release, and test cycle – e.g. an entry wasn’t actually a defect or an entry resulted in an enhancement request.

Fixed

Once the Development team has fixed, unit tested, and made appropriate documentation updates, the entry should be moved to the Fixed state. Note that Fixed does not mean that the Test team is ready to look at the defect yet.

Ready To Test

Normally during testing there is a process for promoting fixed code into the Test environment and most defects (but not normally all) require this code promotion before the Test team can re-test. This is critical part of the testing process – make sure that the Development Team Lead and you have a very clear understanding on how this promotion occurs, and who moves defect entries from Fixed to Ready To Test once the code promotion occurs. The Event in the sample workflow is called Release To Test – you’ll also notice an arrow from Open leading to the Ready To Test State – not all fixes require a code promotion.

Closed (Verified)

The Development Team has fixed the defect and the Test Team has verified the fix.

Open (Verify Failed)

Sometimes fixes don’t fix the problem, and that’s what this State captures. Many people don’t use this State and instead cycle back to Open, but I prefer to track it in a separate status. I’ve argued in a previous post that the defect pass rate is an important factor in estimating test completion (as well as overall software quality). Maintaining a separate State makes calculating this ratio much easier.

Closed (Fixed)

This State is necessary, but should be used sparingly. Sometimes defects do require a fix, but don’t require a release and re-test cycle. If your software doesn’t have a good audit capability, you may want to consider not having this state – it can be abused.

Open (Re-Opened)

Just what it says. Some test managers prefer to start over again with a new defect – it’s your call.

Using the Defect Life-Cycle

So it should be obvious how you want to utilize the different States and what kind of metrics and reports you’ll need. Here’s a list to start with:

• Developers – Their report should show Open, Open (Verify Failed), and Open (Re-Opened), perhaps with a special report for the Development Lead highlighting the Re-Opened and Verify Failed items
• Testers – Their report shoud show Ready To Test
• Management – Several reports would be appropriate – one highlighting overall volume and status of defects, another highlighting trends, another showing daily activity (# reported, fixed, tested), and another highlighting items of concern

I’ve posted a template for producing test statistics. The template is a follow-up on my previous on 10 Tips for Test Statistics. Drop me a comment and let me know what you think.

Here are some tips for generating and managing test statistics.

Use a Robust Bug Tracking System

This may seem like a given, but don’t leave the selection and installation of a bug tracking system/database to the last minute. If you don’t have one in place already, you’ll need time (and possibly other resources within your organization) to install and setup the system, plus time to tailor it to your specific needs and train the project team. I’ve used numerous tools and don’t really have a preference – there are a ton of places you can go and get people’s opinions on which one to use. Just make sure it supports:

• Multiple users (preferably with support for different access groups)
• Modification to your unique needs (i.e. the ability to add/remove fields and customize values in drop-down fields)
• Attachments

There are some free systems out there (like Bugzilla) and tons of commercial products. My firm is currently using IBM’s Rational Portfolio Manager, and while I’m not wild about the product (it is more of a Project Manager tool), it gets the job done.

Plan Your Metrics Ahead of Time

If you want to produce a metric on mean time to resolution, then you need to capture an open date and a resolved date for each defect. Think about the metrics you want to produce and make sure that the fields and values you are capturing support the metrics you want to generate. Make modifications to your bug tracking system before you begin test – it’s not something you want to be doing during test execution.

Think About Future Projects, Not Just Your Current One

In addition to supporting your current project, test metrics should also assist in estimating future projects and continuous improvement. If you identified X number of defects in project A, and project B has similar characteristics, then it makes sense that you will identify some number near X when testing project B. Information from previous projects should give you some guidance in future projects, such as expected problem areas, expected volume of defects, expected resolution times, etc… They should also support continuous improvement – if you’ve identified problem areas with the application, for example, then more attention should be paid to those areas in the future (e.g. increased design reviews, peer code reviews, additional unit testing, etc…).

One suggested approach is to capture and track the phase the defect was injected. To do this you need to list out the phases of your project (e.g. Requirements, Design, Development, Unit Test, etc…) and for each defect determine when the defect was introduced (injected) into the application. This takes some work, and developers typically won’t want to go to this level of effort (since they are normally the ones that have to make the determination), but it has real value long term. If you determine, for example, that 25% of your defects come from the Requirements phase, then you have real ammo to go back to Project Management and suggest that that particular phase needs more attention in future releases. It is a great tool for quality improvement over the long term.

Consider Capturing Additional Information

There’s certain data that every system captures (e.g. id, description, status, creation date, resolution date, severity), but it’s worth spending a bit more time capturing additional information. Note that some defect tracking databases will already have some of these fields present. Here are some candidates worth considering:

• Subsystem/Component – Tracking the subsystem can help you identify issues in certain aspects of the application that may be caused by inadequate requirements, sub-standard design, immature technology, under-performing developers, etc…
• Root Cause – Like the phase injected field described above, capturing a root cause is a great way to improve the quality of your projects over time.
• Re-Open Flag – Tracking whether a defect was re-opened (or failed test after being fixed) allows you to measure the effectiveness of the bug fix process. If you notice a large percentage of defects being re-opened, then you need to find out why (e.g. developers are not properly unit testing their fixes or the testers are not properly documenting the defects).

Have a Defined System for Producing Metrics

If you’re lucky, the bug tracking database/system you’re using will produce the metrics you want. If not, you’ll need some kind of method for cranking those numbers out. Whatever method you use, you need to figure it out prior to the start of testing and work out any kinks in the system. Don’t leave this until the day you need to produce your first status report!

I’m generally never happy with the metrics and graphs that come out of bug tracking databases. I usually setup an export out of the database (check your tool’s documentation – most systems have the capability to export a list of defects) and import it into a spreadsheet, which I use to produce metrics and graphs. Once I get my system working, it’s relatively painless to crank out the daily information I need. I’m working on a generic template which I’ll post here soon. Please keep in mind that I am not advocating using a spreadsheet to manage the defects themselves – unless you’re a team of one, this is impossible.

Use Trend Data

Snapshots in time are fine, but to truly understand what’s occurring you need to examine how the data is changing over time. Trends help you analyze data against the broader context of testing. Trends can assist in many ways, including:

• Identifying what’s happening to the defect backlog
• Estimating volume (and severity) of un-found defects
• Spotting problems (see my previous article on Using Control Charts in Testing)

Educate the Project Team on Using Metrics

Not just the immediate project team, but anyone who will be reading the information you publish (e.g. management, project sponsors). Data without guidance on how to interpret it can be dangerous. Let me give you a couple of examples:

• You’re half way through a test phase. The number of defects logged each day hasn’t dropped significantly. Should you be concerned? Well, what’s happening with the resolved count? And what is the severity of the defects being logged? If the resolved count is steadily growing (i.e. the team is getting through the backlog of defects) and the severity of defects is dropping, that’s probably not a bad position to be in half-way through the test phase.
• The resolved defects count continues to climb, and the average time to resolution has been dropping recently. Things are going great, right? Well, what if the average days outstanding count is climbing at the same time. What does that tell you? It means that the backlog of defects is increasing – you may want to look at the severity of the defects being resolved. Developers may be cherry-picking easy, lower severity defects to fix (which would decrease the mean time to resolution since these are quick to fix), but at the expense of more difficult, higher severity defects (which results in an increased backlog).

Use a Dashboard

A dashboard is simply a summary (usually a page or two) that gives the reader a quick view into the most important aspects of the project. A testing dashboard should include statistics and trends about defect counts, but also:

• Progress of test scripts/cases
• Schedule and resources used vs. planned
• Top level risks and issues

Qualitatively Describe Metrics

Sometimes numbers speak for themselves, but sometimes they don’t. While you may know why a specific metric or trend significantly changed, everyone reading your status report or looking at the metrics may not. Think about adding callout boxes to graphs and a sentence or two to status reports described what’s impacting the numbers.

Get Organized!

To sum this all up – get organized. Find a system that works for you and stick to it. Here are some things I do that I find helpful:

• Carve out a set time each day to do metric production. For me, I like to get in early in the morning and run the reports from the previous day.
• Make sure you have a backup system in place and operating in case something happens to the defect database. I also do a daily print of the outstanding defect list and keep it in a binder (this has the added advantage of providing a historical snapshot if your defect tracking system can’t provide one).
• Develop a naming convention for spreadsheets, documents, etc… and a central repository so people can quickly find metrics, summaries, and dashboards.

General Testing

How to Break Software: A Practical Guide to Testing By James A. Whittaker
A quick read (just 174 pages with the appendices), but lots of good info. If you’re looking for something to help you develop a tester’s mentality, this is it. Not everyone can be a good tester – you need the right kind of perspective. Most developers when they do unit testing make sure that their code is working. Most good testers I know want to find out how they can break the code. The book is mostly focused on PC applications – there are sections on UI testing and system interface testing. The book also comes with some software, which to be honest I’ve never tried.

Effective Software Testing: 50 Specific Ways to Improve Your Testing By Elfriede Dustin
This is a reference that is essentially 50 fully documented suggestions/best practices. While some of them are obvious (like #2 “Verify the Requirements”) the author does a good job at explaining the whys and hows behind each tip. The suggestions are grouped together logically (there’s a section on the requirements phase, test planning, test design, test management, etc…), so it’s easy to pick up this book over the life of your project and get specific hints tailored for where you are in the project life-cycle.

Software Testing By Ron Patton
There’s a second edition out now, though I haven’t yet picked it up. This is a great introductory book to software testing. It covers the different forms of testing, how to review documentation and code, test planning, and test case writing. One thing that Ron Patton really focuses on is the role of the tester in software quality: “The goal of a software tester is to find bugs, find them as early as possible, and make sure they get fixed.” Couldn’t say it better myself!

Test Management

Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing By Rex Black
Second Edition. Chapters include planning, mapping out the test coverage, defect tracking, managing test cases, staffing, and organizational considerations. I enjoy Rex Black’s writing style – it is very down-to-earth with a focus on practical advice. Each chapter ends with a real-life case study and exercise questions. While I think there’s good material for any test manager in this book, it would be particularly helpful for those either new to test management or are setting up a new test system/organization.

Critical Testing Processes: Plan, Prepare, Perform, Perfect By Rex Black
Builds on Black’s previous book. As the title suggests, the book’s focus is on testing processes. You can think of this book as a method for implementing continuous improvement in your testing process. Like his previous book, this one is full of practical advice and tips. I particularly like the planning chapters, and the chapters on reporting defects and producing test metrics. Throughout the book the author illuminates many of the chapters with a hypothetical case study – so you can see how to apply suggestions from the start of test planning all the way through post-test execution process improvement. Very highly recommended.

Quality Software Project Management By Robert T. Futrell, Donald F. Shafer, and Linda I. Shafer
The main audience for this book is Project Managers. Each chapter is well documented with footnotes and has it’s own bibliography, which is quite handy. There are a large number of templates and checklists. The book ties in very well with PMI standards, but is probably a bit more practical than your average PMI project management book (and since it’s on just software, more relevant to the software PM). The book’s strength is tying basic software engineering and software quality assurance concepts in with the process of planning, executing, and controlling a project. The book weighs in at over 1,500 pages, so it may not be exactly what you need – but if your PM doesn’t have a copy, they should.

Quality Assurance and Certification

Software Quality Assurance : From Theory to Implementation By Daniel Galin
This is a college textbook, but has a lot of very practical information. The book does a good job at laying a foundation about quality testing, but I think it’s stronger points are the sections on quality assurance. Looking at establishing programs for quality improvement, software/quality metrics, certification, or standards? Then this is a great addition to your library. There is also a section on organizational alignment for quality assurance.

The Certified Quality Manager Handbook Edited By Duke Okes and Russell T. Westcott
This book is a must if you’re getting certified by ASQ in Quality Management. It’s a handbook, so don’t expect in depth treatment on any single topic, but it’s a great overview of quality management. Use this as a reference to help identify areas that you want to research/learn more about.

General Skills

Flawless Consulting: A Guide to Getting Your Expertise Used By Peter Block
So what’s this book doing on a test manager’s bookshelf, you may ask. Testing is a funny activity in software development, if you think about. We don’t produce the product and in many cases we don’t have any real say in whether the software goes out the door or not. We should be honest, impartial, and knowledgeable producers of information that others (such as project management and sponsors) utilize – in other words, we are consultants. This book, more than any other on the subject IMO, will help you hone skills so that the information you produce is valuable, pertinent, and used by others.