Carnegie Quality

I recently received a question on a previous post of mine about what test documentation should be pulled together for management when you get assigned to manage a test project. I’ve been asked this before, so I thought the answer warranted a post.

Not all of us have managed a test phase or team before. I would summarize preparing and running a test project in five groups of activities:

1. Define the Test Strategy
2. Document the Test Plan
3. Prepare the Test Scripts
4. Execute the Test Plan
5. Deliver Metrics and Lessons Learned

Test Strategy

You should start with the test strategy. The strategy document defines:

• Any applicable quality assurance guidelines to be followed
• Overall approach to testing (e.g. use of automated test tools, need for regression testing, etc…)
• Overall responsibilities for testing
• The test phases that will be executed and various information about each phase (e.g. description, timing, roles and responsibilities, etc…)

Some people don’t see the need for an overall test strategy and write test plans instead, but I prefer a summary level document like this one. One advantage to this approach is that you can develop your test strategy fairly early in the project – but test plans generally require that you’re further along in the project life cycle.

The Test Plan

Don’t make the mistake of leaping immediately into the definition of test scripts after you’ve defined your test strategy – you need to flesh out a test plan for each test phase of the project. This includes unit testing (Development should develop this plan). I’ll be publishing a test plan template shortly, so I’ll provide more information in that post, but here’s the type of information the test plan should have (at a minimum):

• Scope (including what won’t be tested)
• Quality risks
• Schedule (including planned test iterations)
• Entry, exit, and stopping criteria
• Test configuration(s)/environment(s)
• Roles and responsibilities
• Defect reporting (including references to documentation on how to use the defect tracking tool)
• Release management
• Test case status and metrics reporting

Test Scripts

Here’s a couple of posts that will help in planning and writing test scripts – the first one is on test case planning and execution, and the second one describes a test script template.

Test Execution

If you’ve put together a good test plan and developed a thorough set of test scripts, test execution is a snap! Well, not really – but a large amount of your total test effort should be spent in preparing for test execution. That way you won’t be swamped when all of the unforeseen emergencies land (and they will).

Test Metrics and Lessons Learned

You should regularly deliver test metrics while you execute each test phase. I’ve written a previous post on test statistics, and I’d suggest you start there for some ideas. Also, at the end of testing don’t forget to put together a lessons learned, which should include what went well/could be improved for the next time. You don’t want to constantly hit the same difficulties over and over, do you?

Hopefully no one questions the need for testing to be involved in the requirements definition process. But what should that look like? In a large number of companies I’ve worked with, requirements get tossed over the wall to development and testing – perhaps with some discussion between the requirements team and the development team, but not much with the testing team. I’d like to share some suggestions on how the test team can improve requirements and increase the speed to production.

Validation

Requirements validation is a process whereby various groups sign-off on the requirements and essentially state – “Yes, this is what we want/agree to build.” Typically people think of validating requirements with the business representatives/stakeholders – but really validation should be performed with all project groups, including development and test. The test team should examine each requirement for the following:

• Are the requirements testable? Look for those “system shall be available 99.99% of the year” and other non-testable statements.
• Are the requirements complete? Are there obvious conditions/flows the system could reach that are not detailed? Testers are particularly good at finding these kinds of misses.
• Are the requirements internally consistent? Are there requirements that conflict with each other?
• Are the requirements traceable? Any SRS that you receive that you can’t build a testing traceability matrix from should be sent back for re-work.

Some of the better managed organizations I’ve seen have at least one person from the test team involved during the requirements gathering process itself, not just at the end. This way, they can help steer the direction requirements take to make sure those items above get addressed properly.

Another item that the test team should provide feedback is on the overall testability of the requirements. Yes, requirements should be testable – but what if a certain requirement can be verified, but testing it is beyond the current capabilities of the test team? For example, a requirement may necessitate an automated test tool that the team does not have. Does this mean that the requirement should be stricken? Or raised to Project Management/PMO as an issue? Or kept as is, but with a disclaimer? It depends on how your organization works. By the way, there’s an interesting discussion on Seilevel’s message board on this particular topic.

Education

Another way that testing should be involved in requirements is on-going education. This is a bit different than validation, which is specific to a given project. Education deals with on-going efforts to improve the deliverables from the requirements process. Ideas to consider include:

• Presentations/brown bag lunches/training on how to write testable requirements (chances are you’ll find that this is the biggest problem with the requirements deliverables).
• Sharing information about the current test team’s capabilities (e.g. demoing any test tools that are available). Pointing out any gaps that exist can also be helpful.
• “Get to Know Your Test Team” activities. Look for opportunities to build relationships – it’s much easier to negotiate with people that know rather than strangers.
• Share problems and successes. What kinds of repeating issues are there with requirements? What things are going well?

Hopefully this post has given you food for thought. Remember, it’s a lot easier to fix issues with requirements than it is to deal with defects in the software, so make sure to take some time to address the items I’ve highlighted.

I’ve just posted a template for test case planning and execution. If you don’t have access to a testing tool, you’ll find this helpful in planning and tracking test cases.

You have probably heard both the terms Quality Assurance and Quality Control. But did you know that there is a difference between the two? In this post I’ll do a quick description and contrast of the two. For further ready, I recommend ASQ’s The Certified Quality Manager Handbook.

Quality Control (QC)

Let’s start with Quality Control – if you have a testing background, this is probably what you’ve been doing all along. ASQ describes QC as “the operational techniques and activities used to fulfill requirements for quality.” The QC function is a gate-keeper – it prevents defective products or services from being released. In the software arena, QC is a set of validation and verification (or V&V) activities. Don’t fall into the trap of thinking that this just means software testing – it doesn’t. Verification takes place (or should, at least) at every stage of the software development life-cycle and should include reviews, inspections, and walkthroughs.

Quality Assurance (QA)

QA is a bit harder for some to understand, and is frequently incorrectly used to refer to QC activities. ASQ describes QA as “all the planned and systematic activities implemented within the quality system that can be demonstrated to provide confidence that a product or service will fulfill requirements for quality.” Some key words to consider in that definition are planned and systematic. QA takes place throughout the software development life-cycle, and is not limited to verification activities. QA involves consideration of risk, budget, resource plans, and schedules. The primary objectives of QA are: 1) reduce the rate of deliverables that do not meet quality targets, and 2) reduce the costs of meeting quality objectives. Remember, validation and verification costs are high because they normally require re-work. It is much cheaper to identify problems early in the process, improve your overall processes to reduce non-conformity, and/or consider alternative approaches that reduce risk.

Ok, it’s been awhile since I’ve posted anything. I’ve been swamped with work and life and all that – but I still have plenty of items that I want to get posted here. So let me get back to the swing of things with a post today on managing software defects. As I discussed on a previous post, planning how you are going to manage the testing process ahead of time is critical. One of the key processes you need to have worked out prior to testing is the defect life-cycle you’re going to use.

The defect life-cycle defines what states (or statuses) your defects can reach, and how they move from one state to another. In this article I’ll be demonstrating a fairly standard approach – but it’s an approach I’ve been using for years with a lot of success. I’ll also describe some details that are specific to the software I’m currently using – TestTrack Pro by Seapine Software. Most of what I’ll describe can be leveraged over to the software you’re using, depending on it’s capabities.

First some definitions:

• State – Status of the defect
• Workflow – Definition of how defects move from one state to another – this is usually embedded within your testing software
• Event – Actions that happen with a defect

Grab this picture of my sample workflow – this was created in TestTrack Pro, so you may need to tweak it to work with whatever software you are using.

The first thing you notice is a lot of boxes/rectangles with arrows going all over the place. The boxes are States and the arrows are Events. I’ll describe each one below.

Open

Initial state for defects. Valid actions for Open defects are Fix and Force Close. You would Force Close an item that didn’t require a normal fix, release, and test cycle – e.g. an entry wasn’t actually a defect or an entry resulted in an enhancement request.

Fixed

Once the Development team has fixed, unit tested, and made appropriate documentation updates, the entry should be moved to the Fixed state. Note that Fixed does not mean that the Test team is ready to look at the defect yet.

Ready To Test

Normally during testing there is a process for promoting fixed code into the Test environment and most defects (but not normally all) require this code promotion before the Test team can re-test. This is critical part of the testing process – make sure that the Development Team Lead and you have a very clear understanding on how this promotion occurs, and who moves defect entries from Fixed to Ready To Test once the code promotion occurs. The Event in the sample workflow is called Release To Test – you’ll also notice an arrow from Open leading to the Ready To Test State – not all fixes require a code promotion.

Closed (Verified)

The Development Team has fixed the defect and the Test Team has verified the fix.

Open (Verify Failed)

Sometimes fixes don’t fix the problem, and that’s what this State captures. Many people don’t use this State and instead cycle back to Open, but I prefer to track it in a separate status. I’ve argued in a previous post that the defect pass rate is an important factor in estimating test completion (as well as overall software quality). Maintaining a separate State makes calculating this ratio much easier.

Closed (Fixed)

This State is necessary, but should be used sparingly. Sometimes defects do require a fix, but don’t require a release and re-test cycle. If your software doesn’t have a good audit capability, you may want to consider not having this state – it can be abused.

Open (Re-Opened)

Just what it says. Some test managers prefer to start over again with a new defect – it’s your call.

Using the Defect Life-Cycle

So it should be obvious how you want to utilize the different States and what kind of metrics and reports you’ll need. Here’s a list to start with:

• Developers – Their report should show Open, Open (Verify Failed), and Open (Re-Opened), perhaps with a special report for the Development Lead highlighting the Re-Opened and Verify Failed items
• Testers – Their report shoud show Ready To Test
• Management – Several reports would be appropriate – one highlighting overall volume and status of defects, another highlighting trends, another showing daily activity (# reported, fixed, tested), and another highlighting items of concern