Carnegie Quality

I’ve just posted a new Test Plan Template. Feedback is always appreciated, so feel free to leave a comment.

I recently received a question on a previous post of mine about what test documentation should be pulled together for management when you get assigned to manage a test project. I’ve been asked this before, so I thought the answer warranted a post.

Not all of us have managed a test phase or team before. I would summarize preparing and running a test project in five groups of activities:

1. Define the Test Strategy
2. Document the Test Plan
3. Prepare the Test Scripts
4. Execute the Test Plan
5. Deliver Metrics and Lessons Learned

Test Strategy

You should start with the test strategy. The strategy document defines:

• Any applicable quality assurance guidelines to be followed
• Overall approach to testing (e.g. use of automated test tools, need for regression testing, etc…)
• Overall responsibilities for testing
• The test phases that will be executed and various information about each phase (e.g. description, timing, roles and responsibilities, etc…)

Some people don’t see the need for an overall test strategy and write test plans instead, but I prefer a summary level document like this one. One advantage to this approach is that you can develop your test strategy fairly early in the project – but test plans generally require that you’re further along in the project life cycle.

The Test Plan

Don’t make the mistake of leaping immediately into the definition of test scripts after you’ve defined your test strategy – you need to flesh out a test plan for each test phase of the project. This includes unit testing (Development should develop this plan). I’ll be publishing a test plan template shortly, so I’ll provide more information in that post, but here’s the type of information the test plan should have (at a minimum):

• Scope (including what won’t be tested)
• Quality risks
• Schedule (including planned test iterations)
• Entry, exit, and stopping criteria
• Test configuration(s)/environment(s)
• Roles and responsibilities
• Defect reporting (including references to documentation on how to use the defect tracking tool)
• Release management
• Test case status and metrics reporting

Test Scripts

Here’s a couple of posts that will help in planning and writing test scripts – the first one is on test case planning and execution, and the second one describes a test script template.

Test Execution

If you’ve put together a good test plan and developed a thorough set of test scripts, test execution is a snap! Well, not really – but a large amount of your total test effort should be spent in preparing for test execution. That way you won’t be swamped when all of the unforeseen emergencies land (and they will).

Test Metrics and Lessons Learned

You should regularly deliver test metrics while you execute each test phase. I’ve written a previous post on test statistics, and I’d suggest you start there for some ideas. Also, at the end of testing don’t forget to put together a lessons learned, which should include what went well/could be improved for the next time. You don’t want to constantly hit the same difficulties over and over, do you?

Hopefully no one questions the need for testing to be involved in the requirements definition process. But what should that look like? In a large number of companies I’ve worked with, requirements get tossed over the wall to development and testing – perhaps with some discussion between the requirements team and the development team, but not much with the testing team. I’d like to share some suggestions on how the test team can improve requirements and increase the speed to production.

Validation

Requirements validation is a process whereby various groups sign-off on the requirements and essentially state – “Yes, this is what we want/agree to build.” Typically people think of validating requirements with the business representatives/stakeholders – but really validation should be performed with all project groups, including development and test. The test team should examine each requirement for the following:

• Are the requirements testable? Look for those “system shall be available 99.99% of the year” and other non-testable statements.
• Are the requirements complete? Are there obvious conditions/flows the system could reach that are not detailed? Testers are particularly good at finding these kinds of misses.
• Are the requirements internally consistent? Are there requirements that conflict with each other?
• Are the requirements traceable? Any SRS that you receive that you can’t build a testing traceability matrix from should be sent back for re-work.

Some of the better managed organizations I’ve seen have at least one person from the test team involved during the requirements gathering process itself, not just at the end. This way, they can help steer the direction requirements take to make sure those items above get addressed properly.

Another item that the test team should provide feedback is on the overall testability of the requirements. Yes, requirements should be testable – but what if a certain requirement can be verified, but testing it is beyond the current capabilities of the test team? For example, a requirement may necessitate an automated test tool that the team does not have. Does this mean that the requirement should be stricken? Or raised to Project Management/PMO as an issue? Or kept as is, but with a disclaimer? It depends on how your organization works. By the way, there’s an interesting discussion on Seilevel’s message board on this particular topic.

Education

Another way that testing should be involved in requirements is on-going education. This is a bit different than validation, which is specific to a given project. Education deals with on-going efforts to improve the deliverables from the requirements process. Ideas to consider include:

• Presentations/brown bag lunches/training on how to write testable requirements (chances are you’ll find that this is the biggest problem with the requirements deliverables).
• Sharing information about the current test team’s capabilities (e.g. demoing any test tools that are available). Pointing out any gaps that exist can also be helpful.
• “Get to Know Your Test Team” activities. Look for opportunities to build relationships – it’s much easier to negotiate with people that know rather than strangers.
• Share problems and successes. What kinds of repeating issues are there with requirements? What things are going well?

Hopefully this post has given you food for thought. Remember, it’s a lot easier to fix issues with requirements than it is to deal with defects in the software, so make sure to take some time to address the items I’ve highlighted.

I’ve just posted a template for test case planning and execution. If you don’t have access to a testing tool, you’ll find this helpful in planning and tracking test cases.