Checkout Testing Defined

Checkout testing has a very specific purpose – that is, to verify that a deployment (usually to a Production environment) was successful.  Checkout testing is concerned with making sure that the deployed system is complete, stable, and in working order.  Checkout testing is not concerned with finding defects within the developed code – hopefully the code went through numerous testing phases (e.g. unit, system, UAT) that identified all of the issues already.

Think about the typical weekend deployment – the technical team does all of the code migration activities, then usually one or two people are responsible for making sure things work.  The last thing you want after a long deployment weekend is to get a call at 8:00am Monday morning from a user complaining that the system doesn’t work, right?  Making sure that doesn’t happen is the purpose behind checkout testing.

Why Document a Test Plan for Checkout Testing?

If you take a look at my template, you’ll see that it is not a large document – so it shouldn’t take a lot of time to produce the document.  Why do it?  Two main reasons:

1. Rather than let your checkout testers just wing it, if you put some time into planning the checkout test you’ll stand a better chance of making sure you don’t miss something.
2. Having a documented checkout test plan will save you time and effort across deployments.  You’ll be able to pull out the previous deployment’s plan when the next release comes.  You can also improve the plan by making sure you perform lessons learned checkpoints at the end of your project (e.g. was something missed during checkout testing?  If so, document it so it gets into the next version of the plan).

Communication

As a test manager, you communicate – a lot. You need to be aware of the different consumers of your information (such as business stakeholders, developers, project managers, etc…) and their different needs. You need to be able to produce both written and verbal communication at different levels. For example, a status report needs to have some sort of executive summary so the readers can quickly understand the testing status, but also have sufficient detail included for those who need it. During status meetings, you’ll be asked to briefly summarize testing activities, but you’ll also need to be prepared to drill down on specifics. One of the key pieces of value the test manager brings to the project is data – and people will quickly assess your ability to provide that data in the form that they need.

Political Agility (Leading Without Authority)

I almost placed this one at the top of the list – it’s that important. Imagine you just got the first release of code from Development and it sucks – completely. Or you’re need the end of a planned test phase but due to quality problems you’re only 50% complete and you run into the key business stakeholder and they ask you how testing is going. Or you tell the project manager that you need an additional week to complete regression testing and they tell you that you can’t have it. As test managers, you don’t produce the code, you don’t write the requirements, you don’t control the schedule – but you rely on others to do these things so you can get your job done.
Your ability to respond to and influence others in a politically astute manner is key to being an effective test manager. For example, simply telling the Development Manager that his/her team’s code sucks and wasn’t unit tested is not going to win you any friends. The manner in which you deliver these kind of messages is important – you need to be tactful, unemotional, and stick to facts. To affect change in other groups without true authority can be a difficult skill to master. I’d recommend reading Dale Carnegie’s How to Win Friends and Influence Others.

Time Management

This is an obvious one, but true nonetheless – especially during test execution. You must be able to use your time wisely enough so you can effectively respond to the critical test system failure, last minute modification to the test iteration plan, the emergency staffing issue, and the weekly status report (all happening simultaneously). Basic time management skills (like delegation, prioritizing, developing processes to deal with routine tasks, etc…) are really essential over the long run. Testing is one of those things that is always stressful, pressed for time, and hectic – and if you can’t manage this over the long run, you’re headed for burn-out.

Technical Ability

A lot of experienced testers came up through the programming ranks and like to dissect code and write testing stubs. Personally, I didn’t start as a programmer – but much of my testing experience has been with packaged software, so extensive programming skills were not really a requirement. Whatever your background, an effective test manager needs enough technical ability to be credible with the test engineers and developers.

Attention to Detail

You can’t be a “big-picture” person and be a great test manager, IMO. Good test managers are expected to know the overall status of testing, details around why planned test cases haven’t been executed, the exact status of major defects, and be able to drill down on any given defect and talk about it. You also need to know enough detail about the requirements and test cases to have confidence that your test team will have the proper test coverage.

Analytical Ability

I’ve written about test metrics before. You know the old saying – you can make a set of statistics look like anything you want it to. Well, to an extent that’s true. You have to be able to dig and really understand what’s driving your numbers. For example, if the open defect count is going down – that’s good, right? Well, maybe – but what if that is combined without a corresponding increase in passed test cases? A possible explanation for that scenario might be the Development team clearing a lot of lower priority defects while leaving high priority/blocking defects open. You’d want to examine the priority level of recently closed defects and the reason test cases are not getting passed.
The ability to truly understand what the numbers are telling, and be able to educate others on the team is a must.

Testing Discipline

The last trait for successful test managers is testing ability. In order to effectively manage and lead your team, you must be intimately familiar with the testing discipline. This includes how to develop a test strategy/plan, how to plan and write test cases, how to use testing tools effectively, how to conduct exploratory testing, and many more – and be able to train your staff.

In this post, I’ll discuss some common techniques for analyzing and identifying solutions – and also provide some links to additional information.

Assumption Analysis

List out all of your assumptions about the problem, and then analyze each of them. Are they all necessary? If it were possible to change one or more assumptions, would a solution to the problem exist? Check out Virtual Salt for more details on this approach.

Cause and Effect Diagrams

Cause and effect diagrams – also called Ishikawa diagrams (after Kaoru Ishikawa) or fishbone diagrams (due to their appearance) – allow you to analyze the causes of a particular problem. You begin by drawing a horizontal line that represents the problem. Then you draw a “spine”, or diagonal line to the left off from the main problem line, that represent factors of the problem. These factors might be systems, people, processes, policies, materials, etc… Then for each identified factor, drill down and draw a vertical line for each possible cause of that factor. Asking “why” is a great technique to uncover possible causes. You can then use additional techniques, like a Pareto chart, to get additional insight into the problem. There’s a good explanation and example at MindTools.

SWOT

SWOT analysis is a strategic analysis approach that analyzes strengths, weaknesses, opportunities, and threats. First – a business objective/desired end state is defined. Then, key strengths (assets/characteristics that can help meet the objective), weaknesses (attributes of the situation that hurt the objective), opportunities (external conditions that can help meet the objective), and threats (external conditions that can hurt the objective) are identified. These are usually visually represented in four quadrants on a single page. The key to using SWOT analysis effectively is to start with a business objective, and then make sure you properly segment out internal conditions (strengths and weaknesses) from external ones (opportunities and threats).
From a testing perspective, SWOT analysis can assist when making strategic decisions, such as whether or not to outsource a portion of the testing effort. There’s a very good article at Wikipedia and some templates over at Businessballs.com.

Affinity Diagrams

Affinity diagrams were developed by Jiro Kawakita and are also known by the KJ method. Affinity diagrams are a good method for organizing the problem space. This method is also good for driving consensus. The process involves writing down ideas on sticky notes during a brainstorming session. Then, without anyone talking, the group organizes all of the notes into related areas. After that, the group can talk again and should discuss the patterns that emerged and develop headings for each collection of notes. Essentially, affinity diagrams help you “chunk” the problem into more manageable pieces.
There is a very good description at ASQ and an Excel add-on available (for purchase) at BaRaN Systems (I haven’t tried this, but it looks interesting).

Risk Analysis

I’ve previously posted about risk identification from a testing perspective and I also have a template available for developing a risk management plan. Risk analysis is a great tool when trying to spend limited resources, since it forces you to identify, prioritize, and plan for risks. You can use this approach in conjunction with decision trees to help, for example, predict how long a test phase might be.
Check out MindTools for more information.

Brainstorming

No doubt you’ve heard of brainstorming – a technique where a group generates a large number of ideas to solve a problem. There are lots of different ways to do brainstorming, but most of the methods share four common traits:

1. A focus on quantity
2. No criticism
3. Encouragement of “out of the box”/unusual ideas
4. Combining and improving on generated ideas

I should note that many people argue that brainstorming generates lower quality ideas that a group working independently. You can head over to the University of North Carolina or Wikipedia for more reading.

Decision Matrix

A decision matrix can come in handy when trying to select from two or more alternative solutions. First – come up with dimensions that you want to use to assess the solution (e.g. cost, ease of implementation, effectiveness, etc…). Then weight each factor – it’s easier if you start off with a given total of points to use (say 100), and then allocate those across each factor based on their importance. Next, assess each solution against each factor – use a numeric scale (such as 1-10). Finally, produce a weighted score (solution’s score for a given factor * that factor’s weight) and sum the results. The solution with the higher score is the better solution. Look at this article on the ASQ website for more details.

There are plenty of other techniques, but hopefully this list will get you started. Check out MindTools and the American Society for Quality for more.

But first let me ask a question about why we do risk identification. Is it possible to release a defect-free product? I suppose it depends on your point of view, but speaking practically, the answer is no. In the real-world we have constraints – limited testers, limited schedule, not enough tools, etc… So it is with this realization – no software can be perfect – that risk identification becomes so critical.

Identifying risks on a project lets us answer the question “where do I spend my limited test resources?” I’d like to suggest the following as sources to use when finding risk potentials.

Previous Defect Reports

If you are on a project that has an associated history (e.g. a previous phase), then perhaps the best place to look is the defect history. Which components had the largest number of defects? Which developers had the biggest share of defects? Is there a previous history of defect fixes causing related new defects (if so, then heavy regression testing is called for)?

Extended Team Members

Talk with the team and get their input. Ask the developers (not just the lead) and get their perspective. Talk with the requirements team, with the PM, with the stakeholders. What are their areas of concern?

Change Requests

Change requests are another good source, particularly ones that occur later in the development process.

Project Assumptions

Project assumptions are a great source for not only identifying risks, but also for designing test scripts. What happens when an assumption doesn’t hold true? Does the application handle it gracefully, or just fail?

Status Reports

Do the status reports reveal certain areas of the application that design or development has had issues with?

Requirements

Requirements volatility is a measure of how frequently requirements change over time. Sometimes, this reflects poorly stated requirements that should probably demand more attention during testing (of course, changes could also be the result of stakeholders not knowing what they want). If your team is using a requirements management tool, then getting a report on volatility is fairly straightforward. Even if you don’t have a tool, talk with the requirements engineers and get their input.

Audits

If you are part of an organization that has regular process audits, then their findings are often valuable.

Lessons Learned

Last, but certainly not least, are lessons learned from previous testing efforts. Best practices calls for these to be collected and documented after each project. If you are not doing this (you should be), you can still talk with people to get historical input.

A couple of thoughts after I read this article:

• In my experience, it’s the role of the developer to assess the reason for the defect, not the tester. I would never go through someone’s code – I wouldn’t know what to look for! Though I understand that others are much more technical in nature and enjoy this.
• While the activities that are described in the article are spot on, I think there is too much emphasis placed on the Quality Control (testing) team. The real owner of this function is Quality Assurance. QA is responsible for making sure there are lessons learned – doesn’t mean they will do all the work, but they need to be driving this.
• I’ve had mixed experiences doing these things myself. Many times I’ve done defect analysis and root cause analysis; documented my findings with charts and graphs; and presented them to project management/development to discover not much enthusiasm. Maybe it’s me, but I hope not! What have others experienced?

Not all of us have managed a test phase or team before. I would summarize preparing and running a test project in five groups of activities:

1. Define the Test Strategy
2. Document the Test Plan
3. Prepare the Test Scripts
4. Execute the Test Plan
5. Deliver Metrics and Lessons Learned

Test Strategy

You should start with the test strategy. The strategy document defines:

• Any applicable quality assurance guidelines to be followed
• Overall approach to testing (e.g. use of automated test tools, need for regression testing, etc…)
• Overall responsibilities for testing
• The test phases that will be executed and various information about each phase (e.g. description, timing, roles and responsibilities, etc…)

Some people don’t see the need for an overall test strategy and write test plans instead, but I prefer a summary level document like this one. One advantage to this approach is that you can develop your test strategy fairly early in the project – but test plans generally require that you’re further along in the project life cycle.

The Test Plan

Don’t make the mistake of leaping immediately into the definition of test scripts after you’ve defined your test strategy – you need to flesh out a test plan for each test phase of the project. This includes unit testing (Development should develop this plan). I’ll be publishing a test plan template shortly, so I’ll provide more information in that post, but here’s the type of information the test plan should have (at a minimum):

• Scope (including what won’t be tested)
• Quality risks
• Schedule (including planned test iterations)
• Entry, exit, and stopping criteria
• Test configuration(s)/environment(s)
• Roles and responsibilities
• Defect reporting (including references to documentation on how to use the defect tracking tool)
• Release management
• Test case status and metrics reporting

Test Scripts

Here’s a couple of posts that will help in planning and writing test scripts – the first one is on test case planning and execution, and the second one describes a test script template.

Test Execution

If you’ve put together a good test plan and developed a thorough set of test scripts, test execution is a snap! Well, not really – but a large amount of your total test effort should be spent in preparing for test execution. That way you won’t be swamped when all of the unforeseen emergencies land (and they will).

Test Metrics and Lessons Learned

You should regularly deliver test metrics while you execute each test phase. I’ve written a previous post on test statistics, and I’d suggest you start there for some ideas. Also, at the end of testing don’t forget to put together a lessons learned, which should include what went well/could be improved for the next time. You don’t want to constantly hit the same difficulties over and over, do you?

Validation

Requirements validation is a process whereby various groups sign-off on the requirements and essentially state – “Yes, this is what we want/agree to build.” Typically people think of validating requirements with the business representatives/stakeholders – but really validation should be performed with all project groups, including development and test. The test team should examine each requirement for the following:

• Are the requirements testable? Look for those “system shall be available 99.99% of the year” and other non-testable statements.
• Are the requirements complete? Are there obvious conditions/flows the system could reach that are not detailed? Testers are particularly good at finding these kinds of misses.
• Are the requirements internally consistent? Are there requirements that conflict with each other?
• Are the requirements traceable? Any SRS that you receive that you can’t build a testing traceability matrix from should be sent back for re-work.

Some of the better managed organizations I’ve seen have at least one person from the test team involved during the requirements gathering process itself, not just at the end. This way, they can help steer the direction requirements take to make sure those items above get addressed properly.

Another item that the test team should provide feedback is on the overall testability of the requirements. Yes, requirements should be testable – but what if a certain requirement can be verified, but testing it is beyond the current capabilities of the test team? For example, a requirement may necessitate an automated test tool that the team does not have. Does this mean that the requirement should be stricken? Or raised to Project Management/PMO as an issue? Or kept as is, but with a disclaimer? It depends on how your organization works. By the way, there’s an interesting discussion on Seilevel’s message board on this particular topic.

Education

Another way that testing should be involved in requirements is on-going education. This is a bit different than validation, which is specific to a given project. Education deals with on-going efforts to improve the deliverables from the requirements process. Ideas to consider include:

• Presentations/brown bag lunches/training on how to write testable requirements (chances are you’ll find that this is the biggest problem with the requirements deliverables).
• Sharing information about the current test team’s capabilities (e.g. demoing any test tools that are available). Pointing out any gaps that exist can also be helpful.
• “Get to Know Your Test Team” activities. Look for opportunities to build relationships – it’s much easier to negotiate with people that know rather than strangers.
• Share problems and successes. What kinds of repeating issues are there with requirements? What things are going well?

Hopefully this post has given you food for thought. Remember, it’s a lot easier to fix issues with requirements than it is to deal with defects in the software, so make sure to take some time to address the items I’ve highlighted.