Newsletter Archives

The following are archived Testing Treats e-Newsletters from the period of 2001 to 2005.

April 2005, by Trevor Atkins

One of the foundation processes in any company that produces software is the defect lifecycle. It is primarily this process that describes how Development and Testing interact around an issue or defect report.

There is typically an emphasis on how setting and tracking of the severity and/or priority of a defect is done. That rating ties to the likelihood of the defect being encountered in the field and the impact or cost of that encounter - Clearly describing a focus on reducing External Failures as they relate to the Total Cost of Quality within the organization. This is certainly an important area to look for and implement improvements given the potential for high return on investment.

However, the vision of a test manager must look for potential improvements in all areas included in the formula of Total Cost of Quality (Continuous Quality Improvement and Outsourcing) and the definition of interfaces between the different organizational groups and the supporting processes. The test manager must also look to driving the capability of the test team and working with the individuals within that team to be their most effective.

Definition of a Defect

"A software error is present when the program does not do what its end user reasonably expects it to do." (Myers, 1976).

"The extent to which a program has bugs is measured by the extent to which it fails to be useful. This is a fundamentally human measure." (Beizer, 1984.).

Attributes of a Defect

There are many attributes that can be ascribed to a defect that are usable in classifying, organizing, and analyzing the associated issue. Aside from a unique Identifier (DefectID), a Description of the issue with Reproduction Steps, and Expected and Actual Results, a defect report might have some of the following:

When including these attributes in the recording of defects, and as part of your defect lifecycle, you can leverage the information to make observations and draw conclusions, typically via metrics (Establishing Effective Metrics).

Using Defect Resolution

One of the classifications that can be applied to a defect would be its resolution.

Not all defects are simply fixed by development. Developers may resolve a defect as: not a bug, by design, not reproducible, duplicate bug, etc. as the reason for moving the defect out of their queues.

Cem Kaner suggests in "How To Win Friends, And Stomp Bugs" the following list of choices for the resolution of a defect in your defect database:

• Fixed: the programmer says it's fixed. Now you should check it.
• Cannot reproduce: The programmer can't make the failure happen. Add details, and notify the programmer.
• Deferred: It's a bug, but we'll fix it later.
• As Designed: The program works as it's supposed to.
• Need Info: The programmer needs more info from you.
• Duplicate: This is just a repeat of another bug report (cross reference it on this report.).
• Withdrawn: The tester who reported this bug is withdrawing the report.

The resolution field can contain other options in addition to the above, such as Enhancement, Spec Issue, Not Implemented, Deferred, and Third Party.

In this field, you can capture much of the philosophy or underlying meaning of the defect resolution process. The core of this philosophy should center on getting a more detailed picture of the defect counts and on allowing an analysis of that picture for more accurate and useful metrics (without providing too many choices).

For example if a defect is given the resolution:

• "Fixed" implies that there really was a problem in the code and it has been addressed now.
• "As Designed" implies that the tester may not have the latest information about the functionality OR may not have the necessary understanding of the product.
• "Enhancement" implies that the tester has not found a defect per se, but that the issue is a new feature or feature modification request. In other words this is not a defect but it has been implemented in the current release (as opposed to those that have been Deferred). This information is valuable for the future, as these records can then be distinguished from the others for easy collection and inclusion in the requirements document and help files.
• "Cannot Reproduce" implies that there is not enough information in the report for the developer to be able to reproduce the defect and that the tester needs to clarify/add information or Withdraw the defect. There may be a hardware setup or set-up preconditions required for seeing this defect that needs to be added to the report, or pointed out to the developer if already in the report (eg: the build number or environment information). This resolution should appear only as a transitory value before the true resolution is set.

In each of the examples above, you could come to a similar view if you were looking at just the surface of the defects. However, your experience no doubt also contains many instances (such as deadline pressure) where some resolutions are sometimes inappropriately used as a way for developers to clear out their queues. In these cases it often becomes the job of the tester, through the mechanism of the defect lifecycle, to make sure the defects get to the right audience and aren't simply let go.

"Need More Info" and "Cannot Reproduce" are examples of resolutions that can create a lot of churn between the developer and the tester. Examination of how many defects get these resolution and the reasons why can provide good insight into training opportunities within the team to reduce this rework. Improvements to the application or tools to help with diagnosis of problems may also suggest themselves.

"As Designed" may be a defect that should not have been logged as implied above. But what if the design is flawed? Referring back to the definition of a defect can be helpful in deciding the next step.

A defined and visible defect lifecycle process provides supports for improved defect resolution. For example, if a defect is resolved "As Designed" or "Deferred" perhaps that defect is then assigned to the business analyst responsible for the functional area to confirm before it goes back to the tester for review, or it might even be escalated to the product manager.

"Duplicate" defects could indicate a higher likelihood of the defect to be encountered, potentially poor organization in terms of resource effort overlap, maybe poor processes in terms of making sure the majority of duplicates are caught before going to development, or even poor training in terms of the resources doing the work.

This is another example of where the defect lifecycle can help in making sure there is stronger chance to reduce the effort spent on duplicate defects. Before assigning a defect to a developer, perhaps the tester reviews all currently open defects (or searches for similar defects via keywords). If it is determined that the defect is a duplicate, the tester then may add any additional information to the existing defect or if the differences are of a greater degree, log a new defect record and relate the two together. Following a similar process to this may mean that duplicate defects are still getting logged, however there is now a valid process behind how they get logged and the testers avoid creating extra work for development and possibly looking like they just aren't taking enough care in their work.

Summary

Just as defect reports provide valuable insight into the types of common classes or types of defects that have occurred in the past, analysis of other attributes can provide useful information for improvement. Without specific tracking of defect resolutions, the true defect find rate and defect clustering in the code is obscured by duplicates, not repros, by designs, and enhancement requests.

Recording and analyzing this information helps ensure you are able to investigate and address the root causes of these Quality Costs. An adaptive approach to testing processes, communication and training goes a long way to show that you have a strong and capable test team. Including a resolution attribute as part of your defect database, and more specifically within the definition of your defect lifecycle, can help achieve this.

December 2004, by Trevor Atkins

Effective defect tracking is a critical activity in any software development project, and having the right tool to do the job is just as critical.

A defect tracking system is a tool solution intended for: the tracking of project issues, defects, or bugs. The tool provides the efficient mechanisms for:

• Recording and communicating such issues to the various team members,
• Tracking the state of each issue as it moves through the defect lifecycle, and
• For generating various reports on the data associated with the set of collected issues for further analysis and interpretation.

In "New Project? Where are the Templates?", it was mentioned that a guiding principle in the software industry, considering the wide range of project scope and constraints, is to "use the processes and tools appropriate to the size, complexity, and impact of your project". This is certainly true when selecting the right defect tracking solution for your team.

There are literally dozens of publicly available defect tracking tools to choose from. A total of eighty-eight of them are listed on Danny Faught's www.testingfaqs.org web site.

While it is still possible to track defects by way of email or even on paper, off-the-shelf solutions have the common intent of trying to: accelerate defect resolution; generally improve project organization and resource planning; promote communication within the project team; and increase transparency of and track quality levels for every functional area throughout the project. All reasons to invest some time in looking at the available options for capable tool-based solutions.

Steps to Making the Right Choice

A decision has been made to assess potential solutions, either because you don't have one or the current solution is not meeting your present or anticipated needs. The following steps outline a process that you can use as a starting place to undertaking an objective tool selection.

Getting Approval for the Assessment

Selecting a defect tracking tool is not necessarily a simple matter, and it will require time and potentially other resources from the organization to complete. A well described ROI proposal to management will go a long way to getting their buy-in to this project.

In a document of only a few pages describe; what is the anticipated ROI of getting a new tool, the major steps in the selection process, and why it is important to undertake this process as part of choosing the defect tracking tool. You will also want to consider these questions: what are the quantifiable goals that the team hopes to achieve with the tool; and are these benefits something that can be measured?

Quantifying these benefits (and the inability of any current solution to provide these benefits) is important to getting through the first gate of determining how much a Defect Tracking solution is worth to the team, and getting a budget defined for the new tool, before you start looking.

Cost will be a big factor and how much you can invest will have a significant impact on which tools you will be able to consider as your solution. Note that the budget must include the actual licenses as well as the assessment itself, and costs for any future training and implementation for the selected tool.

As noted in "Does ROI Matter To You?" by Wolfgang Strigel, ROI is a widely used approach for measuring the value of a new and improved process or product technology. For further information on the ROI calculations that you can apply, refer to "Practical Metrics and Models for Return on Investment" by David F. Rico.

Document the Process of Assessment

Make it clear and visible in a document what steps will be undertaken as part of the selection process. This will avoid confusion or misunderstandings as to any decision gates, short-list criteria, and possible reasons for choices made or delays in the process. You can use this article as a starting place for the outline of this document.

Determine the Method for Evaluation

Define how you are going to measure each tool against the needs of the team and against each other. The solution being assessed needs to be describable in terms of how it is better or worse than an alternative solution and the reasons why need to be recorded in an objective manner.

A sample method of evaluation may be to simply give a rating from 0-10 for each functional requirement where: 0-2 = non-existent or unusable; 3-5 = present but not useful in current form; 6-8 = present but requires configuration or changes; 9-10 = present with little to no configuration or changes needed.

Requirements could be enumerated and grouped into the following:

1. Critical Requirements: List and describe the critical requirements and why they are critical to your company.
2. Functional Requirements: List and describe what functionality or abilities the tool must have.
3. Non-Functional Requirements: List and describe what constraints (cost, environment, quality, etc) the tool must meet or comply with.

The following is a sample of what this section of the assessment document might look like:

At a minimum, you will need to prioritize the requirements so that you can measure each tool against what is most important to your team. In "Evaluating Tools", Elisabeth Hendrickson recommends a face-to-face meeting as a good forum for prioritizing:

• Invite everyone with a say in the tool selection decision.
• Post a list of the requirements, printed large enough to be read from a distance, on the wall or whiteboard.
• Everyone at the meeting gets three votes. (If you have a very large number of requirements, you may want to give everyone five votes instead of three.)
• Each person may cast his or her votes in any combination: one at a time, or multiple votes for a particularly important requirement.
• At the end of the meeting, tally the votes and the requirements are now prioritized.

Determine the Needs from Stakeholders

One of the most important steps in the tool selection process is to involve representatives of the various groups of stakeholders in enumerating the needs for the solution. It is easy to determine that testers and developers need to have their feature and workflow requirements met. But team leads, project management, technical support and others may all have inputs or need outputs from the Defect Tracking system.

Of course not all functionality is critical - there are "nice to have's" and "maybe in the future" types of features from each group of stakeholders. Remember to look at both your process and technical needs for requirements. Also, don't be afraid to challenge or rework the existing processes at this time - it is a great opportunity to improve and strengthen your processes (note: better doesn't necessarily mean more). In the end the tool that you choose should support (and perhaps guide) your process, but not impose one of its own.

Re-define Needs in a Form for Evaluation

Although a need may be stated easily enough by a stakeholder, it may not be expressed in a measurable manner by the person doing the evaluation of the candidate tools. Similar to business requirements in development of a new software product, these statements express the needs for which the application must provide a solution. But, in order to implement the business requirement accurately, the actual functional requirements need to be enumerated, scoped, understood, reviewed, and signed-off. It is the same case when assessing Off-The-Shelf solutions.

An example of such an expressed need might be that "the tool must be compatible with our development environment". If this need was expanded to describe exactly how the tool is expected to be compatible by listing specific operating systems, development tools, or other third-party software with which the tool must integrate, then the evaluation of this need can be performed in a much more systematic and objective manner.

Other examples of requirements that are difficult to objectively evaluate and use to compare two solutions might be:

• "Is completely customizable"
• "Has an intuitive interface"
• "Is reliable"

Select Tools for Detailed In-House Evaluation

In the case of defect tracking systems, your preliminary research will uncover a large listing of possible solutions. The critical requirements should be used to limit those that make it past the point where they will be even included in the first cut. This research is typically "hands-off" and will be used to finalize the first cut against initial criteria. Remember, any tool you can eliminate at this point will allow you to spend more time on potentially better matches in the next steps of the assessment process.

In determining what your choices are, you will want to search the Web for vendor websites and relevant lists of tools (such as www.testingfaqs.org), read and participate in forums or newsgroups, possibly attend tradeshows and conferences, and ask co-workers and colleagues in other companies about tools they have experienced. If you work for a large company, ask other divisions what they use (maybe they even did a similar assessment already).

Note: During the assessment process, you may find you need to add or modify your requirements or constraints (perhaps even for budget or time for the assessment itself) in order to arrive at the best choice. Remember to keep such changes and the reasons why, visible to the stakeholder representatives and to those that approve the budget.

Obtain Demos of Top (3) Tools

Prior to deciding on the tools that you want to have demos for, talk with the vendor's sales people.

Elisabeth Hendrickson warns in her paper that when you ask the sales representatives up front if their tool meets the requirements you have, it's very likely that they will respond positively. The important part of this conversation is not when the sales person says, "We can do that." It is when the sales person says, ".and this is how."

It is also reasonable to ask the sales person how their tool compares to their competition:

• How can I compare your product with other products on the market?
• Under what situations is your tool the best choice?
• Under what situations is your tool probably not the best choice?
• What features differentiate your tool from the competition?
• What don't you like about your tool?

Again, you will want to reduce the number of tools you proceed with to the actual demo step. It is more than likely that you do not have all the time you would need to look at every tool in sufficient detail, so it is better to examine a much smaller group than to sacrifice the depth of the evaluation by looking at a larger group of candidate tools.

During the demos, the objective is to get the selected tool vendors to perform a demonstration of their tool to stakeholder representatives, either in person or on-line, where they prove their earlier claims on how their tool matches all your criteria.

Perform Detailed In-House Evaluation in a Practical Environment

After the demo, obtain evaluation versions of the successful tool(s) for further evaluation on your own to identify potentially hidden problems or irritating behaviours not uncovered in the demo, or to investigate the concerns of the stakeholder representatives expressed after the demo.

Try to arrange to use the tool(s) on an actual project if possible so that the users of the system have an opportunity to experience the solution in a real-world situation and offer feedback. If this is not possible, assign resources to use the evaluation copies of the tool(s) in a simulated project environment.

During this stage of the evaluation, make sure to perform the various activities of the entire process (not just the day-to-day) that your team intends to follow.

Final Selection is Made - Roll-out the New Tool

When rolling out the new tool to your team, remember to make time for training the various stakeholders in how to apply the tool and how it addresses their needs. Also, start collecting data immediately that you can use to measure how well the tool is working in your project environments. This will allow you to track metrics over time and be able to adjust course sooner than later if anything unforeseen begins to develop in implementation and on-going usage.

For more information on selecting tools for your organization and defect tracking tools in particular refer to "Tracking Down a Defect Management Tool" by Hung Quoc Nguyen and "Evaluating Tools" by Elisabeth Hendrickson. The above selection process has been adapted from the tool assessment process described in "Introduction to Practical Test Automation", a public course available through UBC Continuing Studies as part of the Software Engineering Certificate - Quality Assurance and Testing Track (http://www.tech.ubc.ca/softeng/).

July 2004, by Trevor Atkins

In the fast-paced changing world of software product development there is a continuous challenge to document the expectations for the system and its internally and externally facing behaviours. Requirements often suffer because of the challenges of keeping up with an iterative project life-cycle, evolving product scope, and uncertain or changing GUI/Screens.

However, the need remains for all stakeholders to optimize agreement, minimize risk, and minimize rework costs. From a tester's point of view this translates in part into how test coverage of the system can be assured and made visible to all the stakeholders?

In "Testing Without Requirements", we suggested using checklists, matrices, and user scenarios as ways to approach testing when requirements are non-existent, not complete, or not ready at the time testing needs to start.

Even when you have minimal or out-of-date requirements, you can use different methods to help you rapidly define the application, describe its functions and derive an efficient plan to drive your testing effort.

A first step in developing these tools is to think in pictures.

"Imagery is the most fundamental language we have. Everything you do the mind processes through images," says Dennis Gersten, M.D., a San Diego psychiatrist and publisher of Atlantis, a bi-monthly imagery newsletter.

Benefits of creating User Scenarios / Use Cases:

• Easy for the owner of the functionality to tell/draw the story about how it is supposed to work.
• System entities and user types are identified.
• Allows for easy review and ability to fill in the gaps or update as things change.
• Provides early 'testing' or validation of architecture, design, and working demos.
• Provides systematic step-by-step description of the systems' services.
• Easy to expand the steps into individual test cases as time permits.

User scenarios quickly provide a clearer picture of what the customer is expecting the product to accomplish. Employing these user scenarios can reduce ambiguity and vagueness in the development process and can, in turn, be used to create very specific test cases to validate the functionality, boundaries, and error handling of a program.

And, every picture tells a story and stories or scenarios form a basis for testing. Using diagrams can be very effective to visualize the software, not only for the tester but for the whole project team.

Creating user scenarios/use cases can be kick-started by simply drawing a flowchart of the basic and alternate flows through the system. This exercise rapidly identifies the areas for testing including outstanding questions or design issues before you start.

In her article "A Picture's Worth a Thousand Words", Elizabeth Hendrickson notes, "Pictures can pack a great deal of information into a small space. They help us to see connections that mere words cannot."

The Unified Modeling Language (UML), which is a standard language for specifying, visualizing, constructing, and documenting the artifacts of software systems can be employed to help provide these pictures. However, there are many less formal types of notations you can use to put together different simple diagrams, such as activity flow charts, data flow diagrams, state diagrams, and sequence diagrams that can be just as useful for meeting your project needs.

As long as you can achieve the goal to obtain enough information to help you with the task of generating comprehensive test cases ideas, it doesn't matter what notation you use. Start with a basic diagram, depicting the main modules of the system and when, why, and how they interact; from there, you can create more detailed diagrams for each module.

What should be in the initial picture? The very basic information you have. Is it a client-server application? Is it web-based? Is there a database? What are the major tasks the system is supposed to perform?

You have to focus on how the system behaves. End users can help define user scenarios (or use cases) in a diagram format, providing details of the system that will help you not only understand what the client is expecting but also will allow you to validate the diagrams previously drawn.

Describing the tasks and subtasks in detail will provide test scenarios and analyzing the relationships among the modules will help determine the important inputs for the overall testing strategy.

Flow Charts

A flow chart is commonly seen as a pictorial representation describing a process, defining the logical steps including decision points and activities. Flow charts are useful for defining the paths you want to verify or to force into an error condition.

Flow charts can take different forms, such as top-down flow chart, detailed flow chart, workflow diagrams, and deployment diagrams. Each of the different types of flow charts provides a different view or aspect to a process or task. Flow charts provide an excellent form of documentation for a process, and quite often are useful when examining how various steps in a process work together.

State Diagrams

Another option to capture the software behaviour is the use of state diagrams. State diagrams are used to describe the behaviour of a system. State diagrams describe all of the possible states of an object as events occur and the conditions for transition between those states. The basic elements of a state diagram are rounded boxes representing the state of the object and arrows indicating the transition to the next state. The activity section of the state symbol depicts what activities the object will be doing while it is in that state.

All state diagrams begin with an initial state of the object. This is the state of the object when it is created. After the initial state the object begins changing states. Transition conditions based on the activities determine the next state of the object.

Summary

Flowcharts and state diagrams provide similar and at times complementary methods for visualizing, or picturing, the core information to be captured in a user scenario or use case. Throughout the process of creating these diagrams, test case ideas will come to the fore to be rapidly captured for later detailing.

Time well-spent to better understand the software to be implemented and tested not only improves your actual testing activities, but also helps improve the organization's understanding of the product, and thereby can significantly improve the product as a whole.

May 2004, by Trevor Atkins

The biggest challenge in establishing an effective metrics programme is not the formulas, statistics, and complex analysis that are often associated with metrics. Rather, the difficulty lies in determining which metrics provide valuable information to the project and/or organization, and which procedures are most efficient for collecting and applying these metrics.

IEEE Standard for a Software Quality Metrics Methodology IEEE Std 1061-1998 (Revision of IEEE Std 1061-1992) relates software quality to metrics in the following: "Software quality is the degree to which software possesses a desired combination of attributes. This desired combination of attributes shall be clearly defined; otherwise, assessment of quality is left to intuition. For the purpose of this standard, defining software quality for a system is equivalent to defining a list of software quality attributes required for that system. In order to measure the software quality attributes, an appropriate set of software metrics shall be identified."

It is important to be aware that a common misapplication of software metrics is to use them to measure team members productivity against an industry standard. Such comparisons do not earn support for the metrics programme and in fact are likely to cause resentment among the project staff. A metrics programme must focus on much more than productivity measures.

Software metrics are used to quantify software, software development resources, and/or the software development process. Consider these areas of software development that can benefit from inclusion in a planned metrics programme:

• Product quality
• Product performance
• Schedule and progress
• Resources and cost
• Development process

Goals of a Metrics Programme

A metrics methodology for measuring quality allows an organization to:

• Identify and increase awareness of quality requirements and goals.
• Provide a quantitative basis for evaluating and making decisions about software quality in a timely manner.
• Increase customer satisfaction by predicting and then quantifying the quality of the software before it is delivered.
• Reduce software life cycle costs by improving process effectiveness and customer satisfaction.
• Provide feedback on the metrics programme itself and validate the set of metrics being tracked.

Defining the Metrics Programme Framework

The key to the effective software metrics within an organization is to prepare a plan describing how metrics will be used to meet strategic management goals.

The first component of a metrics programme is a framework that describes the metrics to be collected, how to collect the data, and how to apply the results of analysis.

• A software quality metrics framework hierarchy begins with the establishment of quality requirements and quality goals.
• Then, by the assignment of various quality factors, the project team outlines the definitions for each of the quality requirements.
• Next, direct metrics for each quality requirements are obtained by decomposing each quality factor into measurable attributes. The direct metrics are concrete attributes that provide more useful definitions than quality factors to analysts, designers, programmers, testers, and maintainers.

The decomposition of quality factors into direct metrics facilitates objective communication between management and technical personnel regarding the quality objectives.

Keep the following questions in mind when considering the direct metric for each quality factor and its quality requirement or goal:

• What is this metric supposed to tell us?
• What is the theoretical relationship between the characteristic or attribute to be measured and the measurements being taken?
• Are you taking these particular measurements because they're the right ones or because they're convenient?

Beware: Often there is a lack of relationship between the metrics and what we want to measure. This makes the metric gathering process difficult and drawing valid conclusions improbable.

Example Metric and Interpretation

A sample metric that should be easy to gather from the Defect Database would be the number of existing Defects versus their Status over a series of Builds.

The corresponding abbreviated information table for this metric would be as follows:

-adapted from IEEE Standard for a Software Quality Metrics Methodology

Summary

With a number of well-defined metrics measured and recorded over time, the subjectivity of future estimates and software evaluations is greatly reduced. The metrics provide a firm quantitative basis for decision-making.

As just one example, if you knew that in past projects of certain size and duration that the testing effort consisted of X hours with Y test cases, this information would provide a starting point for estimates in future projects. Of course metrics do not eliminate the need for human judgment in software evaluations; they only provide a starting point for such an estimate.

April 2004, by Trevor Atkins

It is a common theme in software projects and testing in particular that there is never enough time to do all that you need to do. Given the limited time that you have available, how can you know that you did the best job testing? You know there are always defects left unfound when the application is released. For Testing, the objective is to minimize risks by improving product quality, and this is done in part by constructing a specific set of test cases to put the application through its paces and more.

IEEE Standard 610 (1990) defines a test case as:

1. A set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.
2. (IEEE Std 829-1983) Documentation specifying inputs, predicted results, and a set of execution conditions for a test item.

Of course you will find it difficult to execute all your test cases on each build of the application during the project lifecycle. But how will you know which test cases must be executed for each build, what should be executed and what could be executed if you have time?

Prioritize Your Test Cases

Your application doesn't have to be perfect; but it does need to meet your intended customer's requirements and expectations. To understand the expectations for your project you need to determine what is important about your application, what are the goals and what are the risks.

Sue Bartlett discusses this exercise in detail in "How to Find the Level of Quality Your Sponsor Wants". She comments in that article that: "When we do communicate quality goals ahead of the detailed planning, design or coding, we have a better chance to avoid a quality mismatch at the end. That means meeting schedules, covering costs, and making a profit will have a better chance of success."

For the purposes of test planning, the organization and scheduling of your test cases for test execution in the context of your project's build schedule will help achieve these goals. As part of this organization, we are concerned with the prioritization of individual test cases. Grouping your test cases by priority will help you to decide what is to be tested for each type of build and therefore how much time is needed. If you have a limited amount of time, you can see what will fit.

Ross Collard in "Use Case Testing" states that: "the top 10% to 15% of the test cases uncover 75% to 90% of the significant defects."

Test case prioritization will help make sure that these top 10% to 15% of test cases are identified.

How To Prioritize Test Cases

How many times have you looked at your test cases and were able to easily pick out a small subset that are the most important? That answer is probably not often. It is really difficult to stop thinking that "all of these are equally important".

When it comes to test cases, assigning a priority is not easy and is not necessarily static for the duration of the project. However, we can get started by constructing an example prioritization process to address the first-cut of prioritizing the test cases.

Let us assume that you have just finished creating your test cases from your functional specifications, use cases, and other sources of information on the intended behaviours and capabilities of your application. Now it is time to assign each test case a priority.

Test Case Priorities

First, you must decide what your types of priorities are and what they imply. For our purposes we will begin with an assumption that there is a parallel between the severity of a defect that we might find and the priority of the corresponding test case.

1 - Build Verification Tests (BVTs): Also known as "smoke tests" are the set of test cases you want to run first to determine if a given build is even testable. If you cannot access each functional area or perform essential actions that large groups of other test cases depend on, then there is no point in attempting any of those other tests before performing this first test case, as they would most certainly fail.

2 - Highs: These are the set of test cases that are executed the most often to ensure functionality is stable, intended behaviours and capabilities are working, and important error and boundaries are tested.

3 - Mediums: This is where the testing of a given functional area or feature is to get more detailed, and the majority of aspects for the function are examined including boundary, error and configuration tests.

4 - Lows: This is where the least frequently executed tests are grouped. This doesn't mean that the tests are not important but just that they are not run often in the life of the project - such as GUI, Error Messages, Usability, Stress, and Performance tests.

We have chosen to group test cases into one of four categories: BVTs, Highs, Mediums, and Lows. The trick now is to figure out which test cases belong to which priority. After all, the priority will indicate which test cases are expected to be executed more often and which are not.

How To Go About Prioritizing

1) Arbitrary Assignment: These first three steps will leave you with an arbitrary grouping of the test cases, based on the idea that if you don't have enough time to test at least make sure all the product requirements have been confirmed to do what they are supposed to under assumed good conditions. If you stop to think about what each test case is testing they all become important too, so just:

1. Label all your Functional Verification (or Happy Path) tests as High Priority.
2. Label all your Error and Boundary or Validation tests as Medium Priority.
3. Label all your Non-Functional Verification tests such as Performance and Usability as Low Priority.

2) Promotion and Demotion: Not all the functional tests are as important as each other and the same is true for Boundary and Non-Functional Tests. Think about the importance of the test and how often you would want to check this functionality relative to others of the same priority - consider the quality goals and requirements of your project.

1. Divide the Functional Verification tests into two groups of Important and Not Quite As Important.
2. Demote the "Not Quite As Important" Functional Verification tests to Medium Priority.
3. Divide the Error and Boundary tests into two groups of Important and Not Quite As Important.
4. Promote the "Important" Error and Boundary tests to High Priority.
5. Divide the Non-Functional tests into two groups of Important and Not Quite As Important.
6. Promote the "Important" Non-Functional tests to Medium Priority.
7. Repeat the divide and promote/demote process for each set of High, Medium, and Low Priority test cases until you reach a point where the number of test cases being moved between priorities has become minimal.

3) Identify Build Verification Tests: Now, which tests must be checked with every build to ensure that the build is testable and ready for the rest of the team to start testing?

1. Divide the High Priority tests into two groups of Critical and Important.
2. Promote the "Critical" High Priority tests to BVT Priority.

Note: Do not identify BVTs first! BVTs are a selection of High priority test cases that are determined to be critical to the system and testing

At the end of this process, a rule of thumb is to check that the percent distribution of the priorities is along the lines of BVTs 10-15%, Highs 20-30%, Mediums 40-60%, and Lows 10-15%

When promoting and demoting test cases, aspects to consider are how frequently the user will require this feature or functionality. Likewise, how critical is this behaviour to the users day-to-day or month-end activities. Robyn Brilliant provides a list in Test Progress Reporting using Functional Readiness that you could apply when considering the test cases for promotion or demotion:

Using a scale from one to five, with one being the most severe and five the least severe, quantify the Reliability Risk as follows:

1. Failure of this function would impact customers.
2. Failure of this function would have a significant impact to the company.
3. Failure of this function would cause a potential delay to customers.
4. Failure of this function would have a minor impact to the company.
5. Failure of this function would have no impact.

This and similar scales can aid you in arriving at your final first cut of test case priorities.

Summary

This is a simplified example of a test case prioritization process. However, it can serve you well as a basis for rapid organizing of your test cases and getting your test schedule, efforts, and which test cases are done when mapped into the project plan.

Remember, how you prioritize your testing tasks and the test cases to be executed will depend on where you are in your project cycle. It is likely that you will re-prioritize your test cases as you move towards release and as you determine by investigation and observation where the risks and defects are manifesting. Establishing your testing objectives up front for each phase and making sure they are reflected in the individual priorities of your test cases will make your life a lot easier when it comes time to explain and execute your plan.

Finally, having prioritized test cases also gives you a good starting place for your potentially pending automation project. Ie: Automate the BVT Priority test cases, measure the benefits, improve the automation, automate the High Priority test cases, etc.

March 2004, by Trevor Atkins

User involvement is one of the major factors in designing UI for software. However, as much as we would like the users to be involved, user devotion to a project is never a free or unlimited resource. To maximize the benefit of their involvement, the design should be as free as possible of trivial bugs so that the users do not have to waste time encountering and overcoming these issues during the evaluation. [Task-Centered User Interface Design - A Practical Introduction, C. Lewis, J. Rieman, 1994]

Also, performing an evaluation with just user participants will not reveal all types of issues. For example, an interface used by thousands of individuals and tested with only a few users will not uncover problems that the evaluating users and the tests they perform don't happen to encounter. It also won't uncover problems that users might have after they get more experience with the system.

Human-Computer Interaction (HCI)

Software is created for users. Human-Computer Interaction (HCI) is a multidisciplinary study of people, computer technology, and how they influence each other. Not only does it touch on the technology and design, it also involves the studies of cognitive psychology and sociology. Understanding HCI principles and applying them to the UI design will make the software more usable for people.

Understanding Human Factors

Human factors - human abilities, human limitations, and other human characteristics from a physical and psychological perspective that are relevant to the design, operations, and maintenance of complex systems. [Northrop Grumman]

It's a common mistake for software projects to focus more on the technology aspects and neglect the relevant human factors and cognitive impact of the UI design.

"Humans are limited in their capacity to process information." [Human-Computer Interaction, Dix, Finlay, Abowd and Beale, 1998] These limitations refer to the capabilities of the mental processes used when gathering knowledge. UI designers must make sure to take these limitations into consideration when performing their work.

Memory

• Short-term memory is limited in capacity (~7 units) and it can be maintained for about 20 to 30 seconds.
• Long-term memory is unlimited in capacity and it is permanent in duration.
• It is faster to retrieve frequently accessed long term memory than less frequently accessed information from long term memory.

Perception

• Human perception is selective. When a lot of information is presented, our brain will filter the information to intake. It is easier to recognize and interpret objects that are familiar based on memory.

Learning

• Familiar, structured and organized information is easier to process.
• Smaller units of information are easier to learn.
• Because short term memory is limited, we select what we want to learn or store in our long term memory.

Know Your Users

Although we share the same model of information processing system (the human brain), different users have different ways of learning and form different conceptual and mental models about what they see. Therefore, it is important to know:

• Potential users' job and tasks.
• Potential users' knowledge and experience.
• Potential users' physical and psychological characteristics.
• Potential users' physical environment.

Human Factors Evaluation

The human factors in HCI can be used as the driving force behind a component-based review in producing a more 'user friendly' UI.

Grouping

• Group information - A group is treated as a single unit in short-term memory.
• Place commonly used buttons closer and make them larger in order to minimize hand and eye movement.

E.g. Font style, font type, font size, alignment controls are grouped in the Formatting toolbar.

Relationship

• Provide visual structure and organization of objects on screen.
• Use pattern recognition.
• Use relationship that is familiar to the user.
• Use metaphor and knowledge that can be transferred from the real world but, do not duplicate the limitation in the real world to software.

E.g. The graphics used for Play, Stop, Pause, etc. buttons in a CD Player software are similar to the ones used on an actual CD Player or tape deck.

Clarity

• Avoid using terms that are hard to distinguish in sound.
• Map only a one control to one piece of functionality.
• Avoid making the user learn more than what is necessary to perform certain functions.
• Avoid testing user's intelligence.
• Minimize the need of for the user to memorize information by displaying information on screen for as long as the user needs it.

E.g. On a multi screen application, buttons with the same name should perform the same functionality.

Cues

• Provide visual cues.
• Use pictures or icons - A picture is worth a thousand words.
• Use sounds to get the user's attention.
• Make information and controls visible.

E.g. Buttons are disabled instead of invisible when they are not available for use.

Cognitive Walkthrough

In addition to the more general design evaluation using human factors, another way to evaluate the UI is by performing a cognitive walkthrough. The cognitive walkthrough is a technique for evaluating the design of a user interface, with special attention to how well the interface supports "exploratory learning," i.e., first-time use without formal training. [Usability Evaluation with the Cognitive Walkthrough, John Rieman, Marita Franzke, and David Redmiles]. Walkthroughs should be done when the interface begins to grow and when components begin to interact with each other.

Lewis and Rieman suggested the following information is needed for a walkthrough:

• A description or a prototype of the interface.
• A task description.
• A complete list of actions needed to complete the task.
• An idea of who the users will be.

During a walkthrough, the evaluator will perform the task using the prototype given. This is a good way of imagining the user's thoughts and actions when using the interface.

Heuristics Evaluation

In addition to the cognitive approach to evaluate UI, another approach is by heuristic evaluation. Heuristic evaluation involves having a small set of evaluators (not users) examine the interface and judge its compliance with recognized usability principles. [Heuristic Evaluation, Jakob Nielsen]

Here are some recommended heuristics:

• Be consistent.
• Provide clear exit.
• Prevent error if possible.
• Provide clear error message in easy to understand language.
• Use familiar language and logic that it is easier to learn and understand. Avoid using technical jargons.
• Display information on screen until it is not needed by the user.
• Provide feedback to the user within a reasonable timeframe.
• Account for both experienced and inexperienced user.
• Irrelevant or rarely used information should not be displayed unless the user asks for it.
• Provide documentation.

Summary

A good non-user evaluation, or expert review, using established usability assessment principles can catch problems that an evaluation with only a few users may not reveal. If some key evaluation and design guidelines are followed the critical problems can be detected and resolved.

Of course, performing just an evaluation without users won't uncover all the problems either. Once the evaluation without users is complete, and appropriate design changes are made, the next step will be to get the users to participate.

February 2004, by Trevor Atkins

A guiding principle in the software industry, considering the wide range of project scope and constraints, is to "use the processes and tools appropriate to the size, complexity, and impact of your project".

For instance, each of the following situations demand different approaches to process and tools:

• You are a company that has had to reduce staff and now work with a very tight budget.
• Your company's recent successes have driven significant growth, but these days project closure is not as tidy as it once was.
• There are strong demands about the level of documentation detail you are required to have in place for each project. You are targeting the European ISO-based market, or your client expects you to develop to the higher CMM standards, or you are developing devices to strict government (eg: FDA) requirements.
• On your large or multi-phase project you find that you are reinventing the wheel each time someone leaves and key information disappears with them.

We are all familiar with templates - applications that create artifacts typically come with a number of them, with more available for download. Templates are provided with such applications to give a user a starting place for something unfamiliar or something that is repeated often. They provide formatting and style to documents or presentations, canned code modules for development, and capture the procedures and organize the data gathered.

Templates and Their Benefits

Documented methodologies and practices typically include:

• Process Guides and Procedures - form an overview of the "how and why" steps in the software development, quality, and project management processes, providing a reference and assistance for bringing new staff up to speed quickly.
• Checklists - succinctly capture the necessary steps to be taken at key points throughout a process, and can be used as safeguards and memory triggers.
• Templates - support standardized formatting and content consistency for each artifact, and can explicitly define what information should be contained in each document via inline guidance text.

Once the templates have been created and the supporting guidance documents are complete and consistent, you can benefit from rapid reusability and ease of customization. Customizing for a new project simply consists of inserting project specific needs and editing out sections that do not apply this time around - a much simpler task than adding information to an incomplete collection of previous examples or creating them from scratch each time.

Within a template you can capture aspects of industry best practices and then make the conscious choice to address them or not for your project. In fact, templates are a strong component of achieving and maintaining CMM Level-2 Repeatable, where the goal is to show the adherence of software products and activities to applicable standards, procedures, and requirements. This goal would be met in part by the adoption of a set of checklists and document templates that can be applied as project standards. Adding a simple web page to your Intranet that outlines the project lifecycle and which templates to use at each stage will make it quick and easy to communicate these artifacts throughout your organization.

If all team members use a common format, it is much easier for the entire team to create the necessary artifacts and interpret and apply the recorded information. When working with 3rd parties or outsource organizations, templates are an excellent vehicle to ensure the consistency of the work produced - to the point where internally and externally produced artifacts should not appear significantly different - and this greatly facilitates the review of these deliverables.

The following are some of the benefits that can be attributed to proper use of templates:

The following is a brief sample from a template that contains guidance text for what information is to be captured in the given section:

1. Project Overview

1.1 Quality Standards

Proceed with Caution

A common mistake with templates is to focus on following the structure of the template, rather than allowing the headings and guidance text to function as a framework for quality content. By themselves, process and tools cannot lead a project to success, and a template is just another tool. Appropriate skill and experience in the project team; the ability to collect and analyze information in the context of the project and the team member's role, are necessary for successful projects.

Matthew Edwards points out in "Basic Test Form Templates" that "...management can direct a test engineer to prepare a test plan, and even provide training and a planning template. But, there's no guarantee that the plan will be a good one."

In "Are Templates Dangerous?", David Gelperin describes the hazard as follows: "Those who understand testing as a game of checkers see test documentation as an exercise in filling in the blanks of a template. Those who understand the chess-like complexities of testing see doc templates as a guide to recording the results of a difficult decision process."

A template can identify and organize the important elements for a given artifact; however, to capture the intent of those elements as part of an effective project solution requires experience and insight.

Summary

Although the ROI from creating and using templates is significantly enhanced when implemented as part of an on-going process improvement initiative, defining templates for critical project artifacts is a good first step on the way to allowing your organization to:

• Define your development, quality and project management processes
• Implement best practices and standards throughout your organization
• Communicate these policies, procedures and standards throughout your software organization
• Create and maintain a single reference point for your entire organization for policies, procedures and standards

Which in turn can lead to:

• Greater control over projects
• More effective change management
• Standardized, streamlined and repeatable processes across your organization
• Managed schedules and predictable project costs
• Increased customer satisfaction

December 2003, by Trevor Atkins

How can you be sure that an application will behave properly when users perform actions or combinations of actions that were not considered during the development of the functionality? During the testing phase, you have to plan for what is sometimes called "bad user" testing, or negative testing.

Boris Beizer's definition of negative testing in "Software Testing Techniques" is: "Testing aimed at showing software does not work".

In his paper, "A Positive View of the Negative Testing", James Lyndsay stated the objectives of bad user testing to be:

• Discovery of faults that result in significant failures; crashes, corruption and security breaches
• Exposure of software weakness and potential for exploitation
• Observation and measurement of a system's response to external problems

Why Is Bad User Testing Important?

During negative or bad user testing, the tester seeks to abuse the functionality of the product in an effort to create odd program states by exercising functionality that deals with state management, input validation, boundary conditions, fault recovery, and more.

Bad user testing is generally performed as part of integration or system testing and does not have a distinct phase of its own. The basic principle to follow is if the tester can perform bad user tests without error, there is a significantly lower chance that the users will inevitably find such defects later on.

As Lyndsay points out in is his paper, negative testing can find significant failures and also can produce invaluable strategic information about the risk model underlying testing, and allow overall confidence in the quality of the system.

Where To Start?

It is possible to design a bad user test plan starting from the specification documentation. The most important thing to keep in consideration when designing a bad user test plan is to not test what is described, but what is not. The tester should look at the specification documentation as a guide to what the boundaries of the software and then look beyond the boundary to the extreme edges of the software's functionality. Employ your creative destructiveness to take you beyond these boundaries and perform actions and tasks that you are sure will fail, or should be impossible. These are the areas where the most interesting defects can be found.

Of course, each product is specific and all tests cannot be applied in all circumstances. The few brief examples below can be used as a guide when starting a bad user test plan. Use your creativity to add to and expand this list.

Generic Bad User Test Scenarios

What is expected behaviour when you:

• Manually shut down, or reboot, the computer while the application is running
• Manually restart the computer by pushing the reset button while the application is running
• Restart the computer while the application is running using the start menu
• Log off while the application is running

Boundary Test Scenarios

What is expected behaviour when you:

• Attempt to go below the minimum input limits
• Exceed the maximum input limits

Stress Test Scenarios

What is expected behaviour when you:

• Run the program concurrently with many other programs
• Set memory to a stressed state (low Virtual Memory, low RAM)
• Run on slower or older machines
• Load large volumes of data
• Create large numbers of concurrent connections or open files

Performance Test Scenarios

What is expected behaviour when you:

• Test on a system lower than the minimum configuration requirements
• Open extremely large files
• Attempt to import a corrupt file
• Create a situation requiring an error message

Install/Uninstall Test Scenarios

What is expected behaviour when you:

• Run the installation script again while the application is running
• Cancel the installation midway through the install using the task manager
• Install the application onto a hard drive without enough free space
• Install the application onto a network drive and disconnect the drive partway through
• Uninstall the application when the application is still running
• Change or remove the registry settings before trying to uninstall the application

Summary

Bad user testing is performed by the tester to find weaknesses in the design or the code by attempting actions that are likely to occur after deployment. Although it is hard to make predictions about real-world use, real users will find all ways of using the system including those that were considered unreasonable or improbable.

Including this type of testing as part of your project will result in a more robust and user-friendly application and will of course save effort and costs after the product is released.

November 2003, by Trevor Atkins

Error messages are displayed by applications in response to unusual or exceptional conditions that can't be rectified within the application itself.

The need for "useful error messages" can be defined, in the simplistic case, to be a need for some form of error handling and reporting that enables the user to understand what has happened in the case of an error and what must be done to remedy the situation.

Most testers are no doubt familiar with the feeling of reluctance to log usability issues, fearing that they could be misunderstanding the functionality or they are "wasting valuable time reporting trivial bugs". The project team can further drive this feeling by tending to postpone or ignore such issues under the premise that "at least there is some feedback isn't there?", or "there isn't going to be time to address those kind of issues", and besides "the user wouldn't do that."

Issues with Error Messages

"Error messages are often less than helpful or useful because they're written by people who have an intimate knowledge of the program. Those people often fail to recognize that the program will be run by other people, who don't have that knowledge." Michael Bolton, 1999.

Furthermore, Byron Reeves and Clifford Nass suggest in 'The Media Equation', that even text-only interfaces are felt by users as having some "personality" and that "people respond socially and naturally to media."

As noted by Julianne Chatelaine in 'Polite, Personable Error Messages', Byron Reeves and Clifford Nass determined that if the application does not have the ability to assess each user's personality and adapt to it, the next best thing is to select one personality or tone and be consistent to avoid contributing to confusion and even dislike. The published TME findings were underscored by Nass' remarks at UPA '97 where he said that when an application's textual messages were written by a variety of different people, using different styles and degrees of strength or dominance, it made the product seem "psychotic."

Guidelines for Error Messages

"You may design the perfect system but eventually, your system will fail. How it does so, however, can make all the difference in the world in terms of usability." Tristan Louis, 'Usability 101: Errors'.

"The guidelines for creating effective error messages have been the same for 20 years." Jakob Nielsen, 'Error Message Guidelines'.

The following checklist, compiled from several of the referenced sources, will help you confirm that your application meets basic usability requirements with respect to error messages.

• Message Exists: the problem with an error is often that no message is actually attached to it. Notify the user when the error happens, every time it happens. The error may be due to a flaw in the software or a flaw in the way the user is using the software but if the user doesn't know of the error, they will assume that the problem is with the software.
• Polite Phrasing: the message should not blame users or imply that they are either stupid or doing something wrong, such as "illegal command."
• Visible and Human-readable: the message should be highly noticeable and expressed clearly in plain language using words, phrases, and concepts familiar to the user rather than in system-related terms.
• Precise Descriptions: the message should identify the application that is posting the error and alert the user to the specific problem, rather than a vague generality such as "syntax error".
• Clear Next Steps: error messages should provide clear solution steps and/or exit points. An application should never capture users in situations that have no visible or reasonable escape.
• Consistent: users should not have to wonder whether words, icons, colors, or choices mean the same thing or not in different instances.
• Helpful: the message should provide some specific indications as to how the problem may be resolved and if possible let users pick from a small list of possible solutions. Links can also be used to connect a concise error message to a page with additional background material or a detailed explanation of the problem and possible solutions. Finally the message should provide extra information, such as an identifying code, so that if technical support is helping the end-user they can better analyze and remedy the problem.

Error Message Presentation

When deciding on the style your error messages will adhere to, you should consider the presentation of your error message:

• Tone: be firm and authoritative, stating the facts of the situation in a neutral and business-like manner.
• Color: an error message printed in red may call attention to itself, but to use color solely as the way to present an error message is generally a poor idea. People that are color blind for example will not read the text with any additional meaning attached to it.
• Language: if your application is used by people in different countries, consider that your error messages will have to be translated and need to be presented in a format flexible enough to accommodate the translated text.
• Icons: if you use icons to present your error messages make sure they are intuitive to the end-users and that they are appropriate to the circumstance of the error message.

To highlight how careful you should be when considering icons, Tristan Louis cites in 'Usability 101: Errors', the case of when an Apple Macintosh crashed, it used to show an icon presenting a little bomb with a burning fuse along with the message in the error dialog. He comments that users in many countries were terrified by this icon and would not touch the computer for fear that it would actually explode.

Summary

Remember that errors will happen but what will make all the difference is if they are handled properly. Unclear and unhelpful error messages tend to mean that errors will recur, or take longer to resolve. The resultant frustration can lead users to mistrust the interface or even abort the task in question.

Your error message must convey useful information -- useful information saves time and for more than just the end-user. The message will also need to be understood by and useful to the technical support person who handles the call, the quality assurance analyst who helps to track down or replicate the problem, and the maintenance programmer who is charged with fixing the problem in the code. Each person in this process represents a cost to your company, cost that could be greatly mitigated by a small investment made now.

October 2003, by Trevor Atkins

Testing costs can be a significant part of the project with software project managers spending up to half of their project budget on testing. But how do you make testing more cost effective so that you are getting more done with less?

One effective solution is automated testing or "tool assisted test activities performed with the objective of evaluating the software against pre-defined results/expectations that require no operator input, analysis, or evaluation." [QA Labs Inc., 1999]

Many people try to add automation to their projects, only to end up frustrated, and annoyed. After one or two disastrous attempts, many just give up and stop trying. However, implementing automated testing is a basic cost-benefit analysis.

It is well-recognized that an automation undertaking will require significant investment up-front before actual savings can be experienced. At the same time, the positive effects of having automation can be experienced by the organization in advance of the anticipated actual break-even point.

To help approach the undertaking of automation from a realistic perspective, keep in mind that Automated Testing is not:

• Immediate effort reduction
• Immediate schedule reduction
• A silver bullet to find all the application defects
• Automatic Test Plan Generation for 100% test coverage
• One tool fits all and is easy to use
• Cheap to implement

The savings and benefits will come, though, if you recognize and plan for the following.

Avoiding Pot-holes

The following are some typical issues that those implementing test automation run into:

• Pot-hole: Test automation is not treated as a project with proper project planning and design
  Solution: Treat test automation as you would a development project and manage the scope, resources and schedule appropriately. Implement a pragmatic approach to testing such that: the project can be decomposed into modular, defined tasks with assigned resources and timelines; others can easily carry forward the process that has been defined; the effort and results are quantifiable; each test cycle becomes more efficient in uncovering defects; and the most critical test types and application functionality are targeted.
• Pot-hole: No reusability (use of functions and utilities) in automation scripts
  Solution: Implement an effective test automation framework through abstracting navigation, data access, verifications, reporting, and other common functions into libraries to modularize scripts thereby minimizing maintenance costs as there are changes to the application functionality.
• Pot-hole: Testers untrained in programming techniques are assigned automation tasks
  Solution: Testers performing test automation must be able to create and maintain automated test scripts. This requires strong knowledge of software development practices, experience with procedural programming languages, and experience with the test automation tool to be used.
• Pot-hole: Automation test suite is not maintained
  Solution: Test suites need to be maintained with each new build and release of an application. Maintenance of robust scripts typically requires ~10% of the time of originally creating the automation scripts, assuming an automation framework is firmly in place and that major additions or redesigns are not being done to the application.
• Pot-hole: Testing is typically performed at the end of the project life cycle
  Solution: The test process should begin where the development process does, at the beginning. Moving testing up the life cycle increases the ability to find defects sooner and provides more time for effective test planning, design, execution and tracking, and stability for automation.

Scripting Best Practices

Test Automation Scripts are software. And even if the current intended use is for testing the current project only, you never know where those scripts could end up, or for what other purpose they could be used.

You may develop a framework and individual scripts for one version of the product, but require subtle modifications for a version of the product customized for another customer.

You may have large automation effort ongoing, including potentially multiple applications or versions of your product and have multiple people modifying the core framework scripts.

To help in succeeding with an automation undertaking, keep the following best practices at the front of your mind:

• Document
• Manage expectations
• Keep in mind the overall project testing goals
• Plan to control progress and recovery of the execution of the testing
• Control the scope of the automation
• Use coding standards when writing your scripts
• Version Control is important
• Get early feedback
• Test your scripts!

Summary

"... when all the pieces come together - the right people, the right processes, the right time, the right techniques, the right focus - then we can achieve truly impressive returns on our testing investment." [Investing in Software Testing, Rex Black]

If implemented thoughtfully, the automated test suite will prove to be much more efficient than manual testing in terms of hours spent and defects uncovered in previously manually tested functionality. The automation suite can be left unattended to run at night, on weekends and holidays. The tools never get bored or tired and never assume the application/architecture works while emulating as many users as needed, accessing the application and performing any mix of transactions desired.

Therefore, the return on investment (ROI) of automated testing can be tremendous, as long as you can avoid the pot-holes along the way.

March 2003, by Trevor Atkins

How do you know what a system is supposed to do and what it is not supposed to do? Requirements are intended to create an easily validated, maintainable and verifiable document describing a system's planned functionality.

What is lacking in today's requirements? Requirements are typically described in natural language because both the customer and vendor must understand them. However, many words and phrases have meanings that can be interpreted based on the context in which they are used. Requirements described in this form can have several severe problems including: ambiguity, inaccuracy and inconsistency.

The criticality of correct, complete, testable requirements is a fundamental tenet of software engineering. The success of a project, both functionally and financially, is directly affected by the quality of the requirements. [Doing Requirements Right the First Time!, Theodore F. Hammer, Linda H. Rosenberg, et al., 1998]

Who Needs Requirements?

The communication of requirements is the most critical, difficult, and error-prone task in IT projects. Research has shown that projects that proceed to the coding phase with missing or incorrect requirements are almost certain to fail. [A Systematic Approach for More Effective Communication of Functional Requirements and Specifications, Bill Walton]

Consider the various stakeholders or audiences for the requirements - on which each is dependent for their own role in the project lifecycle:

• Customer - Statement of work for vendor, acceptance criteria
• Marketing - Documented product capabilities
• Project Management - Estimates, project plans, project goals, risk planning
• Development - Design and coding of the system
• Testing - Verification of the system
• Technical Writing - User manuals and tutorials

Unless there is effective communication of information the same words can mean different things to each party, or different words can appear to repeat the same thing, becoming a source of misunderstanding and therefore error.

Quality Requirements

Customer dissatisfaction often surfaces at the acceptance phase as discrepancies appear between what was built and what the customer thought was being built - a clear result of lack of quality requirements.

Some of the commonly recognized crucial attributes of quality requirements are:

• Completeness
• Consistency
• Correctness
• Understandability
• Unambiguousness

These quality attributes tend to be considered subjectively. However, some of these quality attributes can be linked to indicators to provide evidence that the attribute is present or not. The NASA Goddard Space Flight Center's (GSFC) Software Assurance Technology Center (SATC) defines categories of quality indicators related to individual specification statements as: Imperatives, Continuances, Weak Phrases, Directives, and Options.

The SATC's studies give the following specific words and phrases to be indicators of a document's quality as a specification of requirements:

• Imperatives - Words and phases that command that something must be done or provided. The number of imperatives is used as a base requirements count. Eg: Shall, must or must not, is required to, are applicable, responsible for, will, should
• Continuances - Phrases that follow an imperative and introduce the specification of requirements at a lower level, for a supplemental requirement count. Eg: As follows, below, following, in particular, listed, support
• Directives - References provided to figures, tables, or notes.
• Weak Phrases - Clauses that are apt to cause uncertainty and leave room for multiple interpretations measure of ambiguity. Eg: Adequate, as applicable, as appropriate, as a minimum, be able to, but not limited to, be capable of, effective, easy, effective, if effective, if practical, not limited to, normal, timely
• Incomplete - Statements within the document that have TBD (To be Determined) or TBS (To Be Supplied).
• Options - Words that seem to give the developer latitude in satisfying the specifications but can be ambiguous. Eg: Can, may, optionally

The SATC developed the Automated Requirements Measurement (ARM) tool which project managers can use to assess the quality of their requirements documents easily and on an on-going basis during the life of the documents. The ARM tool searches the requirements document for terms the SATC has identified as quality indicators.

Requirements Style Guide

Similar to a development coding standard, a requirements style guide outlining when and where such terms and structures must and alternately may be used can help maintain control over ambiguity, consistency, and completeness.

However, although many organizations have published documentation standards that include standards for specifying requirements, none are universally accepted. The standards that are imposed seldom go beyond providing an outline or template of the general information to be provided. In many cases no style guidelines are established. As a consequence, requirements documents from various sources tend to bear little resemblance to one another. [Automated Quality Analysis Of Natural Language Requirement Specifications, William M. Wilson, Linda H. Rosenberg, Lawrence E. Hyatt]

The ultimate symptom of vague requirements is that developers have to ask the author, analyst or customers many questions, or they have to guess about what is really intended. The extent of this guessing game might not be recognized until the project is far along and implementation has diverged from what is really required. At this point, expensive rework may be needed to bring things back into alignment. [Karl Wiegers Describes Ten Requirements Traps to Avoid, Karl Wiegers, 2000]

Karl goes on to suggest that requirements authors avoid using subjective and ambiguous words like minimize, maximize, optimize, rapid, user-friendly, easy, simple, often, normal, usual, large, intuitive, robust, state-of-the-art, improved, efficient, flexible, "and/or" and "etc.". Indicate current uncertainties or areas to further clarify with "TBD" markers to make sure they get resolved before design and coding proceeds.

Summary

Quality documentation is complete, clear and concise. These used to be considered intangible concepts, difficult to measure. With a well defined style guide that addresses the quality attributes like those identified by SATC, metrics can be rapidly developed and analyzed to reveal the strengths and weaknesses of the requirements documentation.

Most projects come with that sense of urgency attached - we need to start coding and we need to start now. In this kind of rushed atmosphere, it's hard to convince yourself to take the time to even perform documentation activities at all, let alone to develop and apply a new documentation technique.

So why should you go to the trouble? Because addressing the aspects of quality (or lack thereof) on a project leads to working smarter rather than harder, better products, more satisfied customers, and higher profits. But the perfect and easy time for doing the things you know you should do will never come.

Remember, industry data suggests that approximately 50 percent of product defects originate in the requirements. Perhaps 80 percent of the rework effort on a development project can be traced to requirements defects. Anything you can do to prevent requirements errors from propagating downstream will save you time and money. [Inspecting Requirements, Karl Wiegers, 2001]

For more information on the NASA SATC quality attributes and a free download of the ARM tool, visit: http://satc.gsfc.nasa.gov/tools/arm/

February 2003, by Trevor Atkins

Each of us has very likely had to do an estimate in the past, whether it was for a set of assigned tasks, for a project, or for an entire organization. As a tester, the question is commonly presented as, "How long will it take to test this product - and what resources will you need?" and then the person asking stands there, likely somewhat impatiently, and waits for the answer.

One common approach is to not base test effort on any definitive timeframe. The testing simply continues from when the code is ready until some pre-decided timeline set by managerial personnel is reached. Another common approach is to estimate testing effort as a percentage of development time. Development effort is estimated using some techniques such as Lines of Code or Function Points and the allocated test effort is derived using a pre-determined ratio. Both practices rely heavily on the test team's ability to work to the strategy and uncover the significant defects up front. However, with this approach there is little ability to invest in planning the test effort or creation of tests. These methods are not based on any assessment technique that takes into account the additional complexities of the test effort, such as deployment configurations and human language support.

As noted by Capers Jones in Assessment and Control of Software Risks, most projects overshoot their estimated schedules by anywhere from 25% to 100%, but some few organizations have achieved consistent schedule-prediction accuracies to within 10% and 5%. Just as it is critical to offer something more than an off-the-cuff answer for the development activities, it is important to know how to perform an estimate of the testing effort for a project.

A good simple definition of an estimate consists of a description of the size or scope of the undertaking, the level of confidence or uncertainty in the estimate at the time that it is made, and a description of the technique used to arrive at the estimate. "It is very difficult to make a vigorous, plausible, and job-risking defense of an estimate that is derived by no quantitative method, supported by little data, and certified chiefly by the hunches of the managers," according to Fred Brooks in The Mythical Man-month.

Getting a Starting Point

The basic elements to consider when performing an estimate for test effort is the size of the system to be tested, the components of the system, the quality requirements for each component, the resources available, and the level of productivity of those resources. From these elements we are first able to determine the overall size of the test effort in terms of test cases or verification points (eg: within a Use Case). Then considering the resource availability and level of productivity the total effort and schedule can be determined.

Regardless of the uncertainties and risks that may come into play at the beginning or during the project, we still need to get a starting number around which we can base our estimate. A well-defined requirement or specification, being a structured document (or documents) that likely follows certain standards in authoring and describes the scope of the system to be produced, is the best source for producing an estimate. These qualities allow the system to be sized using a variety of techniques that can quantify both the systems' functionality and its complexity (eg: performance, stress, security, and other non-functional requirements).

With an algorithmic approach to generating an estimate, the first step is to enumerate the collected requirements. If a requirements style guide has been used it should be easy to identify the number of requirements captured in the text. Also consider the number of screens and input fields for each. You may find it useful to group the requirements by type: imperative, weak phrase, list, etc. and weight them with the number of estimated tests for each type. Next further group and weight the requirements by complexity and the ability of developers to implement. Here you can make use of historical information for the organization or team from past projects - perhaps look at bug counts for the modules that are similar to those in your project or the estimate overruns for the different phases of the project.

That sounds straightforward and simple; just count the requirements, group them, and apply a set of formulae. But is it that easy? As Steve McConnell comments in Rapid Development about the accuracy of the first estimate of the project, "Some organizations want cost estimates to within plus-or-minus 10 percent before they'll fund work on requirements definition. Although that degree of precision would be nice to have that early in the project, it isn't even theoretically possible. That early, you will do well to estimate within a factor of 2."

More Than Test Execution

Depending on information and time available, your formulae can be made increasingly complex to factor in different influences and trade-offs in scope, resources and schedule. As more understanding of what influences your estimates is gained and more iterations of the estimate are completed you may find your model increasing in sophistication; similar to the increase in understanding gained between Dalton's and Bohr's atomic models.

However, you can still rapidly complete a list to account for all the activities you perform in a project that are related to actual test execution such as:

• Reviews of requirements and designs
• Test strategies and test plans (including test cases)
• Test analysis/matrices and test data preparation
• Test automation

These "overhead" factors to test execution depend on the quality requirements and extent of investment in upfront planning. The percentage of effort as it relates to test execution can often be directly tied back to the number of test cases or verifications calculated earlier and therefore be 'formularized'.

Uncertainty Factors, Multipliers, and Other Influences

Counting the requirements and applying formulae is certainly the basis of the approach, however there are a number of uncertainty factors, multipliers and influences to be considered when examining the project for test effort.

• Are requirements, designs, plans, and so forth available and are these documents clear, concise, and accurate?
• Do project stakeholders have realistic expectations in terms of schedules and functionality?
• Are there clearly defined milestones during the project for testing? (eg: Alpha, Beta, Gold)
• How well managed are the change control processes for project and test plans, requirements, designs, and code?
• Does the project team have the skills, experience, and tools needed for this project?
• Is the project team established or is there expectation of ramping up or turnover during the life of the project?
• To what extent can the project re-use test assets from previous projects?
• What is the required investment in the test environment set-up and maintenance?
• Have meetings, vacations, and sick times been built into the schedule?
• How many builds are planned to be delivered to testing? What if there are additional builds required or what if one is delayed?
• How many deployment configurations are to be supported and need to be tested? Do all of them need to be tested to the same degree?
• How many human languages are to be supported? Are special skills required for this type of testing?
• What amount of non-functional testing is required or planned?

All of the above uncertainty factors can be mitigated through upfront planning and investment. But if you don't have the time to address training issues, requirement reviews for clarity and testability, or change control standards make sure to take this into account when considering the certainty of your estimate.

Finally, don't just have one person do the estimate. Discussion of differences in numbers can make visible and clarify assumptions or advantages of approaches. Don't forget to include a level of confidence in each phase of the estimate and the final overall number. A Guide to the Project Management Body of Knowledge (PMBOK), Project Management Institute defined an estimate as: "An assessment of the likely quantitative result. Usually applied to project costs and durations and should always include some indication of accuracy (+- x percent)." As an estimate is refined as the project progresses, the effort may change but the confidence level should rise.

Summary

Benefits of Quality and Costs of Quality are in a balance and it is important to find and commit to the right equilibrium when facing the continuing challenge of not enough time for testing. There are many approaches to estimation of the test effort of a project and though the outlined approach described above is in no way rigorous, it offers the ability to approach the task in a systematic manner with a defined technique and supporting data - a significant practical advantage over ad hoc techniques that allows further research and experimentation to improve the methods used to arrive at valid estimates.

December 2002, by Trevor Atkins

A typical software project lifecycle includes such phases as requirements definition, design, code and fix. But, are you shipping software applications with minimal to no requirements and little time for testing because of time-to-market pressures? Build it, ship it, then document and patch things later.

'Time-to-market' products can lack detailed requirements for use by testing because of their huge pressures for quick turnaround time. You don't want to slow down development, or testing, by having to create detailed documentation. At the same time, the test effort needs to be useful and measurable. So, how can this product be tested to achieve adequate effective coverage and overall stability of its functionality?

A Starting Point

Effective ad-hoc testing relies on the combination of tester experience, intuition, and some luck to find the critical defects. Adequate test coverage involves a systematic approach that includes analyzing the available documentation for use in test planning, execution, and review. Ad-hoc testers can greatly benefit from up-front information gathering, even when they don't have time for formal testing processes and procedures. Ad hoc testers must still understand how the software is intended to work and in which situations.

Ask developers, testers, project managers, end users, and other stakeholders these basic questions to assist with clarifying the product's undoubtedly complex tasks:

• Why is the system being built?
• What are the tasks to be performed?
• Who are the end users of the system?
• When must the system be delivered?
• Where must the system be deployed?
• How is the system being built?

Also, the risks of the system need to be identified. (See our article Risk Based Testing Article #11 for more on this topic.) Correlate these risks against the time available to prioritize the test focuses.

With this information you are well on your way to being able to define an applicable strategy for your upcoming test effort.

User Scenarios

User Scenarios (sometimes called Use Cases) define a sequence of actions completed by a system or user that provides a recognizable result to the user. A user scenario is written in natural language that pulls together from a common glossary. The user scenario will have the basic or typical flow of events (the 'must have' functionality) and the alternate flows. Creating user scenarios/use cases can be kick-started by simply drawing a flowchart of the basic and alternate flows through the system. This exercise rapidly identifies the areas for testing including outstanding questions or design issues before you start.

Benefits of creating user scenarios:

• Easy for the owner of the functionality to tell/draw the story about how it is supposed to work
• System entities and user types are identified
• Allows for easy review and ability to fill in the gaps or update as things change
• Provides early 'testing' or validation of architecture, design, and working demos
• Provides systematic step-by-step description of the systems' services
• Easy to expand the steps into individual test cases as time permits

User scenarios quickly provide a clearer picture of what the customer is expecting the product to accomplish. Employing these use cases can reduce ambiguity and vagueness in the development process and can, in turn, be used to create very specific test cases to validate the functionality, boundaries, and error handling of a program.

Checklists

Are there common types of tasks that can be performed on the application? Checklists are useful tools to ensure test coverage of these common tasks. There may be a:

• User Interface checklist
• Error and Boundary checklist
• Certain Features (eg: Searching)

Benefits of creating checklists:

• Easy to maintain as things change
• Easy to improve as time goes by
• Captures the tests being performed in a central location

Checklists used in conjunction with User Scenarios make a powerful combination of light-weight test planning.

Matrices

A test matrix is used to track the execution of a series of tests over a number of configurations or versions of the application. Test matrices are ideal when there are multiple environments, configurations, or versions (builds) of the application.

Benefits of using test matrices:

• Easy to maintain as priorities and functionality change
• Simple to order the functional areas and the tests in each areas by priority
• Clear progress monitoring of the test effort
• East to identify problem areas or environments as testing proceeds

Test matrices provide a clear picture of what you have done and how much you have left to do.

Summary

If you have minimal to no requirements there are still ways that effective testing can be achieved with a methodical approach. You can quickly outline a methodology for yourself that considers the basics of:

• Describing the application in terms of intended purpose
• Identifying the risks of the application
• Identifying the functionality of the application with basic and alternate flows
• Identifying and grouping common tests with checklists
• Identifying how testing records will be traced
• Revisiting and refining each of the above as the project and testing effort proceeds

November 2002, by Trevor Atkins

Are you ready to take on the challenges of the new project? Are you knowledgeable of the latest in testing tools and techniques? What will be the ramification of not testing the product adequately? How will this impact your future business? Can you afford not to test?

Exhaustive software testing standards, frame-works, and techniques, promote the notion that a robust variety of testing techniques and structures will increase the likelihood that defects will be uncovered. But with tight project budgets and short timelines, the accompanying bureaucracy and documentation can greatly reduce the interest in formalized testing, structured or otherwise.

It is well understood that a higher quality product demands a higher upfront price and compromises regarding quality versus cost are made every day. However, the best approach is not likely the most structured or complicated one. Rather a sophisticated approach is required that maximizes the value of the resources available within the organization and without. A good starting place to developing this sophisticated approach is to examine the fundamental skills that make great testers great, enabling them to draw upon their almost innate ability to find the crucial defects quickly.

From this examination it is very probable that you will generate ideas for:

• Managing iterative development and test cycles
• Creating reusable sets of tests and test data
• Compressed delivery schedules
• Newly refactored architectures
• Standardized project metrics
• Migrating from simple to complex deployments
• Changing customer expectations

A few simple principles can provide the framework from which to grow and adapt the test approach. Keeping things simple will make it easy for the benefits and costs to drive further evolution. While there are continuously new development tools and programming languages, many testing requirements remain the same, and simply require additional emphasis within and by the test team.

• Examples - having previous projects from which examples of the "best of breed" can be drawn for each type of process, document, or test technique is invaluable in giving the next project a giant jump start in how to approach the test effort. Asking, "how did we do it last time and what can we improve?" will drive forward improvements to your testing toolkit strongly, and with frequent project iterations, rapidly.

From these previous project examples you can begin to derive reusable tools for your toolkit:

• Guidance Process - generic process frameworks and best practices that can be applied to most project types and be ingrained as habit as much as in any documentation.
• Templates - light-weight documents focused on capturing the critical information and not on keeping resources busy with technical writing.
• Checklists - lists of test, lists of tasks, matrices of test configurations that allow you to rapidly document and check off what has been done and see what is left to do

Note: If you do not have your own history of previous examples there are many resources on the web where others share their experiences and advice such as at www.stickyminds.com

In one such example, James Bach of Satisfice.com provides a number of whitepapers and articles on "Exploratory Testing" wherein he has a set of mnemonics and heuristics in his toolkit. One of these mnemonics is SFDPO where the letters stand for Structure, Function, Data, Platform, and Operations.

• Structure - what the product is
• Function - what the product does
• Data - what the product processes
• Platform - what the product depends upon
• Operations - how the product will be used

Using rules and checklists such as this allow you to quickly focus your test idea generation and ensure that you have systematically visited the major aspects of the product.

"SFDPO is not a template or a test plan, it's just a way to bring important ideas into your conscious mind while you're testing. It's part of your intellectual toolkit. The key thing if you want to become an excellent and reliable exploratory tester is to begin collecting and creating an inventory of heuristics that work for you. Meanwhile, remember that there is no wisdom in heuristics. The wisdom is in you. Heuristics wake you up to ideas, like a sort of cognitive alarm clock, but can't tell you for sure what the right course of action is here and now. That's where skill and experience come in. Good testing is a subtle craft. You should have good tools for the job." - James Bach

However, even with the best tools and techniques, a test team can't create the kind of return on investment managers require as long as the test efforts don't start early and don't involve all appropriate stakeholders and participants. When developing your testing processes, look for those improvements where:

• Errors are detected and corrected as early as possible in the software life cycle
• Project risk, cost, and schedule effects are lessened
• Software quality and reliability are enhanced
• Management visibility into the software process is improved
• Proposed changes and their consequences can be quickly assessed

"... when all the pieces come together - the right people, the right processes, the right time, the right techniques, the right focus - then we can achieve truly impressive returns on our testing investment. Significant reductions in post-release costs are ours for the taking with good testing. In cost of quality parlance, we invest in upfront costs of conformance (testing and quality assurance) to reduce the downstream costs of nonconformance (maintenance costs and other intangibles associated with field failures)." - Investing in Software Testing by Rex Black.

October 2002, by Trevor Atkins

Today's client/server systems are expected to perform reliably under loads ranging from hundreds to thousands of simultaneous users. The fast growing number of mission critical applications (e-commerce, e-business, contents management, etc.) accessible through the Internet makes web site performance an important feature for success in the market.

According to a broad statement in the white paper Web Performance Testing and Measurement: a complete approach, by G. Cassone, G. Elia, D. Gotta, F. Mola, A. Pinnola: "...a survey [in the US] has found that a user typically waits just 8 seconds for a page to download completely before leaving the site!"

Organizations need to perform repeatable load testing to determine the ultimate performance and potential limits of a system on an on-going basis. Poor performance can have direct negative consequences on the ability of a company to attract and retain its customers. Controlling performance of web site and back-end systems (where e-business transactions run) is a key factor for every on-line business.

"Performance Testing" is the name given to a number of non-functional tests carried out against an application. There are three main elements that often comprise what is called Performance Testing. These are:

• Performance Testing - Concentrates on testing and measuring the efficiency of the system.
• Load Testing - Simulates business use with multiple users in typical business scenarios, looking for weaknesses of design with respect to performance.
• Stress Testing - Sets out to push the system to its limits so that potential problems can be detected before the system goes live.

A difference between performance and load testing is that performance testing generally provides benchmarking data for marketing purposes, whereas load testing provides data for the developers and system engineers to fine-tune the system and determine its scalability.

With load testing, you can simulate the load generated by hundreds or thousands of users on your application - without requiring the involvement of the end users or their equipment. You can easily repeat load tests with varied system configurations to determine the settings for optimum performance. Load testing is also particularly useful to identify areas of performance bottlenecks in high traffic web sites.

Top five questions to ask yourself when considering load testing:

• Do you experience problems with performance in production?
• What is the cost of downtime, including monetary, person hours, opportunity cost, customer satisfaction, and reputation?
• Does your application scale with an increase of users?
• Do you have a method for obtaining real performance metrics?
• How do you repeat/reproduce a performance problem?

Defining exactly what you want to get from this type of testing is fundamental. In a comprehensive approach, there are some major questions that have to be considered:

• Who are your end users?
• How can you monitor their experience with the system?
• How can you translate these measurements into solutions?
• What tools and methods can help?

With the answers to the above questions you get started with:

• Strategy and Planning
• Define your specific performance objective.
• Specify the types of users to generate the necessary load.
• Define the scenarios that simulate the work and data flow.
• Define how the scenarios will be measured and tested.
• Define the repository for storing the data to be collected.
• Plan your test environment.
• Identify appropriate tools.
• Development
• Develop/customize test scripts which simulate your user's behaviour.
• Configure your test environment.
• Execution
• Execute your test scripts to simulate user load.
• Monitor the system resources.
• Result Analysis
• Analyze and interpret the results.
• Isolate and address issues.
• Tune your implementation.
• Plan for future marketing requests.

September 2002, by Trevor Atkins

The credit for developing the concept of Total Quality Management (TQM) is given to Dr. W. Edwards Deming, who was requested by the Japanese government of 1950 to come and assist them with turning around the public perception of the poor quality of Japanese products. Following his principles of "total quality control" to continually measure and correct your progress in achieving customer service goals, in less than a generation, "Made in Japan" became associated with quality product.

Sometimes Deming's concepts are referred to as Continuous Quality Improvement (CQI) to reflect that improving quality is a continuous process following the never-ending cycle of "Plan-Do-Check-Act".

Benefits of Quality

Having a high quality product translates into a large number of benefits to the organization such as:

• Less time reworking the code and re-testing interim bug fixes and patches.
• Effort for product updates can focus on new features rather than bug-fixing.\
• Low levels of technical support calls.
• No refunds and recalls of the product.
• Lower expense of supporting multiple versions of the product in the field.
• Good publicity rather than bad.

With a product recognized as high-quality, the number of referential accounts, total sales and customer goodwill will be much higher.

Costs of (Poor) Quality

In "Quality Cost Analysis: Benefits and Risks" (Software QA, Volume 3, #1, 1996), Cem Kaner defines quality costs as those costs associated with preventing, finding, and correcting defective work. He notes that these costs can be of the order of 20% - 40% of sales, and that many of these costs can be significantly reduced or completely avoided through the involvement of effective quality engineering.

In his article, Cem Kaner outlines four types of costs that when added together comprise the overall cost of the current quality of the application. The four types of costs are:

• Prevention - Costs of activities that are specifically designed to prevent poor quality including coding errors, design errors, mistakes in the user manuals, as well as badly documented or unmaintainable code.
• Appraisal - Costs of activities designed to find quality problems, such as code inspections and any type of testing.
• Internal Failure - Failure costs that arise before your company supplies its product to the customer. If a bug blocks someone in your company from doing their job, the costs of the wasted time, the missed milestones, and the overtime to get back onto schedule are all internal failure costs.
• External Failure - Failure costs that arise after your company supplies the product to the customer, such as customer service costs, or the cost of patching a released product and distributing the patch. External failure costs can be huge. It is much cheaper to fix problems before shipping the defective product to customers.

Total Cost of Quality: The sum of the costs: Prevention + Appraisal + Internal Failure + External Failure.

How Testing Fits In

As discussed in "Test Throughout the Development Lifecycle" testing is much more than just finding bugs to squash. It is not an event, but a set of diverse activities capable of playing a critical role in identifying problems of varied types throughout the project lifecycle, far in advance of public access to the software.

Tracking the real costs of software failure such as patches, support, and rework can be difficult, but it is clear that effective testing can help optimize Cost of Quality. Performing reviews, thoughtful test planning, risk-based testing, and strategic automation on your software before it goes to market is a direct investment in the product.

An organization can reap significant returns through investment in Prevention and Appraisal activities - such as QA and Test.

Outsourcing as a Tool for Improving Quality

Put this question to your in-house team and contract service providers:

• How are they helping you keep and satisfy your current customers and attract more?

Put these questions to yourself:

• How would you know if your current Cost of Quality could be reduced by 20% through a handful of "Quick Wins"?
• How would you know if you could get more quality out of your current project budgets?
• How would you know if your current testing solution can do more for you?

In "Roadmap To Successful Outsourcing", Wolfgang Strigel notes that outsourcing parts of software development and maintenance activities is becoming a competitive imperative and that according to research by the Gartner Group, 75% of all Information Technology (IT) companies will outsource parts of their IT efforts by 2003.

In his paper, Wolfgang Strigel also points out that outsourcing is a tool in an organization's toolbox that allows one to:

• Realize significant cost savings with respect to resources. Companies can take advantage of global resources to cut costs while retaining the permanent foundation of expertise in those capabilities that represent their competitive differentiation.
• Use their core staff for strategic work while contracting out other activities. This improves company focus and increases margins by using specialty knowledge only when needed, and avoiding make-work projects for in-house staff who are suddenly not on the critical path.
• Shorten delivery cycles and get quick reaction and greater flexibility to ramp resources up or down as project needs and market conditions demand.
• Access the experts in a Just-In-Time fashion through a reliable pool of freelance or outsource contractors. Increasing complexity and specialization of new technologies make it difficult for companies to have in-house expertise in all areas.

Making Outsourcing Work

Building long-term relationships with a service provider does not mean anything as dramatic as a switch from the current solution to a new one. Serious vendors of contract services will want to participate in an evolution and growth of services that are backed-up with proof by performance while working within and improving the organization's current project process framework.

Starting with a sample or "pilot project" of a controlled size and scope will allow you to measure the results and judge the potential value of the vendor for the future.

June 2002, by Trevor Atkins

What is COTS?

COTS stands for "Commercial Off-The-Shelf" and is often used in reference to software products or tools supplied by third-party vendors. These tools are created to help you in automating some or all of your testing, or to increase your ability to perform specific types of testing.

Another option for automation is to consider scripting languages. Scripting languages offer powerful tool kits that can be used to accomplish many of the testing tasks offered by many off-the-shelf tools.

Regardless of which you use, approach your automation effort as a serious development project and the advantages can be greatly leverages and the disadvantages largely mitigated.

Below we provide you with some analysis of the advantages and disadvantages of both.

Advantages of Commercial Testing Tools

• Shorter Initial Implementation Cycle: many COTS test automation tools are quick to install, configure and use, and thus COTS generated tests that can be more quickly added to your testing cycle than individually created scripts.
• Availability of Specific and Customized Training: training, provided by the vendor, is generally available although sometimes at a prohibitively high cost.
• Integration with Other Tools: many COTS tool vendors make many different tools, not just test automation tools. If you select a COTS test automation tool from a leading tool vendor, it is very likely that they also offer additional tools such as test case management tools, defect tracking tools, source control tools and even requirements management tools. These "suites" of tools can, when used effectively, streamline the flow of information between these tools and the people that use them.
• Mature Product Functionality: most major COTS test automation tools have been evolving along with the industry over the course of many years and will contain many of the features and functionality automators are looking for.

Disadvantages of Commercial Testing Tools

• Cost: Purchasing and licensing COTS test automation tools can be quite expensive. You need to consider also the cost of the support contract, which is can be between 15 to 30% of the cost of the tool license per year.
• Complexity: many COTS test automation tools provide a large amount of functionality and support a wide variety of technologies and environments, but in reality you end up using very little of the functionality and do not need all environments supported. Accessing this advanced functionality can require significant programming skills in some cases.
• Value (functionality as compared to cost): face=Verdana>most organizations need to see a return on investment (ROI) - the money and time spent on automation using a COTS test tool. Sometimes, the ROI for a COTS test automation tool can be so lengthy (often over 2 years), that it is simply not worth an organization's effort to implement such a tool choice.
• Slow adoption of new technologies: with each new technological advance, the tools must be extended to handle these new technologies. This can often be six months to two years later than when the new technology first appeared on your horizon. In the meantime, you will have only a partial solution or perhaps one that has serious defects or deficiencies.
• Proprietary scripting languages and data formats: face=Verdana>most COTS test automation tools store data such as test results and script details in a proprietary format. Some of these formats are usually "flat" but even so it can be a problem if you want to change tools and you do not want to have to copy and paste all of your scripts. Test results and other related data could also be difficult to extract. Most COTS test tools will have some reporting functionality built-in, but this may not be what you want or need in all cases.
• Environment and Configuration Problems are Common: it is a common occurrence to find that scripts you created, that run just fine on your environment, will fail on your fellow tester's environment. Many COTS testing tools rely heavily on specific environment information, such as operating system, screen size/resolution, color settings, locations of network connections or installed programs. Some tools will hard code this specific information into your test scripts, while others incorporate this information into the tool's operating environment ("settings").

Advantages of Scripting Languages

• Cost Savings: A Perl license costs zero dollars. A Java license costs zero dollars. When you compare the cost of licensing a tool (money) and learning how to use it (time) versus learning how to script effectively (time and maybe some money for books or courses), you can spend less on the scripting alternative.
• Flexibility: scripting languages allows automation to grow as your product grows supplying the needed test-oriented functionality. The functionality of your automation is entirely up to you. Whatever testing you want to do, and however you want to do that automation, you have virtually unlimited possibilities. Also scripts are easy to learn, expand and enhance when written in languages such as Perl or Visual Basic (VB), for example.
• Portability: scripting languages can be easily used in different operating systems; scripts are portable between versions of scripting languages.
• Non-proprietary formats for data/test scripts: you can determine what format data is stored, and how the test parameters are stored, read and written. The data is yours; you can control the formatting, the storage, and the retrieval of it.  You can also control any real-time data reduction or analysis.
• Easier integration with new(er) technologies: face=Verdana>while tool vendors have to "catch up" to new technologies, scripting languages, like programming languages, are at the front of the technology curve. They are the first things to change (followed by OS's, then applications to run on the OS's) and to incorporate new technologies.
• Many more resources available: there are lots of books, on-line books, forums and user groups, etc. available. Plus you can get some great help from your internal developers. They are usually more than happy to help you out with a scripting or programming problem. In some cases, it can be easier to get assistance from development with a specific aspect of your automation if you are using scripting, as opposed to a COTS tool with which most developers are unfamiliar (and rarely have the time to learn).

Disadvantages of Scripting Languages

• Steep learning curve: although we feel that most scripting languages are not that difficult to learn, this depends heavily on the person learning to script. If they have any prior scripting or programming experience, the time spent learning the new scripting language is far less than that of someone who has never done any scripting or programming. Also there are people who find this kind of abstract thinking to be difficult, new or just plain annoying. Many people feel that it simply takes too long to accomplish what they can by using a COTS tool.
• Methodology: to take full advantage of using scripting languages for testing, and for those scripts to be effective, you need a deep understanding of both test methodology and software engineering principles. Not every organization has such a dually-skilled resource.
• Functionality: when you create your own scripts, you do not have the help from wizards or report templates. Though you can leverage existing libraries, you will have to create your functionality from scratch.
• Testing: just like any other piece of software, you need to test your scripts. You need to verify that the scripts are performing as expected, and that the results or other data that the scripts generate are meaningful. You need time to prepare data and testing. You may need other resources to help you with testing so you can get a different perspective.

Summary

There is no true "silver bullet" to your test automation needs. Approach the tool selection process with your eyes wide open and be aware of the wide range of plusses and minuses you must consider in making your choice.

May 2002, by Trevor Atkins

One of the obvious truths about software testing is that there will never be enough time to do all the tests.

While everything should be tested, limited resources and schedule require that testing is structured whereby the most critical areas of the software are tested first and most thoroughly.

Performing risk-based analysis on the software will allow intelligent choices to be made regarding what tests need to be run in a time crunch situation. It ensures that the choice between what is tested and what is not is a conscious decision based on an understanding of the technology and the users.

Taking the understanding of "what is risk?", answer the following questions with as many relevant points as possible:

• In what areas is the user most likely to experience a problem?
• What would be the impact of a certain type of problem?
• What is the relative testing priority of each type of potential problem?

Get the ideas of the end-user, the application architects, the developers, customer support - ask lots of questions and dig up internal historical data or industry metrics.

Once the risk areas, impact and priorities have been defined, a strategy can be conceived for how to approach the overall test effort that will maximize the value of your testing budget.

Having this test strategy in advance of actual test execution means that what to test and how long it is estimated to take is known: resources can be obtained, test environments can be set-up, test cases detailed, and test harnesses constructed.

Plan to concentrate most of your testing and attention on areas with known complexity, usage patterns, and technical risk. The accuracy, reliability and readiness of the application for release can be measured much more accurately than just through overall bug counts.

Look at the schedule and determine how much time there really is to spend on testing. Identify and prioritize the risk areas of the application.  Create a test strategy that maximizes the resources and time available.  Remember that risks can change throughout a project and so the test strategy must change to deal with them. Don't get locked into the original test strategy as the _only_ strategy that will work for the project.

In the case of shifting milestones, the strategy can be easily modified to suit the time left and the level of risk that is acceptable when _planning_ not to execute certain tests.

The testing efforts are now focused on portions of the software where the risk of potential failure is greatest, or where potential failure would have the greatest impact. The result of this testing approach provides everyone with confidence in how well the software will hold up in the hands of the end-user.

Summary

With the increasing complexity of modern software and the shrinking time line for development and testing, it is important to get the most out of your testing efforts. With the understanding that it's impossible to test an application and find every bug, the technique of risk-based testing provides the coverage necessary to find the most important (and most costly if not found and fixed) bugs with the least effort.

April 2002, by Trevor Atkins

A Test Strategy is a documented approach to testing where the test effort, test domain, test configurations, and test tools employed to verify and validate a set of functionality are defined. It also includes information on schedules, resource allocations, and staff utilization. This information is crucial to allow the test team (Test) to be as organized and effective as possible.

A Test Strategy is not the same as a Test Plan, which is a document that collects and organizes test cases by functional areas and/or types of testing in a form that can be presented to the other teams and/or customer.

Both are important pieces of the Quality Assurance process since they help communicate the test approach scope and ensure test coverage while improving the efficiency of the testing effort.

What is in the Test Strategy?

The following is a list of some of the sections that are typically included in the Test Strategy document.

• Introduction - contains an overview of the project, lists related documents and references, document conventions, and assumptions made in the course of creating the strategy.
• Scope - describes the scope of the test team's involvement in the project; describes the test areas Test is responsible for and why. It also defines the areas for which Test is not responsible.
• Resources & Schedule for Test Activities - describes the resources available and their roles. Includes a schedule overview for the project, making sure the estimated time for the testing activities and milestone dates are present. The build schedule can also be included if available.
• Acceptance Criteria - defines the minimum criteria that a product must achieve before it is shipped.
• Test environment - describes the hardware and software platforms that are used for testing, including Client/Server configuration, Network, etc...and what will be tested in each platform.
• Tools - describes the tools used for test case management, defect reporting and test automation.
• Test Priorities - describes the priorities of the test effort during the test planning, test automation, test data creation, and test execution phases.
• Test Planning - describes such activities as requirements review and test analysis to determine a list of suitable tests required for verification of the product. It also describes how the tests are expanded into full test cases, complete with descriptions, reproduction steps, and expected results.
• Executing a Test Pass - describes how the test pass execution is performed, and when the testing is executed, in accordance with the types of testing to be performed. For example, test cases that are critical are tested first to ensure the build has the minimum functionality required before further testing.
• Types of testing to be performed - defines the different types of testing to be performed, and the extent to which Test will be carrying out each type of testing. The most common types of testing types are:
• Build Verification Tests
• Functionality Testing
• User Interface Testing
• Usability Testing
• Error Handling
• System Platform
• Stress Testing
• Performance Testing
• Installation Testing
• Print Testing
• Localization Testing
• Regression testing
• Risks and Issues - lists of outstanding risks and issues related to the test effort.

Benefiting from the Test Strategy

The main groups that benefit from having a Test Strategy are the testing team, development and project management, but other groups such as user ed and marketing can also benefit from the information contained in the Test Strategy.

• Test Team: The Test team will follow the Test Strategy and make sure testing is performed in accordance with the plan. They will also analyze the results and make recommendations on the quality of the functionality. The Test Strategy document should help the Test Team answer the questions below:
• Do I have all documentation I need to start the test planning?
• Is the time scheduled for testing planning adequate?
• Do I have the tools to develop the test cases? To log defects?
• Who is going to review the test analysis/ test planning and when?
• Do I have all I need to start testing (equipment/tools)?
• Do I have all the data/files I need to start testing?
• Do I know the functionality I will test on each build?
• Is all functionality being covered during all phases?
• What are the procedures if I find a serious defect?
• Development: Development will understand the functionality being tested and the conditions under which these tests are to be performed. The questions that the Test Strategy document should answer are:
• What is the overall approach to testing on this project?
• Who is responsible for the different types of testing, particularly Unit and Integration testing?
• Do I have time scheduled for reviewing the test plans for depth of coverage?
• Is the test environment adequate relative to the intended production environment?
• Project Management: Project Management will understand the information regarding configurations (hardware and software) under which the product was verified and validated, and the procedure for assessing the quality of the product based on the type of testing being performed. They are also informed about the testing schedule and its impact on the deadlines. The Test Strategy should help with the following questions:
• Do I need to hire more people during the test planning or testing phase?
• Do we have all the hardware and software required for testing?
• Do we have the tools required for test planning and defect reporting?
• If a new tool is required, is the time needed for training the testing team scheduled?
• Are all types of testing defined as required?
• Are all the testing tasks well defined?
• Are the testing priorities clear for each phase of the project?
• Are there enough test execution passes for each phase of the project?
• What are the issues and risks identified by Test that are still outstanding?

There is another important document whose purpose is very often confused with the Test Strategy or Test Plan; and that is the QA Plan. The QA Plan is intended to be a high level document that clearly outlines the boundaries of QA's responsibilities on the project relative to the rest of the project personnel, including any clients, sub-contractors, and co-contractors.

The QA Plan includes descriptions of methodologies, practices, standards, quality requirements, metrics, reviews and audits, code control, media control, etc., in addition to outlining the basics of the responsibilities of the Test Team.

The Test Strategy draws upon this parent document and its information, if available, and further details the responsibilities of the Test Team and its approach to testing.

March 2002, by Trevor Atkins

There are many perceptions of what software testing is all about. Some people see software testing as something developers do as they write code. For some testing is performed all at once when the code is complete. Others believe that testing should be performed along-side development activities.

It is a well-established fact that the sooner a defect is found the less expensive it is to fix. Consider the definition: "Testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results" - The Complete Guide to Software Testing, Bill Hetzel, 1988. Testing is a set of continuous activities performed with the objective of evaluating the software against pre-defined results/expectations.

Testing the application throughout its development lifecycle, emphasizing the early lifecycle activities helps increase the overall effectiveness and efficiency of this effort. Further, early involvement allows evaluation of important planning, design and development decisions with respect to how these decisions aid or impair the testability of the application.

Stages of the Software Development

Test resources have at least one associated activity within any phase of a project lifecycle regardless of the development methodology followed:

Project Planning - When the project plan and associated schedule is created, Testing reviews the plan to ensure that all testing tasks have been included and have been scheduled for an appropriate length of time. As well, assigning resources to these tasks will allow any potential resource allocation problems to be raised at the beginning of the project while there is still time to do something about any shortfall. Testing will also be a critical participant in initial and on-going Risk Analysis/Management.

Requirements and Specifications - While the requirements are being gathered and written, Testing begins writing a Test Strategy to outline the overall approach to testing and related assumptions and constraints. Testing also reviews the requirements for clarity, completeness, ambiguities, and testability. This means preventing defects from occurring in the code and having to be found during a future testing phase, when it is more expensive to fix.

Design - While Development is documenting an architecture and detailed design for the application, Testing continues Requirements Analysis and begins creating Test Cases (for inclusion in the Test Plan) and any other Test Assets (test coverage matrices, test data, test automation frameworks, etc.). Testing also reviews the design documents for accuracy and testability. Design for testability and the easier all areas of the application can be tested - improving quality while saving time and money.

Coding (unit) - Once coding begins, Testing must work towards completing Test Cases, Test Plans and Test Assets. Identification of Build Verification Test Cases (BVTs) and prioritization of the remaining Test Cases are completed at this point. Testing begins performing some unit testing (developing scripts and automation where applicable), depending on the scope of testing outlined in the Test Plan.

Coding (integration and stabilization) - As integration begins, Testing can begin executing BVT Test Cases as part of "Smoke Testing" or "Stability Testing". Testing will then move on to high-priority Test Cases in areas where sufficient development has occurred. As integration progresses, Testing is able to test more and more of the application's functionality, reporting all defects. When the application nears functional completeness, Testing completes one or more test passes (execution of all Test Cases). Finally, focus shifts to system, scenario, stress, and performance testing. If Testing did not have all information available before or the requirements have changed, test execution and creation and/or updating test case steps may have to occur at the same time.

Coding (code freeze and bug fixing) - During this phase, Testing's primary activity is regression of defect corrections. Testing also participates in the triage of defects (reprioritization to reflect risk, schedule slips, or removal/addition of features) as needed.

User Documentation - Testing reviews all user documentation (manuals, help files, etc.) for accuracy, usability and completeness.

Release - Testing can sometimes be responsible for the final builds that lead up to the release. When the release decision is made, Testing collects and archives all project data, documentation, source and tools. Testing also participates in the project Post Mortem, where problems and solutions encountered in a project are reviewed for learning purposes.

Support and Maintenance - As externally-found defects are reported to the support personnel, defect reports are analyzed by Testing to determine if the defect is reproducible and whether it was known prior to release or not.

Starting test activities early means you can catch small quality problems before they become big quality problems later on. Review plans and designs for weaknesses before the software is ready to be tested. Test the software as soon as it is available and log those bugs all the way through the project.

February 2002, by Trevor Atkins

One of the risks of any application is being able to do enough testing in the time available. Another common risk is getting at those critical components behind the GUI to ensure that they are as bug free as possible before and after integration with the rest of the system.

Black box testing can tell you whether the GUI layer is functioning as it should, by taking in user input and responding with a result. But what if there is an error, how do we know what layer (GUI, Application, Database, etc) is creating the error? What if our black box tests don't result in any database or application layer bugs? Does that mean there aren't any? What if the GUI layer is masking the errors?

The tester needs to avoid the risk of automating a fluctuating GUI and at the same time provide the volume of tests needed to ensure that the application and database layers are stable and functioning.

Test Data Creation

In order to begin performing data driven testing you first need to create good test data. Sometimes sources for this data are available from previous testing efforts, from users of the previous versions of the application, or perhaps obtainable by purchasing test suites from a vendor. More often than not, this test data needs to be created from scratch in the specific context of the application to be tested and the test plans to be executed against it.

Sending the Message

Once the database is populated with your initial data, you will want to take that data and construct a message, command, or request to send to the application that you are testing.

Constructing the message can be as simple as extracting the information from the database and putting it together in the order you need. Or there could be additional, more complex, logic required to construct the message that the application needs to receive. Working with scripts and databases together can give you a very versatile and useful solution.

Getting Your Results

To get the results of sending your message to the application, you need to be able to collect the result codes or expect your application to send back a response. Assuming the latter is the case, your test tool would wait for that response, process it as necessary and then compare aspects of the response to the value in an Expected Results column in your tool's database. Depending on the result, a Pass or Fail can be determined and a log entry created with as much detail as you care to capture.

Example Testing Problem

To pull all of what we have talked about so far into a tangible example, let us imagine a certain application that we want to test.

The application is a multi-tier web application that passes information from its User Interface layer to its Application Layer, via XML messaging. These XML messages are made up of a number of fields of all different types. The application takes these messages, processes them, and decides what the appropriate response is - return the information requested, perform the required action, etc.

If we assume that a typical message can contain 20 data elements or fields, and each data element needs to be tested with four different values: valid, error, upper field length boundary, and lower field length boundary: we need to perform over ONE TRILLION tests.

4 to the 20th power = 1,099,511,627,776

This number of fields is not unreasonable, and the different values we mentioned above do not include every case that may need to be considered depending on the constraints for the message's data elements. Are all the fields required? Can they be blank? What about surrounding whitespace characters, are they stripped away for each field? What about fields that take codes like ON/OFF or RED/GREEN/BLUE/BLACK/WHITE? Each field will need a certain number of tests based on its constraints.

This is where you need to make use of Equivalence Classes and other test planning techniques to determine the minimum number of tests for each field. For example, if we can assume that we only need to vary the input of 15 fields out of the 20, and that each field contains only either valid data or invalid data, the number of combinations of test data we now need is much less.

2 to the 15th power = 32,768 <-- still a lot of tests

This is obviously a coarse example, but it makes the point. What if you had 30 potential data elements in your message? Generating this amount of data, executing the tests, and reviewing the results takes a huge amount of time.

Always, make sure that you have well designed test cases before undertaking your automated testing. A little upfront analysis and planning can save you a lot of work.

Defining Your Test Tool

Before creating or purchasing any software application or test tool, the first thing one must do is to collect and enumerate the requirements.

In general the tool needs to:

• Create and/or populate a database based on a predefined schema including an "Expected Results" column that adequately populates the database based on the elements created
• Create a 'message' or formatted string, delimited by tabs or in XML, for sending to a network location, where the application under testing is waiting for connections and incoming requests or data to be delivered
• Send the message to the network location monitored by the application being tested
• Listen for a response from the application
• Compare the response to the value in the appropriate row in the data table
• Report on the success or failure of the test based on the result of the comparison of the actual response received and the corresponding value in the expected result column

Before stating that the basic requirements are complete, it would be much more useful if the tool could execute thousands or millions of tests. The time for these tests to complete could be extensive. It also may be that the tests should really be run at times of the day when there is low usage of the application or low network traffic. Both of these needs require that the tool is able to run unattended and perhaps is able to start up at a scheduled time.

But remember, if you are building your own tool or even buying one, keep it simple. After all this isn't the product you are trying to ship. The first priority is to make the test process work better than it is working right now. You can add in other bells and whistles later.

Summary

The framework required for any basic data driven testing tool is made up of three main components: File Input/Output for reading from configuration files and writing reports, a database for storing the test data, and an engine with which to extract the data from the database and make meaningful directives and requests of the system under test.

And that's that. With a tool comprised of these core components, you can send potentially trillions of messages to an application for testing purposes - far more than you could, or would ever want to do manually.

August 2001, by Trevor Atkins

Risk management can be defined as creating and maintaining plans for controlling real and perceived risks paired with monitoring the effectiveness of those plans.

DO NOT wait until a risk becomes reality before deciding what to do about it. A software development project contains many elements of uncertainty that manifest themselves as risks. And in most cases it will be too late to do anything about them if they become reality. You end up managing the consequences rather than the risk. Risk management allows you to be active in attempting to predict what could go wrong and in addressing those potential problems as early in the project as possible. Remember Smoky the Bear: You too can prevent forest fires. Or in this case, risks becoming reality.

Risk management needs to start at the beginning of a project to be of benefit. Risks feed into the project plan and help determine how you run your project while trying to address the risks in order of priority. Managing risk early is almost always less costly and painful than cleaning-up after the fact. However, introducing risk management at any point in your project will give you some benefit; it is better late than never.

The process of risk analysis and mitigation is comprised of three phases: risk identification, risk estimation and evaluation, and risk planning. These three phases allow one to identify the risks, judge their potential impact, and plan to avoid or minimize the risks. Identifying risks allows you to evaluate and plan for them, and to be prepared to respond when mitigation strategies fail. Much of learning to judge risks and planning ways to address them comes with experience; it is an acquired skill.

The Risk Management Plan

The Risk Management Plan details how to manage the risks associated with a project. It details the risk management tasks that will be carried out, assigned responsibilities and any additional resources required for the risk management activity. In a smaller scale project, this plan may be simply a "Top 10" risk list that is reviewed at regular project status meetings.

The first step in developing your Risk Management Plan is to brainstorm the current risks.

No matter what stage your project currently is in you need to know the kinds of risk you are faced with now, and most likely will be faced with throughout the rest of the project, before you start trying to decide upon strategies for managing those risks. Create an initial list of risks and use this list to prioritize and monitor the risks throughout the project. The risk list should be reviewed regularly to keep it up to date and to evaluate the effectiveness of risk mitigation strategies. You will find the results of these reviews can drive revisions to the project plan itself.

Next, you have to make someone responsible.

Someone on the project needs to be responsible for collecting and recording risks as they are identified, tracking status, and scheduling review meetings. Also a group of concerned stakeholders should be identified. These stakeholders are drawn from both managerial and technical personnel on the project, and should include the project manager, the leads for the test and development teams, and the product manager or customer representative. This group needs to consistently attend the risk review meetings.

Once you have your initial risks recorded and someone has taken responsibility for keeping track of those risks, identify an attribute of the risk that you can measure for each risk.

You can pair these measures with indicators, or thresholds, that tell you when each risk is about to (or has) become a reality. Tracking these trends will help take the guesswork out of whether your mitigation strategies are working. And they will allow you to see how much room you have. The approach that will be used to monitor and reduce each risk should be clearly documented and accessible to the project team. The approach should also outline a contingency plan in case the risk occurs despite any mitigation efforts.

Finally, you need regular reviews.

It is extremely important to realize that risk management is only effective if it is an on-going activity. All too often a project team will produce the initial risk list, update it once, and then put it on the shelf and forget about it, becoming embroiled in fighting the fire of the day. The Risk Management Plan should outline a schedule for regular risk status reports and risk review meetings.

July 2001, by Trevor Atkins

Risk is often defined as the combination of the likelihood of a problem occurring and the impact of the problem if it were to occur.

• Risk concerns future happenings.
• Risk involves change in mind, opinion, actions, or places.
• Risk involves choice and all the uncertainty that that choice itself entails.

Risk is something with which we are all somewhat familiar. We know that it is risky to jump out of planes, or invest money in the stock market. But we don't always think of software development as risky - particularly if our software is not mission critical (that is, people's lives don't depend on it). While insurance companies and actuarial analysts spend most of their days dealing with risk, software development teams often spend a mere fraction of their project schedule thinking about risk.

In this series, we introduce you to some basic concepts and principles of risk management, and how these concepts and principles can be applied to software development projects to increase the probability of success.

In everyday life a risk is an exposure to the chance of injury or loss - a thing or course of action involving uncertainty and therefore danger. A risk, in the software industry, can be anything that may threaten the successful completion of a project: increased costs, delayed completion, loss of quality, or loss of market-share.

Two factors can be used to determine the magnitude of a risk: likelihood of occurrence, and impact of occurrence. When combined into a single indicator the risk can be judged relative to other risks as High, Moderate, or Low.

Many organizations still do not identify risks at the beginning of a project, or review and update those risks throughout the course of the product lifecycle. Estimating and planning is done as if all variables are known, and outcomes certain.

The reality is that risk results from changes in development personnel, market conditions, customer expectations, and even business conditions for the development organization.

Experience shows that the large majority of risks have a direct or indirect impact on schedule, and therefore implicitly on cost. Some have a direct impact on cost. The rest may have no direct impact on cost or schedule, but affect other areas of the project such as quality.

Some common risks include: the technical requirements of the product or system, constraints placed upon the project or product by the customer(s), budget, or schedule.

Types of Risks

It's important to distinguish between direct and indirect risks. A direct risk is one over which some degree of control is possible whereas an indirect risk is one that cannot be controlled.

While one should not be unaware of the indirect risks, they are of little consequence in a practical sense: since one cannot change them, there is little to be gained by worrying about them.

Sometimes, an indirect risk may actually be a direct risk. For example, you may be dependent on an external supplier for a component or platform for your product. This appears to be a risk that cannot be controlled. But, by having contingency plans for those components, you can take control of the risk: develop a relationship with the supplier so they listen to your needs, choose alternate suppliers, or choose to develop the needed components yourself.

With indirect risks, you either have to gain some degree of control, or you need to just make note of them. Having them documented will allow you to easily revisit the risk and the decisions that were made as necessary.

To help determine if your project would benefit from Risk Management, look at your project and ask yourself the following questions.

• Is the project scope firm or does the scope keep expanding?
• Are estimates accurate?
• Is there time to "do it right"?
• Has the technology been proven?
• Are the users of the system experienced with the type of system being developed?

These are just a few of the questions to ask yourself when looking for potential risks related to your project. If your answer was "No" or "Unknown" to any of them, it means that you are facing at least one risk to your project being a success.