Automated Testing Methodology

General Goals

• Tests should improve quality
• Tests should help us understand the system under test
• Tests should reduce risk
• Tests should be easy to run
• Tests should require minimal maintenance as the system evolves
• Tests should be executed as part of the build process if possible

Automated Testing Approaches

We recommend a balance approach to automated testing.  There are a variety of tools and techniques that can be used to automate testing. These techniques and tools include methods to facilitate automation of these types of testing:

• Unit tests
• Customer Test
• GUI Testing

For more information see Automated Testing Approaches.

Beginning or Reviewing an Automated Test Program (things you should do!)

The following ideas can be used with most development methodologies.

Understand your Goals

What are the goals of your automated testing program? These need to align with your overall quality goals. Some factors that may affect your goals:

• Are your planning to automate your regression testing?
• Are you planning to integrate automation into your development process?
• Are looking to solve a specific quality assurance issue; such as a calculation test harness?

Start to Organize

Select Strategy
Select the automated testing approach; it could be a combination of approaches based on

• Goals
• Type of system that we are testing
  • Legacy (a system with no automated tests)
  • Current Automation process in disarray
• Are we approaching this from the product life cycle perspective, for example can we change the development process, the requirements process
• Do we just need automation to solve a specific problem?
• Select the types of test to automate: unit, integration, business requirements

Select the tools based on:

• Strategy selected
• Types of tests to automate (Customer? Unit? UI?)
• Open source or Commercial?
• Does the tool work for our application?
• What is our budget

Define how we expect to manage the tests
Define standards or patterns for implementing the tests
These two items are necessary to facilitate maintainability
For example the Key Word approach is a pattern that can be used to reduce maintenance costs when using the UI based automation approach.

Some of the approaches would benefit from working directly with QA.

Act

The approach will affect who performs the actions:

• Developers will be responsible for creating and maintaining the xUnit type of tests; although QA can help
• A QA automation engineer can create the three other types of tests, as always it is best to integrate directly into the development process

Ensure that testing the tests is part of the process

Review your Automated Test Program

In all cases we review the effectiveness of the process and seek way to improve.

• Are the tests to slow?
• Are they hard to maintain?
• Do they slow down development?
• Are they effective in achieving your quality goals?