Tag: test strategy

“When developing our test strategy, we must minimize the impact caused by changes in the applications we are testing, and changes in the tools we use to test them.”¬† –Carl J. Nagle

This article evaluates the effectiveness of testing approaches and sheds light on the areas we need to concentrate on while testing. In essence 20% of the test cases should account for 80% of the ROI. For this, the testers need to question the logic behind the test case development by asking: 1) What is the probability of the occurence of bugs in this area? 2) Do I really need to automate this test? 3) What is the impact of GUI changes on the test case? The testers who are short-sighted tend to focus only on the testing milestones like pass rate/deadlines and fail to achieve the long term goals of testing like maintainability, robustness and coverage.

Credit: Flickr by e-strategyblog.com


GUI or End-to-End testing:

In this testing approach we try to simulate a customer completing some task involving multiple features and system interactions. This is the least effective test automation approach

1. Exposes Behavioral issues

1. Maintenance nightmare: Whenever the GUI changes the tester needs to revise the test case. GUI test cases are tied to the very trivial details of the UI that almost any change in the GUI can cause the test cases to fail. Thus, maintenance becomes tedious. Considering the time to fix it we could rather re-create the test case from scratch
2. Last minute design changes can cause havoc to the automation team
3. Longer test case execution time
4. Does not find much bugs

While developing the test strategy we must minimize the impact of UI changes on the test cases. I would recommend executing GUI test cases later in the lifecycle when the UI is stable and after all the critical fuctional bugs are exposed by the approach “Component based” testing explained below.


API or Component based testing:

In this testing approach we directly hit/communicate with the component to test the application. A component is an integrated aggregate of one or more units. We parameterize the properties and pass it as the input data to the API.  The corresponding methods are triggered on the data and it tests the underlying business logic/functionality of the application.

1. Maintainable: As opposed to End-to-End testing this is not dependent on UI changes and hence reduces the automation efforts.
2. Faster test case execution time and hence reduces the Mean Time To Test (MTTT)
3. Finds many bugs early on in the lifecycle. Exposes functional issues

The test plan defines the scope, approach, resources and schedule of the intended testing activities. It identifies the features to be tested, the testing tasks and who will do the tasks.

The three main elements of a test plan are:

  1. Test Coverage: The test coverage measures the proportion of the program exercised by the test suite. It is usually expressed in percentage. It helps in finding areas that are not exercised by the test suite. Having complete test coverage ensures 100% test effectiveness and that the test suite reveals all the defects
  2. Test Methodology: The approach, techniques and tools to be used for testing
  3. Test Responsibilities: Includes the features/functions to be tested and the criteria for success/failure for the tests

The Test Plan contains the following:

  1. The Project Overview
  2. Objective and scope of testing
  3. The test methodology
  4. Resources and responsibilities
  5. The testing efforts and schedule
  6. Entry and exit criteria for testing
  7. The features to be and not to be tested
  8. Success/failure criteria
  9. Test environment
  10. Risk factors and contingency plan
  11. References to the supporting documents which includes FSD, SRS, TDD and the like.

The scheduling of the testing tasks and milestones is done considering the below factors:

  1. Complexity of the feature
  2. Competency of the resource
  3. Stability of the requirements

Each of these factors could be rated as high, medium or low.

For large projects, there will be a single test strategy document. There will be many test plans that draws its content from the test strategy. The Test Strategy is a high level document that defines the testing approach to achieve the testing objectives. It is prepared by the project manager and it will not be updated often. The test plan is derived from Software Requirements Specification, Technical Design Document, and Functional Specification Document. It includes test strategy/test approach as well. It is prepared by the test lead. The test plan needs to be updated often to reflect any deviation from the original plan.

Credit:  Jurvetson (flickr)

If a feature is not to be tested the test plan should state why it is not. For ex: It could be: a low risk feature, tested and found to be stable in the past, not pushed in the current release.

There are several inherent software risk factors in a project like complexity or newness. A well rounded test plan notes all the risks and tasks interdependencies in the test plan and communicates to the stakeholders regarding those dependencies and risks.  Dependencies could be like cross-functional impact of a new feature, delay in getting stable builds, late change in original requirements, late delivery of fix and resource availability.

Once the test plan is documented it requires review and approval by the test manager /release management team in order to move the next phase of testing.