Archive for 'Testing practices'

This article brainstorms how design choices for better code testability fits into the big picture of providing a continuous stream of business value to the customer. Exploring the possible software design options even before the coding starts enables decoupling and dependency injection which makes the software all the more robust and flexible to absorb future changes in the customer requirements.

What is a unit? A unit is the smallest testable part of an application. Unit tests are written from the specification (FSD/TDD) based on how the product would be used by the end customer. It should be tested in isolation without any dependency. All the dependencies should be mocked up using stubs so that the unit tests are simple and fast. These tests are supposed to focus only on a specific action without any coding for the dependencies.

For example, the coder develops a method called validate_zip_code. When I enter  zip code as 00000 it should return false. This is a unit test.  As per the requirement spec, on the shipping form if the customer enters invalid zip code he should be re-directed to the help page that explains what is a zip code.  Imagine that this help page is yet to be developed by a different team called consumer team. Testing both the zip code field and the display of help page is not unit testing but functional/integration testing. This dependency on the help page display needs to be separated from the validation of zip code. This separation is called dependency injection and it decouples the code making it more robust. The dependencies could be mocked using tools like mockito or easymock. Thus the coder needs to create a mockup or a simulator to substitute for the dependency.

Why dependency injection can ease unit tests? With dependency injection writing unit tests becomes easy and the execution of unit tests becomes simple and fast. This is because the coders are not required to write test code for dependencies as well. Writing code for dependencies makes unit tests complicated and it is also unnecessary since we are not going to do any functional testing with it.

Advantages of Dependency Injection:
1. Unit testing before coding adopts Test Driven Development. The coders can brainstorm the possible execution paths and corner cases that he had never thought of before. This results in a higher quality code
2. Code with lots of dependencies violates the Law of Demeter or the Hollywood principle. With dependency injection the code becomes loosely coupled
3. Determines if the code is fit for testing and increases the testability of the code

As a Quality Engineer it’s a great experience being able to plan and facilitate design review sessions with developers and architects. This could either be a formal/informal session. The design review provide hints to the testers that help to foresee potential defects in the system. It comes with huge benefits of learning the crucial design elements especially when the discussion happens in a room of developers. At the same time, the developers also get a chance to brainstorm the design when the testers ask questions which inturn adds great business value to the product.
This article explores how fellow test engineers can contribute and influence the design decisions in a development effort. While the developers are better at Object Oriented Programming concepts, the testers can explore the feature at a deeper level identifying corner cases, exceptional conditions or error cases and recommend designs that are flexible enough to accomodate these behaviors.
Issues costing $300K occur due to bad product design that could have been prevented if there were design reviews with testers.
Below are two techniques that are explained indepth to help testers catch design flaws in a product:
1. Apply 5 basic design patterns
2. Analysis models

1. Apply 5 basic design patterns:

Design patterns are named set of problem-solution pairs for the repeated challenges faced by the programmers in the IT industry. Understanding Design Patterns requires us to “think in objects”. Being aware of the design choices and consciously making the best design choice is essentially a programmer’s skill and requires strong knowledge of Object Oriented Programming. Then, how does a tester add value to the design review of a product?
Testers can perceive where the potential issues and risks might be for a particular product design. Usually the unthinkable risks in the product design are not talked about in design reviews. This is the area where a tester can significantly add value. There are 5 basic business design patterns that can get the testers thinking about the unthinkable risks. Some of you might be already thinking on these terms while evaluating the design of a product. Making it more formal with process flow diagrams can make the design review session more useful to the stakeholders.


For understanding each of the below business design patterns let us consider a storyboard of a customer purchasing a product on a shopping website.
Before getting into these patterns first let us define what is an event.
An event defines under what circumstances a particular process/function gets triggered.
We can use logical connectors AND / OR called joins for joining two or more business events.

1) Dissection of events

This pattern helps the tester think: “What are the different starting events that can trigger a function/process?”
For example a Purchase Order Processing function can have different starting events as below:
Customer clicks on add to cart button on a third party shopping website
Customer places order via an email sent to his inbox
Customer places order via mobile phone


2) Sequential events pattern

What is the specific order in which the events should occur?
Should customers login before clicking on checkout? Can guest users also checkout?

3) Parallel Split pattern

What are the multiple processes that should be triggered by a starting event?
Lets assume that the customer clicks on the checkout button. Normally the purchase order process is triggered. What if the database crashes during the transaction? To handle this, the application should have two processes running. Periodic data migration of the transaction to backup database/servers so that the backup server can get a handle over the transaction in case of any crash and another is the payment processing function.


If the product is not designed to handle this crash then there would be a heavy data loss in the midst of the transaction since millions of customers might be using the site. These kinds of unthinkable risk can be better pointed out by a tester using this pattern.


4) Synchronization

When there are a set of activities, which of these must be finished completely before we continue the process?
Fraud check and confirmation of funding source/shipping address should be done before payment is processed.

5) Exclusive choice

What are the different mutually exclusive flows possible for a given process?
As stated in the earlier pattern, to which flow should the program transfer the control in the event of server crash?

2. Analysis Models

Analysis models could be used to probe, critique, review and approve the product design. Asking questions, even the stupid ones, during the design phase of the project can add great business value to the product.
Data models are a visual representation of the relationships between the different entities of a system. During the design phase of a project apart from the requirements specification documents there are class diagrams, use cases and entity relationship diagrams that are usually thought of as programmer’s tools. Similarly, the programmers think of it as a business analyst’s tool. Infact it can be used by both testers and developers to ask good questions that adds new perspectives to the design review discussion leading to the discovery of new facts or pitfalls in the product design.
1. Study the domain\business rules between the different entites
2. Know the right questions to ask
3. Identify pitfalls in the product design
4. Check for missing details in the requirements spec and discover new business rules
5. Filter out requirements that are likely to change

Seperation of concerns

Is the design flexible enough to accomodate the likely future change in the product? I just came across a GTAC (2010) video on Flexible Design? Testable Design? You Don’t Have To Choose! where the developer dumps multiple responsibilities in a single large class because he doesn’t like coding several small distinct behaviors each in a seperate class. This will cause issues in future when we need to change/extend/add one more behavior to the class. Whenever we modify this single large class the developer needs to modify so many methods calling them and so he will resist changes. It requires huge efforts even for a trivial change. This is an example of a bad design.
A tester need not agree with this kind of design pattern. The testers should filter out requirements that are likely to change in future using analysis models and ensure that the programmer implements “Seperation of Concerns” principle to make the design flexible for future changes for these set of requirements.

“When developing our test strategy, we must minimize the impact caused by changes in the applications we are testing, and changes in the tools we use to test them.”  –Carl J. Nagle

This article evaluates the effectiveness of testing approaches and sheds light on the areas we need to concentrate on while testing. In essence 20% of the test cases should account for 80% of the ROI. For this, the testers need to question the logic behind the test case development by asking: 1) What is the probability of the occurence of bugs in this area? 2) Do I really need to automate this test? 3) What is the impact of GUI changes on the test case? The testers who are short-sighted tend to focus only on the testing milestones like pass rate/deadlines and fail to achieve the long term goals of testing like maintainability, robustness and coverage.

Credit: Flickr by


GUI or End-to-End testing:

In this testing approach we try to simulate a customer completing some task involving multiple features and system interactions. This is the least effective test automation approach

1. Exposes Behavioral issues

1. Maintenance nightmare: Whenever the GUI changes the tester needs to revise the test case. GUI test cases are tied to the very trivial details of the UI that almost any change in the GUI can cause the test cases to fail. Thus, maintenance becomes tedious. Considering the time to fix it we could rather re-create the test case from scratch
2. Last minute design changes can cause havoc to the automation team
3. Longer test case execution time
4. Does not find much bugs

While developing the test strategy we must minimize the impact of UI changes on the test cases. I would recommend executing GUI test cases later in the lifecycle when the UI is stable and after all the critical fuctional bugs are exposed by the approach “Component based” testing explained below.


API or Component based testing:

In this testing approach we directly hit/communicate with the component to test the application. A component is an integrated aggregate of one or more units. We parameterize the properties and pass it as the input data to the API.  The corresponding methods are triggered on the data and it tests the underlying business logic/functionality of the application.

1. Maintainable: As opposed to End-to-End testing this is not dependent on UI changes and hence reduces the automation efforts.
2. Faster test case execution time and hence reduces the Mean Time To Test (MTTT)
3. Finds many bugs early on in the lifecycle. Exposes functional issues