TestNG – running tests in parallel

If you have several suites to be run, you could run them in parallel at the same time in different threads.

Parallel suites

In command line the ant task suiteThreadPoolSize can be used to run several suites in parallel using org.testng.TestNG as shown below:

java org.testng.TestNG -suiteThreadPoolSize 3 testng1.xml testng2.xml testng3.xml

This command runs the three suites in 3 different threads in parallel

Parallel methods, tests, classes

In testng.xml file, using the attribute parallel inside the suite tag instructs TestNG to runs tests in parallel in different threads. The attribute parallel accepts certain values as outlined below:

<suite name="mysuite" parallel="methods" thread-count="5">
<suite name="mysuite" parallel="tests" thread-count="5">
<suite name="mysuite" parallel="classes" thread-count="5">
<suite name="mysuite" parallel="instances" thread-count="5">

1. parallel=”methods”: TestNG will run all the test methods in parallel in different threads. The dependent methods will also run in different threads but would respect the order that was specified.
2. parallel=”tests”: TestNG will run all the methods in the same test in the same thread but each test will be run in separate threads. This allows to group the thread-safe classes in the same test so that they will run in the same thread while taking advantage of TestNG’s capability to run tests in several threads.
3. parallel=”classes”: TestNG will run all the test methods in the same class in the same thread and each class will be run in separate thread.
4. parallel=”instances”: TestNG will run all the test methods in the same instance in the same thread and each instance will be run in separate threads.

Also, the attribute threadPoolSize of the @Test annotation could be used to run tests in parallel as shown below.

@Test(threadPoolSize=5, invocation-count=10, timeout=10000)
public void testMethod()
{
  System.out.println("Hello");
}

This will invoke the method testMethod 10 times in 5 different threads. The timeout of 10 seconds ensures that the thread is not blocked for more than 10 seconds. The timout applies for both parallel and non-parallel mode.

TestNG groups using include/exclude

TestNG allows to perform groupings of test methods. The methods can be declared to belong to groups and TestNG can be invoked to include/exclude certain set of groups. This provides maximum flexibility to structure the tests in a more intuitive way.

Step1: Declare groups in java class

Suppose we would like to group the test methods into two categories functional and sanity. We declare the methods to belong to either of the two groups using the “groups” attribute of @Test annotation.

public class Test1
{
  @Test(groups={"functional","sanity"})
  public void testMethod1(){
  }
 
  @Test(groups={"sanity"})
  public void testMethod2(){
  }
 
  @Test(groups={"functional","sanity"})
  public void testMethod3(){
  }
}

 

Step2: Specify groups in TestNG.xml file

Suppose we would like to run the sanity tests alone. We specify the groups in the TestNG.xml file using the <groups> tag inside the <suite> or <test> tag as shown below:

<suite name="mySuite" verbose="1">
  <test name="Regression">
    <groups>
      <run>
        <include name="sanity"/>
      </run>
    </groups>
    <classes>
      <class name="sample.Test1"/>
    </classes>
  </test>
</suite>

TestNG.xml – A Quick Overview

TestNG is a testing framework designed to simplify the testing needs.

 

Quick overview of TestNG.xml file

  • A suite is an xml file represented by <suite> tag. The TestNG suite can contain one or more TestNG tests
  • A test is represented by the <test> tag. A TestNG test can contain one or more TestNG classes
  • A class is a java class having TestNG annotations. It is represented by the <class> tag. A TestNG class can contain one or more TestNG methods
  • A method is method configured with @Test annotation

 

Sample TestNG.xml file

 

 

Also, it is possible to use packages instead of classes in TestNG.xml file. For this package tag is used as shown below:

 

junit vs TestNG – core differences

Junit and TestNG differ in the core design. Junit is a unit testing framework while TestNG addresses testing at a higher level. This article discusses the three main differences between TestNG and Junit.

1.     In Junit, one test case failure can cause a bunch of test cases to fail in the test suite. There is no option of skipping the set of dependent test cases. The dependent test cases are also reported as failures. For example, suppose there is a test case to test login and the next 10 test cases need to perform a transaction after login. If the login test case fails the other 10 test cases will also fail.

TestNG handles dependency between test cases. If one test case failure causes the failure of a group of test cases it skips that group and executes the rest of the test suite. The group that has dependency on the failed test cases is reported as skipped NOT failed.

2.     In TestNG groups can be defined. Groups are specific subsets of the test suite. We can choose to run only specific subset of the testsuite say database related test cases instead of running the entire test suite. This can be done as below:

In the test case we define two groups DBTestcase and deprecated as below:

@Test(groups = {"DBTestcase", "deprecated"})
public void testMethod2()
{
 
}

In TestNG.xml config file we write the below piece of code to include only database related test cases and exclude the deprecated ones.

<test name="Sample">
	<groups>
		<run>
			<include name="DBTestcase" />
			<exclude name="deprecated" />
		</run>
	</groups>
	<classes>
		<class name="example1.Test1" />
	</classes>
</test>

In Junit for a long time it was not possible to run a specific subset of the test cases. We can either run the entire suite or run each test case individually. Junit 4.8 introduced a new feature called “Categories” to overcome this limitation. However groups are much easier to configure in TestNG.

3.     TestNG supports parameterization for objects. For example I can execute a single test case for multiple test data sets through the parameterization of DataProvider object. This makes the implementation of data driven testing more flexible in TestNG as below:

@DataProvider(name = "DP1")
public String[][] testData() throws Exception
{
String[][] sdataArray = {{"james","moosecat$", "james@gmail.com","selenigr","selenigr1$", "selenigr@yahoo.co.uk"}};
return sdataArray;
}
 
@Test (dataProvider = "DP1")
public void evernoteSharing(String user1, String password1, String email1, String user2, String password2, String email2)
{
 
}

In junit test data is not parameterized using a DataProvider hook. Data-driven testing is implemented differently and involves more coding.

In TestNG groups can be defined. Groups are specific subsets of the test suite. We can choose to run only specific groups instead of running the entire test suite.