TestNG – running tests in parallel

If you have several suites to be run, you could run them in parallel at the same time in different threads.

Parallel suites

In command line the ant task suiteThreadPoolSize can be used to run several suites in parallel using org.testng.TestNG as shown below:

java org.testng.TestNG -suiteThreadPoolSize 3 testng1.xml testng2.xml testng3.xml

This command runs the three suites in 3 different threads in parallel

Parallel methods, tests, classes

In testng.xml file, using the attribute parallel inside the suite tag instructs TestNG to runs tests in parallel in different threads. The attribute parallel accepts certain values as outlined below:

<suite name="mysuite" parallel="methods" thread-count="5">
<suite name="mysuite" parallel="tests" thread-count="5">
<suite name="mysuite" parallel="classes" thread-count="5">
<suite name="mysuite" parallel="instances" thread-count="5">

1. parallel=”methods”: TestNG will run all the test methods in parallel in different threads. The dependent methods will also run in different threads but would respect the order that was specified.
2. parallel=”tests”: TestNG will run all the methods in the same test in the same thread but each test will be run in separate threads. This allows to group the thread-safe classes in the same test so that they will run in the same thread while taking advantage of TestNG’s capability to run tests in several threads.
3. parallel=”classes”: TestNG will run all the test methods in the same class in the same thread and each class will be run in separate thread.
4. parallel=”instances”: TestNG will run all the test methods in the same instance in the same thread and each instance will be run in separate threads.

Also, the attribute threadPoolSize of the @Test annotation could be used to run tests in parallel as shown below.

@Test(threadPoolSize=5, invocation-count=10, timeout=10000)
public void testMethod()
{
  System.out.println("Hello");
}

This will invoke the method testMethod 10 times in 5 different threads. The timeout of 10 seconds ensures that the thread is not blocked for more than 10 seconds. The timout applies for both parallel and non-parallel mode.

TestNG groups using include/exclude

TestNG allows to perform groupings of test methods. The methods can be declared to belong to groups and TestNG can be invoked to include/exclude certain set of groups. This provides maximum flexibility to structure the tests in a more intuitive way.

Step1: Declare groups in java class

Suppose we would like to group the test methods into two categories functional and sanity. We declare the methods to belong to either of the two groups using the “groups” attribute of @Test annotation.

public class Test1
{
  @Test(groups={"functional","sanity"})
  public void testMethod1(){
  }
 
  @Test(groups={"sanity"})
  public void testMethod2(){
  }
 
  @Test(groups={"functional","sanity"})
  public void testMethod3(){
  }
}

 

Step2: Specify groups in TestNG.xml file

Suppose we would like to run the sanity tests alone. We specify the groups in the TestNG.xml file using the <groups> tag inside the <suite> or <test> tag as shown below:

<suite name="mySuite" verbose="1">
  <test name="Regression">
    <groups>
      <run>
        <include name="sanity"/>
      </run>
    </groups>
    <classes>
      <class name="sample.Test1"/>
    </classes>
  </test>
</suite>

TestNG.xml – A Quick Overview

TestNG is a testing framework designed to simplify the testing needs.

 

Quick overview of TestNG.xml file

  • A suite is an xml file represented by <suite> tag. The TestNG suite can contain one or more TestNG tests
  • A test is represented by the <test> tag. A TestNG test can contain one or more TestNG classes
  • A class is a java class having TestNG annotations. It is represented by the <class> tag. A TestNG class can contain one or more TestNG methods
  • A method is method configured with @Test annotation

 

Sample TestNG.xml file

 

 

Also, it is possible to use packages instead of classes in TestNG.xml file. For this package tag is used as shown below:

 

Six steps for complete test automation with selenium grid

 

The earlier article “Crucial aspects of browser compatibilty testing with Selenium Grid” gives insight into browser compatibility testing andthe architecture of one of the most popular browser compatibility testing tools, selenium grid. Nowlet‘sget our hands dirty with test automation using Selenium Grid.

 

Pre-requisites:

The practice exercise in this article requires Eclipse, TestNG, ant, Java, Selenium RC and Selenium Grid. You can execute the code on your favorite platform. For this demo I am using ubuntu as platform.


Installation of Selenium Grid:

Please refer the documentation in seleniumhq for selenium grid download and installation instructions

http://selenium-grid.seleniumhq.org/step_by_step_installation_instructions_for_windows.html
http://selenium-grid.seleniumhq.org/get_started.html

Unzip the Selenium Grid folder and execute the below command in the Selenium Grid root directory to check whether it is installed correctly or not:

ant sanity-check

The output should be BUILD SUCCESSFUL

Now let’s get started.


 

Step 1. Launch of selenium hub:

Open a terminal and launch the selenium hub using the below command:

ant launch-hub

The hub is where the selenium remote controls providing the different environments like chrome on ubuntu or IE on Windows register themselves. We can view the list of remote controls registered with the hub at http://localhost:4444/console

Now the available remote controls section in the hub console displays nothing  since we did not start any remote control yet.

 

Step 2. Launch of selenium remote controls:

To start a remote control open a new terminal and issue the below command:

ant launch-remote-control

Then, open the hub console http://localhost:4444/console . The selenium grid hub snapshot displays a remote control on port 5555 for firefox. This is the default environment if we do not specify our own.

We can configure more than one selenium remote control with the same port but more than one remote control cannot run tests on the same port. The reason behind this is simple. For example, more than one person can call the same phone number but cannot talk over the phone to the same person at the same time. Only one call can be answered on a single phone at a time. Similarly it is impossible to run tests in parallel on two selenium remote controls configured with the same port number. We need to configure each remote control with unique port numbers. How do we do this?

If we want to specify our own environment we need to use the Dport and Denvironment options as below:

ant -Dport=5556 -Denvironment=*chrome launch-remote-control

 

Step 3. Automatic environment setup:

Let’s create a bash script for automatically opening up new terminals and launching several remote controls with the desired environment. If you are working on windows you need to create a batch file instead of a bash script. The below script works only on ubuntu platform.

Navigate to the selenium grid directory and create the below bash script. Let’s call it run.sh

#!/bin/bash
gnome-terminal --title="Hub" -x bash -c "ant launch-hub"
sleep 150
echo "10 seconds wait over"
gnome-terminal --tab --title="Chrome" -x bash -c "ant -Dport=5555 -Denvironment=*chrome launch-remote-control" &amp;
gnome-terminal --tab --title="Firefox" -x bash -c "ant -Dport=5556 -Denvironment=*firefox launch-remote-control" &amp;
gnome-terminal --tab --title="Opera" -x bash -c "ant -Dport=5557 -Denvironment=*opera launch-remote-control"

The bash script opens a new gnome terminal called “Hub” and launches the selenium hub. It sleeps for 150 seconds waiting for the hub to get started. Depending on the time taken you can change the sleep period. The script then opens three other new terminals and registers three remote controls with the respective environments specified. On execution of the script the below output is got on http://localhost:4444/console page.

Now the environment is setup for selenium grid automation. Let’s write a simple selenium grid program.

 

 

Step 4. Creating a java project called giftShop:

 

Suppose you would like to cheer up your friend on her birthday and give her the best birthday party of all time. You are planning  to celebrate your friend’s birthday with balloons, cakes, ice creams and of course a birthday gift!. You are going to surprise your friend with “I have got something for you” from www.gifts.com in such a way that no one can ever put a price tag on the friendship. Let’s write a selenium program to test the display of birthday gifts page on www.gifts.com website on multiple browsers.

 

  1. In Eclipse create a new java project called “giftShop”. Click on Next button and click on the “Libraries” tab. Click on “Add External JARs” button. Select the testng and selenium server JAR files as below and click on finish button.

 

 

2. Now create a new TestNG class inside the giftShop project. In the source folder field, browse to /giftShop/src. Give the package name as giftShop and class name as giftBox. Click on finish button.

Step 5. Enable testing in parallel mode in multiple browsers:


Right click on the giftShop project. Click on “Convert into TestNG” option. “Generate testng.xml” window will get displayed. This option will be available only if the TestNG plugin for eclipse is installed. Select “tests” in parallel mode dropdown and click on Finish button.

 

To organize the project structure create a folder called config and place the generated testng.xml file in it.

In the testng.xml file we just created, specify the browser and port names with which we have configured our selenium remote control as in the below piece of code.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="tests">
	<test name="Chrome">
		<parameter name="selenium.host" value="localhost"/>
		<parameter name="selenium.port" value="5555"/>
		<parameter name="selenium.browser" value="*chrome"/>
		<parameter name="selenium.url" value="http://www.gifts.com/"/>
		<parameter name="bdayLink" value="//li[@id='nav-oc']/div/ul[3]/li[3]/a/b"/>
		<classes>
			<class name="giftShop.giftBox" />
		</classes>
	</test>
	<test name="Firefox">
		<parameter name="selenium.host" value="localhost"/>
		<parameter name="selenium.port" value="5556"/>
		<parameter name="selenium.browser" value="*firefox"/>
		<parameter name="selenium.url" value="http://www.gifts.com/"/>
		<parameter name="bdayLink" value="//li[@id='nav-oc']/div/ul[3]/li[3]/a/b"/>
		<classes>
			<class name="giftShop.giftBox" />
		</classes>
	</test>
	<test name="Opera">
		<parameter name="selenium.host" value="localhost"/>
		<parameter name="selenium.port" value="5557"/>
		<parameter name="selenium.browser" value="*opera"/>
		<parameter name="selenium.url" value="http://www.gifts.com/"/>
		<parameter name="bdayLink" value="//li[@id='nav-oc']/div/ul[3]/li[3]/a/b"/>
		<classes>
			<class name="giftShop.giftBox" />
		</classes>
	</test>
</suite>

Step 6: Writing selenium grid program:

This selenium grid program will start selenium on chrome, firefox and opera. It would open up the www.gifts.com website and click on birthday as occasion. It asserts whether the landing page displays the birthday gifts in all the three browsers as expected.

 

package giftShop;
 
import org.testng.annotations.Test;
import org.testng.Assert;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Parameters;
import com.thoughtworks.selenium.DefaultSelenium;
import com.thoughtworks.selenium.Selenium;
 
public class giftBox 
{
	Selenium selenium;
 
	//Declaration of variables used in this program
	String strTimeout="30000";
	String bdayTitle="Birthday Gifts - Birthday Gift Ideas from Gifts.com";
	int intsleepTime=15000;
 
    @BeforeClass
    //Reads the values of host, port, browser and url "www.gifts.com" from testng.xml
    @Parameters({ "selenium.host", "selenium.port", "selenium.browser", "selenium.url" })
    public void beforeClass(String host, String port, String browser, String url)
    {
        selenium = new DefaultSelenium(host, Integer.parseInt(port),browser, url);
        selenium.start();
        selenium.open("");
        selenium.windowMaximize();
        selenium.windowFocus();
        selenium.deleteAllVisibleCookies();
    } 
 
    @Test
    @Parameters({ "bdayLink" })
    public void testB(String bDayGift)
    {
        try
        {
        	//Click on birthday as occasion on www.gifts.com website
        	selenium.click(bDayGift);
        	selenium.waitForPageToLoad(strTimeout);
 
        	//Verify that the birthday gift items page is displayed
        	Assert.assertEquals(bdayTitle, selenium.getTitle());
            Thread.sleep(intsleepTime);
        }
        catch (Exception e)
        {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    } 
 
    @AfterClass
    public void afterClass() 
    {
        selenium.close();
        selenium.stop();
    } 
}

Running the test case:

Please do the following to run the selenium grid test case (after the hub and RCs are launched):
1. Right click on the testng.xml file
2. Click on Run as “TestNG Suite”

Please feel free to post any comments or ask questions.

junit vs TestNG – core differences

Junit and TestNG differ in the core design. Junit is a unit testing framework while TestNG addresses testing at a higher level. This article discusses the three main differences between TestNG and Junit.

1.     In Junit, one test case failure can cause a bunch of test cases to fail in the test suite. There is no option of skipping the set of dependent test cases. The dependent test cases are also reported as failures. For example, suppose there is a test case to test login and the next 10 test cases need to perform a transaction after login. If the login test case fails the other 10 test cases will also fail.

TestNG handles dependency between test cases. If one test case failure causes the failure of a group of test cases it skips that group and executes the rest of the test suite. The group that has dependency on the failed test cases is reported as skipped NOT failed.

2.     In TestNG groups can be defined. Groups are specific subsets of the test suite. We can choose to run only specific subset of the testsuite say database related test cases instead of running the entire test suite. This can be done as below:

In the test case we define two groups DBTestcase and deprecated as below:

@Test(groups = {"DBTestcase", "deprecated"})
public void testMethod2()
{
 
}

In TestNG.xml config file we write the below piece of code to include only database related test cases and exclude the deprecated ones.

<test name="Sample">
	<groups>
		<run>
			<include name="DBTestcase" />
			<exclude name="deprecated" />
		</run>
	</groups>
	<classes>
		<class name="example1.Test1" />
	</classes>
</test>

In Junit for a long time it was not possible to run a specific subset of the test cases. We can either run the entire suite or run each test case individually. Junit 4.8 introduced a new feature called “Categories” to overcome this limitation. However groups are much easier to configure in TestNG.

3.     TestNG supports parameterization for objects. For example I can execute a single test case for multiple test data sets through the parameterization of DataProvider object. This makes the implementation of data driven testing more flexible in TestNG as below:

@DataProvider(name = "DP1")
public String[][] testData() throws Exception
{
String[][] sdataArray = {{"james","moosecat$", "james@gmail.com","selenigr","selenigr1$", "selenigr@yahoo.co.uk"}};
return sdataArray;
}
 
@Test (dataProvider = "DP1")
public void evernoteSharing(String user1, String password1, String email1, String user2, String password2, String email2)
{
 
}

In junit test data is not parameterized using a DataProvider hook. Data-driven testing is implemented differently and involves more coding.

In TestNG groups can be defined. Groups are specific subsets of the test suite. We can choose to run only specific groups instead of running the entire test suite.