Cucumber JVM 4 Parallel execution using TestNG

Introduction

This article deals with running Cucumber JVM in parallel using TestNG and Maven. The Maven Failsafe plugin is used for this purpose. In TestNG the scenarios are run in parallel, which means all the steps in a scenario will be executed by the same thread. But different scenarios in a single feature file may be executed by different threads. In the case of a scenariooutline, rows in an examples table could be executed by different threads. Refer to Cucumber-JVM 4 announcement for more details.

To get a better understanding of parallel running using Maven Failsafe plugin refer to official documentation.

The runners generate the json and testng reports. Cluecumber aggregated html reports are also generated.

To run scenarios in parallel using an out of the box solution without JUnit or TestNG refer to this article. To run Cucumber in parallel using JUnit4 refer to this article.

Source Code

The source code is located here. The Cucumber version is 4.2.6, TestNG version is 6.14.3, Maven Failsafe plugin version is 3.0.0-M3.

The Default runner will execute all six feature files inside the features folder. This will be used as an example of a single runner project. The FirstRunner runs the three features in the first folder. Similarly SecondRunner runs three features in the second folder. These two runners will be used as an example of a project with multiple runners.

The steps are pretty basic and print out the executing thread id along with scenario description text and step number. There is a BeforeStep hook which introduces a two second sleep.

The scenarios1.feature and scenarios4.feature files are the same and contain one scenario and one senariooutline. The scenarios2.feature and scenarios5.feature are similar and contain one scenario each. The scenarios3.feature and scenarios6.feature are similar and contain one scenariooutline each.

Cluecumber report is an aggregated representation of the executed scenarios in html format. It uses the json report and is created in the folder specified by the generatedHtmlReportDirectory parameter. Refer to the plugin section of POM for configuration details.

The sample results of the executions using single and multiple runners with different configurations are stored in the reports folder. The duration in these reports is the time between the @AfterClass method and @BeforeClass method of the runner. In case of multiple runners, the duration is the difference between the last @AfterClass and first @BeforeClass of any of the runners.

The POM needs to be executed using the Maven command “mvn clean install” or a similar one which executes the failsafe plugin as well as the cluecumber plugin in the post-integration-test phase.

Failsafe Plugin Configuration

The settings for Maven Failsafe plugin needs to be added to the build plugins section of the pom. Refer to the POM for complete configuration.

The “PARALLEL EXECUTION SETTINGS” section will be detailed in the remaining article. The POM contains all the various settings and comments to enable or disable them.

The inclusion pattern would be specific to the project, depending on the name and number of runner classes. For automatic inclusion and exclusion of test classes refer here. The inclusion pattern for a single runner for this article is shown below. For multiple runner the inclusion pattern will be something like “**/*Runner.java”.

<configuration>
    <includes>
        <include>**/Default.java</include>
    </includes>               
        PARALLEL EXECUTION SETTINGS
 </configuration>

Parallel Execution

The scenarios to be executed is provided by the scenarios() method in the AbstractTestNGCucumberTests class, which is also the super class of runners. This method also has a dataprovider annotation. For parallel execution one needs to override this method and set the parallel option of the dataprovider to true.

One can overwrite this method in the runner class which extends AbstractTestNGCucumberTests. Or a new class containing this method can be created which in turn is extended by the runner for example AbstractTestNGCucumberParallelTests.

@Override
@DataProvider(parallel = true)
public Object[][] scenarios() {
    return super.scenarios();
}

The default number of threads for a dataprovider is 10. To modify this value the dataproviderthreadcount property needs to be added to the POM in the configuration section.

<properties>
    <property>
        <name>dataproviderthreadcount</name>
        <value>4</value>
    </property>
</properties>

In case of multiple runners, one can set the parallel configuration of “classes” to reduce execution times. In addition the threadCount can be set to to the desired value or useUnlimitedThreads can be set to true in the POM in the configuration section.

<parallel>classes</parallel>
<threadCount>4</threadCount>

Refer to the sample times for single runner and multiple runners for the above settings.

Forked Execution

The forkCount parameter is the maximum count of JVM processes that Failsafe plugin will create. Each test class will be executed in its own forked process which will be reused depending on the value of reuseForks parameter.

<forkCount>2</forkCount>
<reuseForks>true</reuseForks>
<reportsDirectory>
    ${project.build.directory}/failsafe-reports_${surefire.forkNumber}
</reportsDirectory>

In case there is a single runner in the project then Failsafe plugin creates a new JVM process. There will be no benefit in terms of reduced execution times.

When there are multiple runners and the forkCount is more than 1, multiple JVM are spawned for the runners. These runners are then executed in parallel though by a single thread inside one JVM.

If the number of runners are more than the forkCount, then the forks can be reused depending on the reuseForks setting. Reusing forks is a more efficient technique unless greater separation is required.

To easily figure out more details about which scenarios are being run in which fork, one can set the reportsDirectory setting by including the surefire.forkNumber in the folder structure. This generates a xml report which mentions the scenarios run in that specific JVM process fork. This is entirely optional. If this is not set then the xml files are anyways generated in the target/failsafe-reports folder.

The dataprovider thread count can be modified by using the dataproviderthreadcount property as described above.

Refer to the sample times for single runner and multiple runners for the above settings.

Scenario Parallel Execution

As mentioned in the introduction section, TestNG runs scenarios in parallel. This also means that different rows of a scenariooutline can be executed by separate threads. This is true for running with the parallel setting or forked execution.

In the image, the line sequence have been modified to show the scenarios in a feature file to be clubbed together. The thread id 12 runs the scenario in the scenarios1.feature file. The scenariooutline in scenarios1.feature has two rows which are run by thread id 13 and 14. Similarly the same logic can be extended to the other threads and feature files.