Introduction
This article deals with running Cucumber JVM in parallel using JUnit4 and Maven. The Maven Failsafe plugin is used for this purpose. In JUnit4 the feature files are run in parallel rather than scenarios, which means all the scenarios in a feature file will be executed by the same thread. Refer to Cucumber-JVM 4 announcement for more details.
To get a better understanding of parallel running using Maven Failsafe plugin refer to official documentation. For details about the thread configuration settings refer to this link. Though it mentions Surefire plugin, the settings are similar for Failsafe.
The runners generate the json and junit reports. Cluecumber aggregated html reports are also generated.
To run scenarios in parallel using an out of the box solution without JUnit or TestNG refer to this article. To run Cucumber in parallel using TestNG refer to this article.
Source Code
The source code is located here. The Cucumber version is 4.2.6, JUnit version is 4.12, Maven Failsafe plugin version is 3.0.0-M3.
The Default runner will execute all six feature files inside the features folder. This will be used as an example of a single runner project. The FirstRunner runs the three features in the first folder. Similarly SecondRunner runs three features in the second folder. These two runners will be used as an example of a project with multiple runners.
The steps are pretty basic and print out the executing thread id along with scenario description text and step number. There is a BeforeStep hook which introduces a two second sleep.
The scenarios1.feature and scenarios4.feature files are the same and contain one scenario and one senariooutline. The scenarios2.feature and scenarios5.feature are similar and contain one scenario each. The scenarios3.feature and scenarios6.feature are similar and contain one scenariooutline each.
Cluecumber report is an aggregated representation of the executed scenarios in html format. It uses the json report and is created in the folder specified by the generatedHtmlReportDirectory parameter. Refer to the plugin section of POM for configuration details.
The sample results of the executions using single and multiple runners with different configurations are stored in the reports folder. The duration in these reports is the time between the @AfterClass method and @BeforeClass method of the runner. In case of multiple runners, the duration is the difference between the last @AfterClass and first @BeforeClass of any of the runners.
The POM needs to be executed using the Maven command “mvn clean install” or a similar one which executes the failsafe plugin as well as the cluecumber plugin in the post-integration-test phase.
Failsafe Plugin Configuration
The settings for Maven Failsafe plugin needs to be added to the build plugins section of the pom. Refer to the POM for complete configuration.
The “PARALLEL EXECUTION SETTINGS” section will be detailed in the remaining article. The POM contains all the various settings and comments to enable or disable them.
The inclusion pattern would be specific to the project, depending on the name and number of runner classes. For automatic inclusion and exclusion of test classes refer here. The inclusion pattern for a single runner for this article is shown below. For multiple runner the inclusion pattern will be something like “**/*Runner.java”.
<configuration>
<includes>
<include>**/Default.java</include>
</includes>
PARALLEL EXECUTION SETTINGS
</configuration>
Parallel Execution
This uses the parallel configuration parameter of the Failsafe plugin. This can be set to ‘methods’ or ‘classesAndMethods’. The number of threads is set by using the threadCount parameter. As an example, below command creates 2 threads per core. This can be confusing as the actual number of threads will be larger than 2 depending on the number of cores.
<parallel>methods</parallel> <threadCount>2</threadCount>
When there is a single runner it is ideal to use the “methods” setting. If multiple runners are present then one can use either “classesAndMethods” or “methods”. The “classes” setting does not give any benefit for a single runner, as it uses one thread it is similar to a sequential execution. Refer to the sample times for single runner and multiple runners for the above settings.
To limit the thread count to the mentioned setting across all cores set the perCoreThreadCount parameter to false. This is set to true by default. Refer to the sample times for single runner and multiple runners for thread count across all cores.
<parallel>methods</parallel> <threadCount>4</threadCount> <perCoreThreadCount>false</perCoreThreadCount>
One can also set the useUnlimitedThreads parameter to true instead of specifying threadCount and perCoreThreadCount options.
<parallel>methods</parallel> <useUnlimitedThreads>true</useUnlimitedThreads>
Forked Execution
The forkCount parameter is the maximum count of JVM processes that Failsafe plugin will create. Each test class will be executed in its own forked process which will be reused depending on the value of reuseForks parameter.
<forkCount>2</forkCount>
<reuseForks>true</reuseForks>
<reportsDirectory>
${project.build.directory}/failsafe-reports_${surefire.forkNumber}
</reportsDirectory>
In case there is a single runner in the project then Failsafe plugin creates a new JVM process. There will be no benefit in terms of reduced execution times.
When there are multiple runners and the forkCount is more than 1, multiple JVM are spawned for the runners. These runners are then executed in parallel though by a single thread inside one JVM, similar to “classes” setting.
If the number of runners are more than the forkCount, then the forks can be reused depending on the reuseForks setting. Reusing forks is a more efficient technique unless greater separation is required.
To easily figure out more details about which scenarios are being run in which fork, one can set the reportsDirectory setting by including the surefire.forkNumber in the folder structure. This generates a xml report which mentions the scenarios run in that specific JVM process fork. This is entirely optional. If this is not set then the xml files are anyways generated in the target/failsafe-reports folder.
Combined Execution
To reduce the execution times in case of JVM process forking one can also configure the parallel setting. The “methods” or “classesAndMethods” setting can be used. This ensures that the feature files are executed by multiple threads in each fork. Refer to the sample times for single runner and multiple runners for the combination settings.
<forkCount>2</forkCount>
<reuseForks>true</reuseForks>
<reportsDirectory>
${project.build.directory}/failsafe-reports_${surefire.forkNumber}
</reportsDirectory>
<parallel>methods</parallel>
<threadCount>2</threadCount>
<perCoreThreadCount>true</perCoreThreadCount>
Feature File Parallel Execution
As mentioned in the introduction section, JUnit runs feature files in parallel rather than scenarios. This is true for running with the parallel setting or forked execution.
In the image, the line sequence have been modified to show the scenarios in a feature file to be clubbed together. The thread id 12 runs the scenario and sceariooutline present in the scenarios1.feature file in the first folder. Similarly the same logic can be extended to the other threads and feature files.
I’m using fork and parallel configuration with single runner class but seems like multiple threads of multiple feature files are accessing same input file which are corrupting the file. Any solution that you can suggest to remove such issues?
U can have synchronized access method to the input file for write access. Or u can store the updates in a synchronized data structure etc and dump it to the file at the end.
My framework is designed in such a way that we have to read and write data in almost every scenario step. Can you suggest any maven surefire that will run each of my feature files on different jvm or different instances so that each parallel execution works like an execution with different memory space.
In my framework we have to read and write data in almost every scenario step and there are lot of scenarios and feature files. Is there a way in maven configuration through which i can run all my feature files in parallel and in different instances or different jvm memory space so that each feature files has its own files to read and write?
U have mentioned that there is a single runner. JVM forking for a single runner case will not provide much benefits in execution times. This is present in the article also.
For multiple threaded execution u can copy the data files in the “process-test-resources” maven lifecycle phase. Similarly combine the files in the “post-integration-test” phase. To coordinate the whole thing a singleton class with Threadlocal can store thread to data file combination.
U can even look at reading the input data file, with the main thread, in a BeforeClass junit\testng method and storing it in a data structure.
This might help – https://stackoverflow.com/questions/17637176/access-file-through-multiple-threads
Is there any way other then adding new code for read and write data through threads and run each feature file in different instances using maven surefire or any other plugin. My framework is on latest cucumber and junit4.
U can post on stackoverflow or on the slack channel of cucumberjvm. thx
Hi,
I want to run feature files parallely, Is it possible to run without creating separate runners?
That is the default behavior of parallel execution using JUnit. Feature files are run in parallel, if u have one or multiple runners. Hope my understanding of ur requirement is correct.
I have added POM and runner details at
https://stackoverflow.com/questions/60406990/cucumber-4-junit-parallel-execution
HI , Mounish,
I am using cucumber -Junit-Spring-RestAssured for API testing. I am using same configuration as mentioned in your blog but I am getting single thread and my feature files are running sequential.
Could you pls give some pointers
Hard to tell without looking at the code. Any chance you could share it? Which configuration are you trying?
is there anyway you can share your email/gitlab userid/skype to share code?
My email is sunalps2000@gmail.com
This was incredibly helpful to me. THANK YOU!
you can try with the rerun plugin. https://github.com/cucumber/cucumber-jvm/blob/v4.7.2/core/src/main/java/io/cucumber/core/plugin/RerunFormatter.java. You will need to configure the report locations to avoid over writing each other as this uses 2 runners to accomplish the reruns. This will still not append the results for the rerun to the original file which i think is the correct behavior. But it will create the reports of the the reruns separately. U can google this or have a look here – https://tutorial.grasshopper.tech/in-built-custom-plugins/ and search for ‘rerun’ in the page.
I have not tried it with parallel execution but the rerun runner should have a name which is alphabetically the last in an ascending sort so it executes at the end. Also i think a parallel setting of ‘methods’ should be used.
Hope it makes sense and works for you.
if we get 2 separate reports, it would lead to duplication of results for the failed scenarios. This would again create confusion.
You can easily parse both the xml reports in a later maven lifecycle and generate a consolidated json report.
Hi,
I was trying to incorporate retry mechanism for failed scenarios with parallel execution.
To achieve the same, I added 1 tag in failsafe plugin.
For reference : https://maven.apache.org/surefire-archives/surefire-2.19/maven-failsafe-plugin/examples/rerun-failing-tests.html
Failed scenario got executed again but output json i.e. cukejson.json got modified completely.
Instead of appending failed scenario results,output json is showcasing only failed scenario results.
Do you have any idea on the same ?
Thanks & Regards
Karan
Hi
My Feature files are running one by one not in parallel.
I used below setup.
maven-surefire-plugin
3.0.0-M3
test
methods
2
true
true
${module}CukeTest.java
${browsertype}
${Environment}
${Enable_EventLog}
${browser}
Can u add a link to your source as it is not clear from the message here?
Hi,
When I use Hooks however, the plugin is calling the Hooks more than the thread count that I have provided, Have you tried your scenarios with Cucumber Hooks?
Can u explain with some more details? Possibly a link to your source. Thanks
Does this execute a specific runner – feature files in different browsers?
Not exactly sure I understand completely what you are asking. Is it executing a combination of runner and the feature files it points to in a single browser instance?
Thanks, it’s very informative
My project has a single runner and multiple step defination files , using Picocontainer as DI , Added POM configuration as directed in github, but for me using Mvn lifecycyle command, Feature files are running one by one, not parallely.
any idea ?
Which option are u using? Setting parallel property or forking?
Found it, my maven command was wrong. mvn clean verify works. Thanks for a great tutorial
I’m using the above project but when running, they all seem to run in series. Console below. The pom is set to create multiple threads but it only seems like it’s running on a single thread. Any ideas? These should all be different threads right. I added the timeline plugin and it also shows just one thread
Thread 1 -> Scenario 1 – FIRST STEP
Thread 1 -> Scenario 1 – SECOND STEP
Thread 1 -> Scenario Outline 1 – FIRST STEP
Thread 1 -> Scenario Outline 1 – FIRST STEP
Thread 1 -> Scenario Outline 1 – SECOND STEP
Thread 1 -> Scenario Outline 1 – SECOND STEP
Thread 1 -> Scenario 4 – FIRST STEP
Thread 1 -> Scenario 4 – SECOND STEP
Thread 1 -> Scenario Outline 4 – FIRST STEP
Thread 1 -> Scenario Outline 4 – FIRST STEP
Thread 1 -> Scenario Outline 4 – SECOND STEP
Thread 1 -> Scenario Outline 4 – SECOND STEP
Thread 1 -> Scenario 2 – FIRST STEP
Thread 1 -> Scenario 2 – SECOND STEP
Thread 1 -> Scenario 5 – FIRST STEP
Thread 1 -> Scenario 5 – SECOND STEP
Thread 1 -> Scenario Outline 3 – FIRST STEP
Thread 1 -> Scenario Outline 3 – FIRST STEP
Thread 1 -> Scenario Outline 3 – SECOND STEP
Thread 1 -> Scenario Outline 3 – SECOND STEP
Thread 1 -> Scenario Outline 6 – FIRST STEP
Thread 1 -> Scenario Outline 6 – FIRST STEP
Thread 1 -> Scenario Outline 6 – SECOND STEP
Thread 1 -> Scenario Outline 6 – SECOND STEP
12 Scenarios (12 passed)
24 Steps (24 passed)
0m48.173s
DURATION – 48092
Thread Id | Scenario Num | Step Count
Process finished with exit code 0