Reports

Reports

Automated tests need good reporting so that the test results are meaningful. For this purpose, we use the well-known Allure framework. We have a short guide on how to set up Allure.

To get even more insights into your test results we enriched the reports by additional data. This article is about the additional features we have added and how to use them.

Add Links To Report

Sometimes it is necessary to know the exact link a step in the automation opened, and sometimes it's just convenient to jump into a flow at a certain point. For this we have created a function which adds the currently displayed link as a step to the Allure report.

As soon as it is activated, every time a new page is called, a link to exactly this page is stored in the report. This way you can see if the link is correct and even open the page to have a look at it.

This is activated per default but can be deactivated via the neodymium.report.enableStepLinks property in the Neodymium configuration.

Add Test Data To Report

When looking into a failed testcase, it is necessary to get the corresponding test data to be able to replicate the issue. To not be forced to look through a bunch of code just to find the used test data, we have implemented the display of test data in JSON format for each test in the Allure report.

This is activated by default but can be deactivated if the neodymium.report.enableTestDataInReport property in the Neodymium configuration is set to false.

While activated, this feature adds the test data as an attachment with the name always starting with "Testdata" to the Allure report. It is important to mention, that it only applies to test data that is initialized with DataUtils or DataItem, both belonging to Neodymium.

If the test data changes during the test run, it is also possible to add the changes as an additional attachment to the Allure report. To accomplish that, we implemented a helper function addDataAsJsonToReport(String name, Object data) in AllureAddons, belonging to Neodymium. The parameter name is the name of the attachment and the parameter data is the changed testdata.

Add Json Compare To Report

If you compare different json data and the comparison fails, you either get a huge and confusing error message or a short meaningless one, where you don't see what exactly went wrong. In order to prevent wasting time on looking through those kind of error messages, we have implemented the class JsonAssert, for what we utilized the already existing JSONAssert.

If assertEquals fails, an attachment named "Json Compare" containing the differences of the json data is added to the Allure report. If assertNotEquals fails, an attachment named "Json View" containing the json data once is added to the Allure report.

Add custom data to Environment Section of the Report

The Allure report offers an "Environment" section to store information about the test environment. This is especially helpful for keeping track of all the changes made to the test environment and their impact on the test results. If you want to store your own information in this section, Neodymium provides a quick and easy way to do so. To activate the usage of custom data, you have to activate the neodymium.report.environment.enableCustomData property in the Neodymium configuration.

Now to add a line to the environment section of the report simply add a line to a property file starting with neodymium.report.environment.custom. and a custom key value pair. This key value pair will then be added to your report. An example could look like this:

neodymium.report.environment.custom.yourKey = yourValue

The following spaces are eligible for storing such custom data pairs:

1. config/dev-neodymium.propeties in the neodymium testsuite
2. System environment variables
3. System properties
4. config/credentials.properties in the neodymium testsuite
5. config/neodymium.properties in the neodymium testsuite

This is also the order in which the custom data is read.

Note: If there are duplicate keys, the value of the higher ranked file is taken.

Add used Browser Configurations to report

Neodymium provides the ability to define different Browser configurations for tests. It also displays all actually used configurations displayed in the environment section of the Allure report. This can be turned off by setting the neodymium.report.environment.enableBrowserData to false in the Neodymium configuration.

Selenide Error Details Configuration

Selenide prints a lot of information to their error messages by default. Also Allure categorizes errors in each run by similar error messages. Unfortunately, this does not work together well. Due to specific information like the path to the Selenide screenshot, each Selenide error message is unique.

To make sure both features can be used as intended we introduced a toggle to reduce the information of each Selenide error message, to enable Allures category feature.

This can be configured using the neodymium.report.showSelenideErrorDetails property in neodymium.properties file.

If you run you run tests locally without Allure you might want to set this value to true in the dev-neodymium.properties file to see the screenshot path and other information.