Upgrade to Allure 2?

[fhoeben]

Allure1 is deprecated according to https://github.com/allure-framework/allure1

Should we upgrade to https://github.com/allure-framework/allure2. What would be impact and benefit?

[fhoeben]

I made a first stab at implementing Allure2 change in https://github.com/fhoeben/allure-fitnesse-listener/tree/allure2, and using these in https://github.com/fhoeben/sample-fitnesse-project/tree/allure2.

It seems to work, but the advantage is not clear to me yet :-)

[tcnh]

There should be some advantages in using allure 2.

1. A separate listener is no longer needed for basic reporting. Allure 2 can serve a nice html aggregate report directly from the surefire-reports. (this would lack the fitnesse result pages, obviously)
2. There's a Jenkins plugin to generate reports as a build step
3. The new plug in framework should unlock the opportunity to convert the fitnesse results combined with the xunit/surefire output to an allure report containing all info (https://docs.qameta.io/allure/2.0/#_allure_plugins_system)
4. Allure 2 supports reporting multiple runs (trend) in one dashboard. Also giving insight in f.e. flaky tests

Thinking out loud, I'd say that when we create a reader/aggregator plugin to do the report conversion from already existing output, The allure listener could become obsolete, making it easier to use. (as a post-build action) Would have to dig in to how allure 2 works.

[cyborghound]

Is this still something you're interested in developing further @fhoeben? I think what @tcnh said is something that i'm also very interested in. It would be a nice addition to FitNesse.

[fhoeben]

I'm not using Allure so I'm not going to take the initiative to create it. If someone else (e.g. @tcnh or you) builds it, I'm willing to support/merge...

[tcnh]

At this point I have no urgent need for Allure 2 in my projects. The current listener and Allure 1 are currently sufficient, so it's not on my (already too long) to-do list right now.

[roydekleijn]

I made a first stab at implementing Allure2 change in https://github.com/fhoeben/allure-fitnesse-listener/tree/allure2, and using these in https://github.com/fhoeben/sample-fitnesse-project/tree/allure2.

It seems to work, but the advantage is not clear to me yet :-)

will this be released? or should we fork it and release it ourselves?

[tcnh]

Visually I see no difference between the reports generated by the v1 and v2 listeners. (other than that the V1 has better info and structure (packages tab) because the v2 branch is pretty far behind)
Using the v1 listener creates results that are interpreted fine by allure 2.x? What does the allure2 branch add in tems of functionality/compatibility?

[roydekleijn]

True there is not much of a difference, however:

• Allure 1 is deprecated
• allure-fitnesse-listener 1.0.5 no longer works on my mac...
• I tried allure 2 branch and it works like a charm. But would be nicer if it is released...

[tcnh]

allure-fitnesse-listener 1.0.5 no longer works on my mac...

Could you describe what errors it's giving?

I can take a look at getting allure2 branch functionally equivalent to 1.0.5 listener, then I see no reason to stick with V1

[roydekleijn]

when adding -Pallure the tests are no longer executed.

it just says:
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

When removing -Pallure it executes all my tests.

[tcnh]

Can you share the allure profile from your pom.xml?

[roydekleijn]

Sure,

<profile>
			<id>allure</id>
			<properties>
				<allure.report.directory>${project.build.directory}/allure-report</allure.report.directory>

<extraFailsafeListeners>,nl.hsac.fitnesse.junit.JUnitXMLPerPageListener,nl.hsac.fitnesse.junit.allure.JUnitAllureFrameworkListener</extraFailsafeListeners>
			</properties>
			<dependencies>
				<dependency>
					<groupId>nl.hsac</groupId>
					<artifactId>allure-fitnesse-listener</artifactId>
					<version>${allure.fitnesse.listener.version}</version>
					<scope>test</scope>
				</dependency>
			</dependencies>
			<build>
				<plugins>
					<plugin>
						<artifactId>maven-resources-plugin</artifactId>
						<version>3.0.2</version>
						<executions>
							<execution>
								<id>copy-resources</id>
								<phase>site</phase>
								<goals>
									<goal>copy-resources</goal>
								</goals>
								<configuration>
									<outputDirectory>${allure.report.directory}/data/fitnesseResults</outputDirectory>
									<resources>
										<resource>
											<directory>${project.build.directory}/fitnesse-results</directory>
											<filtering>true</filtering>
										</resource>
									</resources>
								</configuration>
							</execution>
							<execution>
								<id>copy-resources-allurehistory</id>
								<phase>site</phase>
								<goals>
									<goal>copy-resources</goal>
								</goals>
								<configuration>
									<outputDirectory>${project.build.directory}/allure-results/history</outputDirectory>
									<resources>
										<resource>
											<directory>${allure.report.directory}/history</directory>
											<filtering>true</filtering>
										</resource>
									</resources>
								</configuration>
							</execution>
						</executions>
						<dependencies>
							<dependency>
								<groupId>org.apache.maven.shared</groupId>
								<artifactId>maven-filtering</artifactId>
								<version>3.1.1</version>
							</dependency>
						</dependencies>
					</plugin>

[tcnh]

Could you try:

<profile>
            <id>allure</id>
            <properties>
                <allure.report.directory>${project.build.directory}/allure-report</allure.report.directory>
                <extraFailsafeListeners>,nl.hsac.fitnesse.junit.JUnitXMLPerPageListener,nl.hsac.fitnesse.junit.allure.JUnitAllureFrameworkListener</extraFailsafeListeners>
            </properties>
            <dependencies>
                <dependency>
                    <groupId>nl.hsac</groupId>
                    <artifactId>allure-fitnesse-listener</artifactId>
                    <version>${allure.fitnesse.listener}</version>
                    <scope>test</scope>
                </dependency>

            </dependencies>
            <build>
                <plugins>
                    <plugin>
                        <artifactId>maven-resources-plugin</artifactId>
                        <version>3.0.2</version>
                        <executions>
                            <execution>
                                <id>copy-resources</id>
                                <phase>site</phase>
                                <goals>
                                    <goal>copy-resources</goal>
                                </goals>
                                <configuration>
                                    <outputDirectory>${allure.report.directory}/data/fitnesseResults</outputDirectory>
                                    <resources>
                                        <resource>
                                            <directory>${project.build.directory}/fitnesse-results</directory>
                                            <filtering>true</filtering>
                                        </resource>
                                    </resources>
                                </configuration>
                            </execution>
                            <execution>
                                <id>copy-resources-allurehistory</id>
                                <phase>site</phase>
                                <goals>
                                    <goal>copy-resources</goal>
                                </goals>
                                <configuration>
                                    <outputDirectory>${project.build.directory}/allure-results/history</outputDirectory>
                                    <resources>
                                        <resource>
                                            <directory>${allure.report.directory}/history</directory>
                                            <filtering>true</filtering>
                                        </resource>
                                    </resources>
                                </configuration>
                            </execution>
                        </executions>
                        <dependencies>
                            <dependency>
                                <groupId>org.apache.maven.shared</groupId>
                                <artifactId>maven-filtering</artifactId>
                                <version>3.1.1</version>
                            </dependency>
                        </dependencies>
                    </plugin>
                </plugins>
            </build>
            <reporting>
                <excludeDefaults>true</excludeDefaults>
                <plugins>
                <plugin>
                    <groupId>io.qameta.allure</groupId>
                    <artifactId>allure-maven</artifactId>
                    <version>${allure.maven.version}</version>
                    <configuration>
                        <resultsDirectory>allure-results</resultsDirectory>
                        <reportDirectory>${allure.report.directory}</reportDirectory>
                        <reportVersion>${allure.version}</reportVersion>
                    </configuration>
                </plugin>
                </plugins>
            </reporting>
        </profile>

Use the site target to run the reporting part

Using properties:

<allure.maven.version>2.10.0</allure.maven.version>
<allure.version>2.13.0</allure.version>
<allure.fitnesse.listener>1.0.5</allure.fitnesse.listener>

That should give you an allure2 report, including trend graphs, history, flakiness and new failed markers.

(be sure to exclude history when cleaning):

<fileset>
    <directory>target</directory>
    <includes>
        <include>**</include>
    </includes>
    <excludes>
        <exclude>allure-results/history/**</exclude>
    </excludes>
</fileset>

[teunisv]

Seems to run fine on openJDK 8,
but with openJDK 13 (macos) the result remain empty until we used allure2 code

Using your example pom still result in no runnend test

[tcnh]

Ah, that could be a thing yes. I'm running everything on JDK8.. if you're able to debug a bit to see what can't be found inwhen using java 13... I expect that some JEE dependency is missing. The runner howerver fails silently in that kind of situation

[tcnh]

Just tested with openJDK13 on windows and test ran fine. So it seems to be osx specific.. I can't reproduce it as I have no Mac.

Could you run with -X to see if that produces any error worth investigating?