Custom Reporting with TestNG

TestNG is a widely used testing framework, popular among both developers and testers. It supports testing across different levels—from unit tests to system integration tests. In most cases, TestNG is integrated with other frameworks to build a complete testing solution.

One common integration is with reporting frameworks to meet specific reporting needs. While TestNG provides a default test result report out of the box, it may not always meet your project’s requirements. You might need additional details or a different format.

The good news is, TestNG allows you to customize its reports. This can be achieved by writing your own report listener. In this article, I’ll walk you through how we customized the TestNG report to meet one of our project’s specific needs.

Requirement

First, I’ll show you how we organize our test scripts and test plans using TestNG to give you a basic understanding.

Based on this setup, we needed a simple report in the format shown below, and we also wanted an easy way to share it with other stakeholders.

To meet this reporting need, we had to customize the TestNG report. I’ll walk you through how we did it, step by step.

(1) Create a sample java project with TestNG (I created a sample java maven project to simulate TestNg test executions.)

(2) Create the Report Template

We created a report template in HTML format and placed it under src/test/resources. This template will be used by our custom TestNG listener to generate the final report. The listener will feed test data into the template dynamically. Essentially, we have five parameters that need to be passed into the template.

$reportTitle
$passedTestCount
$failedTestCount
$skippedTestCount
$testResultsRow

(3) Create a custom listener

Next, create the CustomReportListener class under src/.../framework/listeners and implement the IReporter interface provided by TestNG. As part of this implementation, you’ll need to override the generateReport() method, which is where the custom report generation logic will go.

(4) Implement generateReport() method.

Essentially, we need to implement this method to generate the values required for the five parameters in the report template.

4.1) Generate values for test counts parameters

Within the generateReport() implementation, you have access to a list of ISuite objects provided by TestNG. These contain all the test execution details. By looping through the List<ISuite>, you can extract test result information such as the number of passed, failed, and skipped tests. Here’s how you can retrieve those counts.

public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
  int testsCount = 0;
  int passedTests = 0;
  int skippedTests = 0;
  int failedTests = 0;
  
  for(String key :  suites.get(0).getResults().keySet()) {
         passedTests += suites.get(0).getResults().get(key).getTestContext().getPassedTests().size();
         skippedTests += suites.get(0).getResults().get(key).getTestContext().getSkippedTests().size();
         failedTests += suites.get(0).getResults().get(key).getTestContext().getFailedTests().size();   
  }
  testsCount = passedTests + skippedTests + failedTests;
}

4.2) Generate report title

We wanted our report title to match the TestNG suite name, appended with a timestamp. For example: RegressionTestPlan [19-Nov-2020:18.50.59].

public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
  ......
  String suiteName = suites.get(0).getName(); // with this you can get the TestNg suite name
  // getReportTitle(suiteName)
  ......
}

protected String getReportTitle(String title) {
  return title + " [" + getCurrentDateTime() + "]" ;
 }

//private static String getCurrentDateTime(){...}

We placed our report template in the resource folder. Next, let’s read that file and load the HTML template as a string.

private String initReportTemplate() {
String template = null;
byte[] reportTemplate;
Path resourceDirectoryPath = Paths.get("src","test","resources","ReportTemplateV1.html");
try {
reportTemplate = Files.readAllBytes(Paths.get(resourceDirectoryPath.toUri()));
template = new String(reportTemplate, "UTF-8");
} catch (IOException e) {
LOGGER.error("Error occurs while initializing report template", e);
}
return template;
}

Now, let’s bind the values to the reportTemplate variable.

public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
......
reportTemplate = reportTemplate.replace("$reportTitle", getReportTitle(suiteName));
reportTemplate = reportTemplate.replace("$passedTestCount", Integer.toString(passedTests));
reportTemplate = reportTemplate.replace("$failedTestCount", Integer.toString(failedTests));
reportTemplate = reportTemplate.replace("$skippedTestCount", Integer.toString(skippedTests));
......
}

5) Write a method to save the report template string

Now that we have the test result report as an HTML string, we want to save it to a directory.

private static final String REPORT_FOLDER= “custom-reports”;

private void saveReportTemplate(String outputDirectory, String reportTemplate) {<br>File outputDirectoryFilePath = new File(outputDirectory,REPORT_FOLDER);<br>outputDirectoryFilePath.mkdirs();<br>try {<br>PrintWriter reportWriter = new PrintWriter(<br>new BufferedWriter(new FileWriter(new File(outputDirectoryFilePath.getPath(), REPORT_NAME))));<br>reportWriter.println(reportTemplate);<br>reportWriter.flush();<br>reportWriter.close();<br>} catch (IOException e) {<br>LOGGER.error("Problem saving template", e);<br>}<br>}

(6) Let’s do a sample test run

With the code we’ve developed so far, our generateReport() method looks like this:

public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
String reportTemplate = initReportTemplate();

String suiteName = suites.get(0).getName();
int testsCount = 0;
int passedTests = 0;
int skippedTests = 0;
int failedTests = 0;

for(String key : suites.get(0).getResults().keySet()) {
passedTests += suites.get(0).getResults().get(key).getTestContext().getPassedTests().size();
skippedTests += suites.get(0).getResults().get(key).getTestContext().getSkippedTests().size();
failedTests += suites.get(0).getResults().get(key).getTestContext().getFailedTests().size();
}
testsCount = passedTests + skippedTests + failedTests;
reportTemplate = reportTemplate.replace("$reportTitle", getReportTitle(suiteName));
reportTemplate = reportTemplate.replace("$passedTestCount", Integer.toString(passedTests));
reportTemplate = reportTemplate.replace("$failedTestCount", Integer.toString(failedTests));
reportTemplate = reportTemplate.replace("$skippedTestCount", Integer.toString(skippedTests));
saveReportTemplate(outputDirectory, reportTemplate);
}

a) Call the custom listener from TestNG tesplan xml and run the test plan xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="RegressionTestPlan" parallel="false" thread-count="3" verbose="1">

   <listeners>
      <listener class-name = "framework.listeners.CustomReportListener" />
   </listeners>

 <test name="TransationApiTest">
  <classes>
   <class name="testscripts.TransactionAPITest" />
  </classes>
 </test>

 <test name="UserLoginTest">
  <classes>
   <class name="testscripts.UserLoginTest" />
  </classes>
 </test>

</suite>

b) Open generated report (…../test-output/custom-reports/RegressionTest.html)

Everything looks good so far, except for the test result rows—only the other report parameters are being populated correctly.

(7) Generate test result rows

private static final String ROW_TEMPLATE = “<tr><td>%s</td><td>%s</td><td>%s</td><td class=\”%s\”>%s</td><td style=\”text-align: center; vertical-align: middle;\”>%s</td></tr>”;
public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
  // …….
  final String body = suites.stream().flatMap(suiteToResults()).collect(Collectors.joining());
  // …….
}

 private Function<ISuite, Stream<? extends String>> suiteToResults() {
  return suite -> suite.getResults().entrySet().stream().flatMap(resultsToRows(suite));
 }

 private Function<Map.Entry<String, ISuiteResult>, Stream<? extends String>> resultsToRows(ISuite suite) {
  return e -> {
   ITestContext testContext = e.getValue().getTestContext();

   Set<ITestResult> failedTests = testContext.getFailedTests().getAllResults();
   Set<ITestResult> passedTests = testContext.getPassedTests().getAllResults();
   Set<ITestResult> skippedTests = testContext.getSkippedTests().getAllResults();  

   return Stream.of(failedTests, passedTests, skippedTests)
     .flatMap(results -> generateReportRows(results).stream());
  };
 }

 private List<String> generateReportRows(Set<ITestResult> allTestResults) {
  return allTestResults.stream().map(testResultToResultRow()).collect(toList());
 }

 private Function<ITestResult, String> testResultToResultRow() {
  return testResult -> {
   
   String fullyQualifiedName = testResult.getTestClass().getName();
   String testClassName = fullyQualifiedName.substring(fullyQualifiedName.lastIndexOf(“.”)+1);
   String testGroup = testResult.getMethod().getGroups()[0];
   String testDescription = testResult.getMethod().getDescription();
   
   switch (testResult.getStatus()) {
   case ITestResult.FAILURE:
    return String.format(ROW_TEMPLATE, testGroup, testClassName, testDescription,”bg-danger”, “FAILED”, “NA”);

   case ITestResult.SUCCESS:
    return String.format(ROW_TEMPLATE, testGroup, testClassName, testDescription,”bg-success”, “PASSED”,
      String.valueOf(testResult.getEndMillis() – testResult.getStartMillis()));

   case ITestResult.SKIP:
    return String.format(ROW_TEMPLATE,testGroup, testClassName, testDescription, “bg-warning”, “SKIPPED”,
      “NA”);

   default:
    return “”;
   }
  };
 }

private static final String ROW_TEMPLATE = "<tr><td>%s</td><td>%s</td><td>%s</td><td class=\"%s\">%s</td><td style=\"text-align: center; vertical-align: middle;\">%s</td></tr>";
public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
  // .......
  final String body = suites.stream().flatMap(suiteToResults()).collect(Collectors.joining());
  // .......
}

 private Function<ISuite, Stream<? extends String>> suiteToResults() {
  return suite -> suite.getResults().entrySet().stream().flatMap(resultsToRows(suite));
 }

 private Function<Map.Entry<String, ISuiteResult>, Stream<? extends String>> resultsToRows(ISuite suite) {
  return e -> {
   ITestContext testContext = e.getValue().getTestContext();

   Set<ITestResult> failedTests = testContext.getFailedTests().getAllResults();
   Set<ITestResult> passedTests = testContext.getPassedTests().getAllResults();
   Set<ITestResult> skippedTests = testContext.getSkippedTests().getAllResults();  

   return Stream.of(failedTests, passedTests, skippedTests)
     .flatMap(results -> generateReportRows(results).stream());
  };
 }

 private List<String> generateReportRows(Set<ITestResult> allTestResults) {
  return allTestResults.stream().map(testResultToResultRow()).collect(toList());
 }

 private Function<ITestResult, String> testResultToResultRow() {
  return testResult -> {
   
   String fullyQualifiedName = testResult.getTestClass().getName();
   String testClassName = fullyQualifiedName.substring(fullyQualifiedName.lastIndexOf(".")+1);
   String testGroup = testResult.getMethod().getGroups()[0];
   String testDescription = testResult.getMethod().getDescription();
   
   switch (testResult.getStatus()) {
   case ITestResult.FAILURE:
    return String.format(ROW_TEMPLATE, testGroup, testClassName, testDescription,"bg-danger", "FAILED", "NA");

   case ITestResult.SUCCESS:
    return String.format(ROW_TEMPLATE, testGroup, testClassName, testDescription,"bg-success", "PASSED",
      String.valueOf(testResult.getEndMillis() - testResult.getStartMillis()));

   case ITestResult.SKIP:
    return String.format(ROW_TEMPLATE,testGroup, testClassName, testDescription, "bg-warning", "SKIPPED",
      "NA");

   default:
    return "";
   }
  };
 }

a) testResultToResultRow()

We defined our HTML row template as a string variable called ROW_TEMPLATE, which includes placeholders for the data. Using this method, we take each test result record (ITestResult) and insert the necessary data into the ROW_TEMPLATE string, building the complete HTML code for each test result row.

b) generateReportRows()

This method, called testResultToResultRow(), processes all ITestResult objects and returns a list of HTML test result rows as strings.

c) resultsToRows()

This method takes an ISuiteResult and extracts ITestResult sets for Passed, Failed, and Skipped tests. It returns a functional interface that provides the list of HTML rows generated by generateReportRows().

d) suiteToResults()

This method processes ISuite results to construct test result rows and returns a list of HTML rows as a Java Stream. It’s used in the generateReport() method to compile the full HTML report for all test results.

(8) Let’s do the final test run

Now, run your TestNG XML file — the output should look something like this!

If you’ve made it this far, congratulations! You now know how to customize the TestNG report.

References

https://github.com/BathiyaL/Examples/tree/main/testng-custom-report

Custom Reporting with TestNG

Comments

One response to “Custom Reporting with TestNG”

Leave a Reply Cancel reply

AI-Powered Web Testing Using Playwright MCP(Model context protocol)