TestNG Listeners

There are mainly two kinds of listeners. First is WebDriver Listeners, and the second is TestNG Listeners. In this tutorial, we will learn TestNG listeners in detail, covering their types, implementation, and practical use cases with examples.

What are TestNG Listeners?

Listeners in TestNG are special interfaces that allows the users to modify the default behavior of TestNG. It is one of the most powerful feature of TestNG, which allows you to customize and control the behavior of test executions.

They help in monitoring and tracking the execution of tests by listening to events and responding accordingly. You can use the TestNG listener in Selenium by implementing the Listener interface. You can use listeners to:

• Generate logs of test execution in detail.
• Customize TestNG reports.
• Capture screenshots on test failures.
• Retry failed tests automatically.
• Send notifications based on test results.

TestNG provides multiple listener interfaces that users can implement to customize their test execution. Let’s understand them.

Types of TestNG Listeners

TestNG provides several types of Listener interfaces that allow you to customize the behavior of TestNG. Each listener is used for different purpose. The most commonly used listeners in TestNG are:

• ITestListener
• ISuiteListener
• IReporter
• IAnnotationTransformer
• IAnnotationTransformer2
• IConfigurable
• IConfigurationListener
• IExecutionListener
• IHookable
• IInvokedMethodListener
• IInvokedMethodListener2
• IMethodInterceptor

In this tutorial, we will learn the most popular and commonly used ITestListener and its several methods.

ITestListener Interface

ITestListener is a listener interface in TestNG that is used for monitoring and tracking the execution of test cases. It is the most commonly used TestNG listener, especially in Selenium WebDriver automation frameworks because it helps in logging test execution details, capturing screenshots on failures, and generating reports.

ITestListener extends the ITestNGListener interface. ITestNGListener is a marker interface for all objects that can be passed as a listener argument.

Methods of ITestListener

ITestListener interface provides several methods to customize the default behavior of TestNG reports and logs. They are as follows:

1. onTestStart()

This method will be called only when any test method gets started. The general syntax of this method is as follows:

onTestStart(ITestResult result) : void

The onTestStart() method in the ITestListener interface takes an ITestResult interface as a parameter. The ITestResult interface extends IAttributes and java.lang.Comparable<ITestResult>.

The ITestResult interface provides detailed information or result of the test execution, such as:

• Test name (getName())
• Test status (getStatus())
• Start and end times (getStartMillis(), getEndMillis())
• Throwable error (if any) (getThrowable())
• Associated test method and class (getMethod(), getTestClass())

We mainly use it for logging test results, reporting, and handling failures in TestNG.

2. onTestSuccess()

The onTestSuccess() method in the ITestListener interface executes when a test method passes successfully. It is called automatically after the successful execution of any test method. The general syntax of this method is as:

onTestSuccess(ITestResult result) : void

This method accepts ITestResult interface as a parameter that contains information about the executed test. It returns nothing. The onTestSuccess() method is commonly used for logging passed test cases, reporting, or triggering custom actions upon successful test execution.

3. onTestFailure()

The onTestFailure() method is called only when any test method fails. The general syntax of this method is given below:

onTestFailure(ITestResult result) : void

This method also takes ITestResult interface as a parameter and returns nothing.

4. onTestSkipped()

The onTestSkipped() method in the ITestListener interface executes only when a test method is skipped. A test may be skipped due to dependencies, failures in configuration methods, or the use of the dependsOnMethods or enabled=false attributes in TestNG. The basic syntax of this method is as:

onTestSkipped(ITestResult result) : void

This method accepts an ITestResult object that contains details about the skipped test, such as its name, reason for skipping, and status. It returns nothing.

We commonly use onTestSkipped() method for logging skipped tests, capturing screenshots, or generating reports.

5. onTestFailedButWithinSuccessPercentage()

The onTestFailedButWithinSuccessPercentage() method provided by the ITestListener interface is called each time when a test method fails but is still within the defined success percentage.

In TestNG, you can specify a success percentage using the successPercentage attribute in the @Test annotation. If a test method meets or exceeds the required success percentage, it is considered a partial success rather than a complete failure. The basic syntax of this method is as:

onTestFailedButWithinSuccessPercentage(ITestResult result) : void

This method accepts an ITestResult object, which contains information about the test that failed within the success percentage. It returns nothing. The onTestFailedButWithinSuccessPercentage() method is useful for tracking partial failures and logging test results accordingly.

6. onStart()

The onStart() method is called before the execution of any test method in a test suite or test class. We generally use it to set up prerequisites, such as initializing reports, logging frameworks, or test configurations. The basic syntax is:

onStart(ITestContext context) : void

The onStart() method accepts an ITestContext object, which provides information about the entire test context, such as test name, included groups, and methods to be executed. This method also returns nothing.

7. onFinish()

The onFinish() method is executed after all test methods in a test suite or test class have finished running. We commonly use it to close resources, generate reports, and perform cleanup activities after test execution. The general syntax is:

onFinish(ITestContext context) : void

The onFinish() method accepts an ITestContext object, which provides information about the completed test execution, such as test results, passed and failed test cases, and runtime details. This method returns nothing.

Implementing ITestListener Listener in TestNG

Let’s take an example in which we will learn how to use the methods of the ITestListener interface in TestNG. Follow all the steps below:

• Create a TestNG Listener class that implements ITestListener.
• Override the listener methods like onTestStart(), onTestSuccess(), onTestFailure(), etc.
• Attach the listener to the TestNG test class using @Listeners annotation.

1. Create a Listener Class (TestListener.java)

import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
public class TestListener implements ITestListener {

    @Override
    public void onTestStart(ITestResult result) {
        System.out.println("Test Started: " + result.getName());
    }

    @Override
    public void onTestSuccess(ITestResult result) {
        System.out.println("Test Passed: " + result.getName());
    }

    @Override
    public void onTestFailure(ITestResult result) {
        System.out.println("Test Failed: " + result.getName());
        System.out.println("Failure Reason: " + result.getThrowable());
    }

    @Override
    public void onTestSkipped(ITestResult result) {
        System.out.println("Test Skipped: " + result.getName());
    }

    @Override
    public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
        System.out.println("Test Failed but within Success Percentage: " + result.getName());
    }

    @Override
    public void onStart(ITestContext context) {
        System.out.println("Test Execution Started: " + context.getName());
    }

    @Override
    public void onFinish(ITestContext context) {
        System.out.println("Test Execution Finished: " + context.getName());
    }
}

2. Create a Test Class (TestCases.java)

import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(TestListener.class) // Attach the listener
public class TestCases {

    @Test
    public void testSuccess() {
        System.out.println("Executing testSuccess...");
        Assert.assertTrue(true); // This test will pass.
    }

    @Test
    public void testFailure() {
        System.out.println("Executing testFailure...");
        Assert.assertTrue(false); // This test will fail.
    }

    @Test
    public void testSkipped() {
        System.out.println("Executing testSkipped...");
        throw new RuntimeException("Skipping this test");
    }

    @Test(successPercentage = 50, invocationCount = 2)
    public void testPartialSuccess() {
        System.out.println("Executing testPartialSuccess...");
        Assert.assertTrue(Math.random() > 0.5); // Randomly pass/fail.
    }
}

Output:
     Test Execution Started: TestCases
     Test Started: testSuccess
     Executing testSuccess...
     Test Passed: testSuccess
     Test Started: testFailure
     Executing testFailure...
     Test Failed: testFailure
     Failure Reason: java.lang.AssertionError
     Test Started: testSkipped
     Executing testSkipped...
     Test Skipped: testSkipped
     Test Started: testPartialSuccess
     Executing testPartialSuccess...
     Test Failed but within Success Percentage: testPartialSuccess
     Test Started: testPartialSuccess
     Executing testPartialSuccess...
     Test Passed: testPartialSuccess
     Test Execution Finished: TestCases

Explanation of ITestListener Methods Used:

• onTestStart(ITestResult result) → Called before each test starts.
• onTestSuccess(ITestResult result) → Called when a test passes.
• onTestFailure(ITestResult result) → Called when a test fails.
• onTestSkipped(ITestResult result) → Called when a test is skipped.
• onTestFailedButWithinSuccessPercentage(ITestResult result) → Called when a test fails but is within the allowed success percentage.
• onStart(ITestContext context) → Called before the test execution begins.
• onFinish(ITestContext context) → Called after all tests in a test suite have run.

Use Cases of ITestListener in TestNG

The ITestListener interface in TestNG provides multiple use cases that enhance monitoring and reporting of test execution. Here’s how you can use it effectively:

1. Logging Test Execution

You can use ITestListener in TestNG to track test execution status (pass, fail, or skip) and store logs for debugging and analysis. Let’s take an example of it.

Example: Logging Test Execution

import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
public class LoggingListener implements ITestListener {

    @Override
    public void onTestStart(ITestResult result) {
        System.out.println("[LOG] Test Started: " + result.getName());
    }

    @Override
    public void onTestSuccess(ITestResult result) {
        System.out.println("[LOG] Test Passed: " + result.getName());
    }

    @Override
    public void onTestFailure(ITestResult result) {
        System.out.println("[LOG] Test Failed: " + result.getName());
        System.out.println("[LOG] Reason: " + result.getThrowable());
    }

    @Override
    public void onTestSkipped(ITestResult result) {
        System.out.println("[LOG] Test Skipped: " + result.getName());
    }

    @Override
    public void onStart(ITestContext context) {
        System.out.println("[LOG] Test Suite Started: " + context.getName());
    }

    @Override
    public void onFinish(ITestContext context) {
        System.out.println("[LOG] Test Suite Finished: " + context.getName());
    }
}

Output:
     [LOG] Test Suite Started: TestCases
     [LOG] Test Started: testSuccess
     [LOG] Test Passed: testSuccess
     [LOG] Test Started: testFailure
     [LOG] Test Failed: testFailure
     [LOG] Reason: java.lang.AssertionError
     [LOG] Test Started: testSkipped
     [LOG] Test Skipped: testSkipped
     [LOG] Test Suite Finished: TestCases

2. Capturing Screenshots on Test Failures

You can use the ITestListener to take screenshots automatically when a test fails to help diagnose issues. Look at the example below.

Example: Take Screenshot on Failure (Using Selenium WebDriver)

import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.ITestListener;
import org.testng.ITestResult;
import java.io.File;
import java.io.IOException;
import org.apache.commons.io.FileUtils;

public class ScreenshotListener implements ITestListener {
    WebDriver driver;

    @Override
    public void onTestFailure(ITestResult result) {
        driver = new ChromeDriver(); // Initialize WebDriver
        TakesScreenshot ts = (TakesScreenshot) driver;
        File srcFile = ts.getScreenshotAs(OutputType.FILE);
        File destFile = new File("Screenshots/" + result.getName() + ".png");

        try {
            FileUtils.copyFile(srcFile, destFile);
            System.out.println("Screenshot captured for test case: " + result.getName());
        } catch (IOException e) {
            e.printStackTrace();
        }
        driver.quit();
    }
}

Output:
      Screenshot captured for test case: testFailure

The screenshot of failed test will be saved in the Screenshots/ folder.

3. Generating Custom Reports

You can store test execution details and generate custom HTML/PDF reports. Custom reports help to store execution details in a structured format.

Example: Writing Test Results to a File

import java.io.FileWriter;
import java.io.IOException;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
public class ReportListener implements ITestListener {
    
@Override
public void onFinish(ITestContext context) {
  try (FileWriter writer = new FileWriter("test-report.txt", true)) {
      writer.write("Test Suite: " + context.getName() + " Completed\n");
      writer.write("Total Passed: " + context.getPassedTests().size() + "\n");
      writer.write("Total Failed: " + context.getFailedTests().size() + "\n");
      writer.write("Total Skipped: " + context.getSkippedTests().size() + "\n");
      writer.write("------------------------------\n");
      System.out.println("Test Report Generated.");
   } catch (IOException e) {
         e.printStackTrace();
    }
  }
}

Output:
     Test Suite: TestCases Completed
     Total Passed: 2
     Total Failed: 1
     Total Skipped: 1
     ------------------------------

4. Email Notifications on Test Failures

You can send alerts when tests fail, providing immediate feedback to the team.

Example: Sending Email on Failure (Using JavaMail API)

import javax.mail.*;
import javax.mail.internet.*;
import java.util.Properties;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class EmailNotificationListener implements ITestListener {
    
@Override
public void onTestFailure(ITestResult result) {
    sendEmail(result.getName(), result.getThrowable().toString());
}
public void sendEmail(String testName, String failureReason) {
   final String username = "your-email@gmail.com";
   final String password = "your-email-password";

   Properties props = new Properties();
   props.put("mail.smtp.auth", "true");
   props.put("mail.smtp.starttls.enable", "true");
   props.put("mail.smtp.host", "smtp.gmail.com");
   props.put("mail.smtp.port", "587");

   Session session = Session.getInstance(props, new Authenticator() {
         protected PasswordAuthentication getPasswordAuthentication() {
                return new PasswordAuthentication(username, password);
         }
    });

    try {
        Message message = new MimeMessage(session);
        message.setFrom(new InternetAddress("your-email@gmail.com"));
        message.setRecipients(Message.RecipientType.TO, InternetAddress.parse("team@example.com"));
        message.setSubject("Test Failure Alert: " + testName);
        message.setText("Test " + testName + " failed due to: " + failureReason);

        Transport.send(message);
        System.out.println("Email sent for failed test: " + testName);
     } catch (MessagingException e) {
           e.printStackTrace();
      }
   }
}

Output:
     Email sent for failed test: testFailure

This sends an email when a test fails. Here, we have understood important use cases of ITestListener. You can also use it to store test results in a database for tracking and future analysis.

TestNG listeners offer a powerful mechanism to customize test execution, logging, and reporting. By implementing ITestListener interface, you can gain better insights into the test execution and improve automation efficiency. It significantly enhances the quality of automation frameworks.