Stopping JUnit suite if particular test fails

Go To StackoverFlow.com

16

I have a JUnit test suite in the form:

@RunWith(Suite.class)
@Suite.SuiteClasses( { xx.class, yy.cass })

public class AllTests {

public static Test suite() {
    TestSuite suite = new TestSuite(AllTests.class.getName());
    //$JUnit-BEGIN$

    //$JUnit-END$
    return suite;
}
}

This then calls vanilla tests like this:

public class xxx {

@Test
public void test () throws {
    ...

I have a situation where I'd like to stop the rest of the test suite running if there's an error or fail in the first test. But errors / fails in the others are ok and the suite should complete as many other tests as it can. Basically the first test failing would indicate it isn't safe to run the rest.

Is this possible?

2012-04-05 22:28
by Jonathan


7

What's wrong with calling System.exit()?

2012-04-05 23:04
by Hiro2k
How would I programmatically check if an assertion has failed - Jonathan 2012-04-05 23:23
Ok, I see I could catch AssertionError in the first test and then call System.exit(), but then the JUnit test won't pass or fail, just stop running. Seems like there should be a more elegant way - Jonathan 2012-04-05 23:34
Re: a more elegant way, I don't believe JUnit provides this functionality out of the box. It can lead to coupling between tests. Ideally your tests should run independent of one another and not be affected by other failures. If you insist, then Hiro2k's suggestion is best - Ryan Nelson 2012-04-06 00:12
Yeah I didn't want to get into the whole debate but if you did want a more elegant way then you'd have to find a way to decouple your tests from the order in which they run - Hiro2k 2012-04-06 00:37
Thanks Hiro2k, after looking at the other options this seemed like the easiest to handle. I've written up precisely what I did in a new answer - Jonathan 2012-04-06 14:22
assertion and system.exit won't go together.
Ideal soultion should be, @Rule public ErrorCollector collector = new ErrorCollector() - Nishant Shah 2017-11-15 05:05


11

First you need junit RunListener:

import org.junit.runner.notification.Failure;
import org.junit.runner.notification.RunListener;
import org.junit.runner.notification.RunNotifier;

public class FailureListener extends RunListener {

    private RunNotifier runNotifier;

    public FailureListener(RunNotifier runNotifier) {
        super();
        this.runNotifier=runNotifier;
    }

    @Override
    public void testFailure(Failure failure) throws Exception {
        super.testFailure(failure);
        this.runNotifier.pleaseStop();
    }
}

Then prepare a suite:

public class StopOnFailureSuite extends Suite {

    public StopOnFailureSuite(Class<?> klass, Class<?>[] suiteClasses) throws InitializationError {
        super(klass, suiteClasses);
    }

    public StopOnFailureSuite(Class<?> klass) throws InitializationError {
        super(klass, klass.getAnnotation(SuiteClasses.class).value());
    }

    @Override
    public void run(RunNotifier runNotifier) {
        runNotifier.addListener(new FailureListener(runNotifier));
        super.run(runNotifier);
    }
}

And run your suite:

@RunWith(StopOnFailureSuite.class)
@Suite.SuiteClasses({
    FirstTestClass.class,
    SecondTestClass.class,
    ...
})
2015-07-14 03:59
by fab
To implement what the question ask, FailureListener should not stop test on any failure. It might be better to filter the failure type and only stop tests on specific error. eg, creating your own Exception class and throw that in your FirstTestClass. Then use failure.getException() to determine whether to stop the test - artificerpi 2017-08-07 08:04


4

If it's first test then consider moving its validation to @BeforeClass and throw exception if it fails. Then only @AfterClass method would run in case of this exception.

Of course, that way you lack all the fixture artifacts created in test setup method(s).

2012-04-06 03:55
by topchef


3

Based on the answer from Hiro2k (thanks!) I've used the following solution. It's a bit of a hack but it works.

The test which can prevent other tests running goes at the top of the @Suite.SuiteClasses list. That test then has the following:

private static boolean shouldStopRestOfSuite = false;

    @Test
    public void test () throws Throwable {
    try {
            ... run some test code...
    }
    catch (Throwable e) {
        shouldStopRestOfSuite = true;
        throw e;
    }
}

Note the above does need to catch Throwable (not exception) so it catches assertion errors. It also re-throws the error so it's logged by JUnit for analysis.

Then there's another test method:

@Test
public void testKillIfNeeded () throws Exception {
    if (!shouldStopRestOfSuite) {
        return;
    }

    System.out.println ("Test suite killed due to dangerous error / failure");
    System.exit(1);
}

The above is run second and will kill the JUnit process.

Using this method the JUnit test won't end on fail / error if there's an issue but the fail / error is logged for analysis by JUnit and no further tests will run.

Not too pretty but it does the job :)

2012-04-06 14:21
by Jonathan


2

Like your answer but using @Before in an integration test, I did something like this:

public class FooTest {

   private static boolean bar;

   @BeforeClass
   public static void setUpBeforeClass() throws Exception {
         bar = false;
   }

   @Before
   public void setUp() throws Exception {
         assertTrue(bar);
   }

   @Test
   public void test() {
         System.out.println("something");
         assertTrue(true);
   }

   @Test
   public void test1() {
         System.out.println("Something2");
         assertTrue(true);
   }
}

Regards!

2015-01-05 13:01
by Jorge


1

Firstly you should catch an error and check the same before you run the 2nd test.

@Rule
public ErrorCollector collector = new ErrorCollector();

1. Add Error.

collector.addError(new Throwable("first thing went wrong"));

2. Check before the dependent run.

collector.checkThat(getResult(), not(containsString("ERROR!")));

Reference - ErrorCollector

2017-11-15 05:11
by Nishant Shah


0

Are you running tests using ant?

You could write a custom test listener. You can set this in ant http://ant.apache.org/manual/Tasks/junit.html ( enableTestListenerEvents).

2012-04-06 03:41
by Jayan
Although that is an option - I'm generally not using Ant (using Run As > JUnit Test in Eclipse) - Jonathan 2012-04-06 13:14
How would you create that "custom test listener"? I get the ant part with haltonfailure="true", but inside a Suite it does not stop if one of Suite's test fails. What "custom test listener" is able to do - Lorenzo Fidalgo 2017-02-10 09:10
Ads