Upgrade JUnit to the latest version (5.x) in the POM.
Upgrade any related library (assertj, easymock, ...) to a version that supports JUnit 5.
Upgrade the rest of the build (surefire, failsafe, Jenkins plugins, ...) to a version that supports JUnit 5.
Take advantage of JUnit 5 features to optimize our tests (more on this below).
Take advantage of Junit 5 features to make our tests more resilient (more on this below).
Upgrading all tests to JUnit 5 APIs. If there is a way to use JUnit 4 APIs with JUnit 5, or to make JUnit 4 and 5 coexist in the same project, we can start with that.
Upgrading legacy tests to JUnit 5 APIs. We'll probably never do this, as it would be a waste of time.
Some of our tests are designed in such a way that they require the exact same setup (index schema and data) for every single test method, and they do not alter that setup inside the test method (read-only). Yet the setup is done for each method, and tear down after each method.
With the Elasticsearch backend, the creation/deletion of an index can be relatively time-consuming. If there was a way to only perform the setup before all test methods, and the tear-down after all test methods, we would probably bring down the execution time of tests significantly.
JUnit 4's @BeforeClass, @AfterClass and @ClassRule are not completely satisfying solutions, because they cannot be used in conjunction with the Paremeterized runner: we would need the setup/tear-down to be executed before/after the execution of tests for a given set of parameters.
Tear-down and setup should still be performed after a test method fails, just in case the index data was corrupted.
This optimization should definitely be opt-in (an annotation on the test), because it will only work for read-only tests. Applied to tests that write to indexes, it would likely introduce random errors.
.h3 Making our tests more resilient
There are some errors caused by external services that we cannot avoid.
The error I have in mind is the "Cannot delete indices that are being snapshotted" error when running tests against an Elasticsearch service hosted on AWS: https://forums.aws.amazon.com/thread.jspa?messageID=862240󒠠 . We cannot control when the snapshots occur, and they systematically make our tests fail.
JUnit 5 probably offers a way to plug in custom behavior when an exception is thrown by a test.
We could leverage this to detect when the error mentioned above happens. When it does, we would wait a few seconds, execute some recovery code (delete all indexes) then restart the test.
Ideally, such a behavior would be defined project-wide, without needing to annotate each single test.