We're updating the issue view to help you get more done. 

Upgrade to JUnit 5

Description

Goals:

  • Upgrade JUnit to the latest version (5.x) in the POM.

  • Upgrade any related library (assertj, easymock, ...) to a version that supports JUnit 5.

  • Upgrade the rest of the build (surefire, failsafe, Jenkins plugins, ...) to a version that supports JUnit 5.

  • Take advantage of JUnit 5 features to optimize our tests (more on this below).

  • Take advantage of Junit 5 features to make our tests more resilient (more on this below).

Non-goals:

  • Upgrading all tests to JUnit 5 APIs. If there is a way to use JUnit 4 APIs with JUnit 5, or to make JUnit 4 and 5 coexist in the same project, we can start with that.

  • Upgrading legacy tests to JUnit 5 APIs. We'll probably never do this, as it would be a waste of time.

Optimizing our tests

Some of our tests are designed in such a way that they require the exact same setup (index schema and data) for every single test method, and they do not alter that setup inside the test method (read-only). Yet the setup is done for each method, and tear down after each method.

With the Elasticsearch backend, the creation/deletion of an index can be relatively time-consuming. If there was a way to only perform the setup before all test methods, and the tear-down after all test methods, we would probably bring down the execution time of tests significantly.
Note that:

  • JUnit 4's @BeforeClass, @AfterClass and @ClassRule are not completely satisfying solutions, because they cannot be used in conjunction with the Paremeterized runner: we would need the setup/tear-down to be executed before/after the execution of tests for a given set of parameters.

  • Tear-down and setup should still be performed after a test method fails, just in case the index data was corrupted.

  • This optimization should definitely be opt-in (an annotation on the test), because it will only work for read-only tests. Applied to tests that write to indexes, it would likely introduce random errors.

.h3 Making our tests more resilient

There are some errors caused by external services that we cannot avoid.

The error I have in mind is the "Cannot delete indices that are being snapshotted" error when running tests against an Elasticsearch service hosted on AWS: https://forums.aws.amazon.com/thread.jspa?messageID=862240&#862240 . We cannot control when the snapshots occur, and they systematically make our tests fail.

JUnit 5 probably offers a way to plug in custom behavior when an exception is thrown by a test.
We could leverage this to detect when the error mentioned above happens. When it does, we would wait a few seconds, execute some recovery code (delete all indexes) then restart the test.

Ideally, such a behavior would be defined project-wide, without needing to annotate each single test.

Environment

None

Status

Assignee

Unassigned

Reporter

Yoann Rodière

Labels

None

Suitable for new contributors

None

Pull Request

None

Feedback Requested

None

Components

Fix versions

Priority

Major