Mitigate test failures caused by automatic snapshots of AWS-managed Elasticsearch service
Description
Activity
Show:
Fixed
Details
Details
Assignee

Reporter

Components
Sprint
None
Priority
Created September 24, 2019 at 8:00 AM
Updated November 21, 2019 at 2:12 PM
Resolved October 29, 2019 at 5:57 PM
Some time ago, AWS changed the automatic snapshot policy of its managed Elasticsearch services: they are no longer performed on a daily basis, but on a hourly basis.
Whenever we test Hibernate Search against such service, and a snapshot occurs during tests, the tests invariably fail because the indexes cannot be deleted during a snapshot.
This has been making the master builds fail quite frequently over the past few weeks, and it's a real nuisance.
There is no way to change the frequency of automatic snapshots.
We should see how we can change our test suite to mitigate the problem.
One option would be that, each time a test fails, we pause for a time (~1 minute, or as long as necessary for snapshots to complete), and re-run the test. This should only be enabled when testing against an AWS-managed Elasticsearch service.
There are options to retry failed tests in the Maven failsafe plugin, but unfortunately they do not allow to specify a delay.
In JUnit 4, we would be able to implement this behavior, but only by using a custom runner or rule in every single test, which is not acceptable.
In Junit 5, there seem to be several extension-based solutions. I wonder if there is a way to enable extensions by default, without any change to the tests, from the maven-failsafe-plugin configuration or system properties? Better, maybe there's a way to enable extensions even in the JUnit 4 compatibility mode?