Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Zero-downtime/hot schema updates for the Elasticsearch backend

Description

Elasticsearch Alias, allows me to re-index without disconnecting the clients.

For example, I have an e-commerce that connects directly to elasticsearch.

I use hibernate-search in my erp to index my records from the database (100000 records).

But when doing reindex, hibernate-search deletes the records and inserts them again.

It would be cool to have a unique option to recreate the data in another index and after it has finished pointing to the alias. Allowing zero downtime.

Example:

On create:
1- my_index(alias) — > my_index_v1
2- add new records (100000 records) in my_index_v1

On update (reindex):
1- my_index(alias) — > my_index_v1
2- add new records (100000 records) in my_index_v2
3 - change my_index(alias) — > my_index_v2
4 - remove and delete my_index_v1

In the documentation says this:

value:update

The index, its mappings and analyzer definitions will be created, existing mappings will be updated if there are no conflicts. Caution: if analyzer definitions have to be updated, the index will be closed automatically during the update.

Maybe a property like this:

<property name="hibernate.search.default.elasticsearch.index_schema_management_strategy" value="update-with-alias"/> <property name="hibernate.search.default.elasticsearch.create_index_on_startup" value="true"/>

and

fullTextSession.createIndexer() .purgeAllOnStart( true) // true by default, highly recommended // example (use aliases) .zeroDowntime(true) // Zero Downtime // .optimizeAfterPurge( true ) // true is default, saves some disk space .optimizeOnFinish( true ) // true by default .start();

Index Aliases (Guide):
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html

Index Aliases and Zero Downtime (Reference):
https://www.elastic.co/guide/en/elasticsearch/guide/current/index-aliases.html

Activity

Yoann RodièreSeptember 18, 2023 at 3:21 PM
Edited

During a meeting last week, we discussed how one could update an application without downtime when the schema changed in an incompatible way, so I dumped the result of the discussion in a newly created issue: https://hibernate.atlassian.net/browse/HSEARCH-4955 .

If we solve both https://hibernate.atlassian.net/browse/HSEARCH-3499 and https://hibernate.atlassian.net/browse/HSEARCH-4955, we might want to close this issue ( https://hibernate.atlassian.net/browse/HSEARCH-2861) as “Won’t do”, because application developers would technically be able to do a schema update without any additional feature in Hibernate Search.

Yoann RodièreMarch 8, 2019 at 9:02 AM
Edited

An update on this (complex) problem.

I moved the part about implementing near-zero downtime reindexing using aliases (what you suggested initially) to https://hibernate.atlassian.net/browse/HSEARCH-3499: it will not address your use case (application updates) entirely, but might still be useful for other use cases, such as periodic reindexing.

We discussed a bit some solutions regarding the overall problem of updating an application to a new mapping with zero downtime. I will dump my notes here, just for future reference. It's very raw but it will help remind me what we discussed, at least; I might write up a more understandable wall of text on this later.

Need a way for messages sent from slaves to master to be detected as “obsolete” while we are hot-updating the schema.

Also, need a way to actually create a new index and populate it in such case.

Some solutions where the user affects a global “version number” to his application (and optionally we compute hashes for mappings/indexes) could help doing both (detecting obsolete indexing requests and obsolete schemas). See below.

https://imgur.com/xLshBIV

https://imgur.com/MMLj7Ye

Sanne GrinoveroAugust 30, 2017 at 3:07 PM

Hi all,

it's an excellent suggestion but I agree that it's not a simple problem and that it requires some help from the external deployment process, so we'll need to learn more about this before committing to a strategy.

Let's keep the issue open, optimistically scheduled for 6.x as suggested. We'll have to revisit this, more ideas and feedback welcome.

In version 6 we'll try to better differentiate between an "index name" as a logical name for Hibernate Search usage and the index names being used within Elasticsearch. Adding a configuration property to control the prefix (as mentioned above) for example could be useful, but I believe it would be too confusing without a clear separation of the various types of "index name"s we currently have.

For people not relying on events for indexing (e.g. you want to rebuild the index every night and disable the event listeners) the concerns about losing some events might not apply: if others are interested in such a feature and could explain their re-indexing strategy we can revisit this if there's any need; I believe such users can simply invoke some ES management code after their indexing node is done with the MassIndexer job so unless I'm wrong there's no need for changes from our part?

Thanks!

Yoann RodièreAugust 30, 2017 at 7:26 AM

Moving to 6.x, waiting for Sanne's reaction. It won't be implemented in 5.8 in any case, since we're in the CR phase and it's not a good time for new features.

Leandro Kersting de FreitasAugust 28, 2017 at 2:16 PM

Hi ,

Thanks for the feedback.

I thought of this feature so we could further eliminate the external manual process and allow that hibernate-search to manage this.
But you're very right, this external process is needed anyway and making hibernate-search control this would not be good.

A real example, is in my current development project with wildfly I am evaluating how to do a manual deployment, make blue-green for this.

step 1 - In the first version in production my index will be called for example:

public static final String MY_INDEX_NAME="my_index_v1"; ..... @Indexed(index = myClass.MY_INDEX_NAME) ....

step 2 - I will index through a manual trigger;

fullTextEntityManager.createIndexer().startAndWait();

step 3 - I will create an alias manually for my_index;

POST /_aliases { "actions" : [ { "add" : { "index" : "my_index_v1", "alias" : "my_index" } } ] }

step 4 - When a new version is released with the changed mappings, filters and parsers, I'm going to change the name of my index manually for my_index_v2;
Of course all this in a single Node.

public static final String MY_INDEX_NAME="my_index_v2";

step 5 - I will index through a manual trigger;

fullTextEntityManager.createIndexer().startAndWait(); // or with elasticseach reindex POST _reindex { "source": { "index": "my_index_v1" }, "dest": { "index": "my_index_v2" } }

step 6 - I make a blue-green of the index. I change manually the alias "my_index" to index "my_index_v2";

POST /_aliases { "actions": [ { "remove": { "index": "my_index_v1", "alias": "my_index" }}, { "add": { "index": "my_index_v2", "alias": "my_index" }} ] }

step 7 - After everything is OK, I make blue-green with my load balancer, pointing to the new node.

And so on in the next versions.

I thought of doing this inside hibernate-search, but as you said, that responsibility should not be hibernate-search.

Maybe we could provide an configuration option to add a suffix to every index, But I do not know if we really need it, it would be just a cherry on the cake.

Regards.
Leandro K. de Freitas

Details

Assignee

Reporter

Components

Fix versions

Priority

Parent

Created August 26, 2017 at 9:51 AM
Updated September 25, 2023 at 3:53 PM

Flag notifications