Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Credentials error while mass indexing on AWS ES with static credentials: "The request signature we calculated does not match the signature you provided."

Description

Hello everyone,
I have an application that uses Hibernate search 6.0.3Final with an ES backend.
Until today, I used a local elasticsearch node and everything worked as expected.
Today I tried plugging in an AWS managed cluster, using static authentication, eg.

hibernate.search.backend.hosts=mydomain.es.amazonaws.com hibernate.search.backend.aws.signing.enabled=true hibernate.search.backend.aws.region=myregion hibernate.search.backend.aws.credentials.type=static hibernate.search.backend.aws.credentials.access_key_id=myaccess hibernate.search.backend.aws.credentials.secret_access_mysecret

The app manages to connect to the cluster fine, but whenever I try to reindex my data using the MassIndexer, I am getting a bunch of

HSEARCH400588: Call to the bulk REST API failed: HSEARCH400007: Elasticsearch request failed: HSEARCH400090: Elasticsearch response indicates a failure. Request: POST /_bulk with parameters {} Response: 403 'Forbidden' from 'https://mydomain.es.amazonaws.com' with body { "message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details." }

AWS credentials are fine (the app managed to connect, create mappings and index some data).
The machine is configured to use a different IAM user, if it could be of any help.

Activity

Yoann RodièreJuly 5, 2021 at 8:48 AM

I see you created HSEARCH-4254. Let’s continue the discussion there.

Yoann RodièreJuly 5, 2021 at 6:56 AM

I’m afraid that’s not enough information. I added tests with (very) large documents, in particular one where each document is the whole text of “Great Expectations”, and I did experience similar problems, but I found the cause (at least in that case) and managed to fixe it. See https://github.com/hibernate/hibernate-search/blob/32cc18a8a89995d9ab4deb9174253a368544907f/integrationtest/backend/tck/src/main/java/org/hibernate/search/integrationtest/backend/tck/work/IndexIndexerLargeDocumentsIT.java#L46 .

Please make sure you upgraded to the latest version of both hibernate-backend-elasticsearch-aws and also hibernate-backend-elasticsearch, because one of the bugs was in hibernate-backend-elasticsearch (though it affected only the AWS integration). In general you should always use the same version for all Hibernate Search modules, but who knows…

If the problem remains:

  • What are the stack traces exactly, this time? Including all frames and all causes.

  • Are you using custom configuration for the HTTP client (ElasticsearchHttpClientConfigurer)?

  • Are there any enterprise firewalls between you and AWS that could alter the requests, e.g. by changing or dropping signed headers or slightly altering the content of the request (encoding errors, adding a line break at the end, …)?

  • If all else fails, I will need you to pinpoint the exact requests that are failing (by enabling logs), and give me the exact content of the first failing request.

LorenzoJune 24, 2021 at 3:56 PM

I just pulled 6.0.5Final and tried a reindex on AWS.

Seems to me that the situation improved: the lighter entities now get indexed without any exception. Whenever the mass indexer gets to bigger documents though (eg. file contents), it is still throwing a fair amount of exceptions on credentials.

I know these are just vague feedbacks, but I hope they can help you in some way.

This is my MassIndexer configuration:

searchSession .massIndexer( classToIndex ) .batchSizeToLoadObjects( 100 ) .threadsToLoadObjects( 5 ) .idFetchSize( Integer.MIN_VALUE ) .transactionTimeout( 1800 ) .monitor( monitor ) .failureHandler( failureHandler ) .start();

Let me know if I can be of any help.

Yoann RodièreJune 23, 2021 at 4:36 PM

LorenzoJune 21, 2021 at 3:43 PM

Oh well, that’s good news for me anyway, heh.

Keep up the good work!

Fixed

Details

Assignee

Reporter

Components

Sprint

Fix versions

Affects versions

Priority

Created June 11, 2021 at 1:11 PM
Updated July 5, 2021 at 8:48 AM
Resolved June 21, 2021 at 11:03 AM

Flag notifications