We're updating the issue view to help you get more done. 

AWS signing fails when using multiple routing keys

Description

This used to happen before we disabled some tests:

1 2 3 4 5 6 HSEARCH400007: Elasticsearch request failed: HSEARCH400090: Elasticsearch response indicates a failure. Request: POST /indexname/_search with parameters {size=10000, routing=someText_8,someText_9,someText_6,someText_7,someText_4,someText_5,someText_2,someText_3,someText_0,someText_1} Response: 403 'Forbidden' with body { "message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details." }

I wasn't able to pinpoint the actual problem. Everything is fine as long as we use only one routing key, but as soon as we used multiple keys, all hell breaks loose...

I checked that the signing library apparently works fine; or at least it behaves similarly to the official python library for this specific request.

To debug:

  • enable full logging of requests/responses in Apache HTTP Client:

    1 2 3 4 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog -Dorg.apache.commons.logging.simplelog.showdatetime=true -Dorg.apache.commons.logging.simplelog.log.org.apache.http=DEBUG -Dorg.apache.commons.logging.simplelog.log.org.apache.http.wire=ERROR
  • You'll see an aws request id in the response to the request that failed. Maybe we can find more information about what went wrong in Amazon Athena by querying the logs: https://us-east-1.console.aws.amazon.com/athena/home?force&force=&region=us-east-1#query

Environment

None

Status

Assignee

Yoann Rodière

Reporter

Yoann Rodière

Labels

None

Suitable for new contributors

None

Pull Request

None

Feedback Requested

None

Components

Fix versions

Priority

Major