Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Evaluate the need to send the writing schema over with the message

Description

Avro knows to read a message serialized by a different writer assuming certain conditions described in their spec http://avro.apache.org/docs/1.5.1/spec.html

But both reader and writer schema should be provided up deserialization
GenericDatumReader(Schema writer, Schema reader, GenericData data)

Should we embed the schema with each message? It looks like a significant payload.

If we don't, we can use the major.minor version numbers that are before the avro stream. But that will only work if the receiver is more recent that the writer.

In all case, we need a prototype to test backward compatibility.

Activity

Show:

Emmanuel Bernard August 26, 2011 at 5:25 PM

This issue is being folded in

Sanne Grinovero August 6, 2011 at 7:42 PM

I'm wondering if we could send it optionally, this would need some way for the server to answer with a "I need the schema".
We could have it send the schema once when the backend starts up, but this doesn't help if the master is restarted; it would be good if the master could store it in it's local filesystem.
Could an "availables schema store" be shared? A database table, an Infinispan entry, a jar containing resources to make an old version "forward compatible" to be deployed on the master.

These look complex, a simple alternative could be to send it periodically every N messages, where N can be 1 (all), 0 (never) or another positive number. If this was JMX configurable, admins could force it to send an updated schema to masters when they need.

Even simpler: we support it only on demand (the "I need the schema" request from the server), and in the JMS case in which we can't answer people will have to upgrade the master first.

Emmanuel Bernard August 6, 2011 at 2:20 PM

note I've set the community flag up. It is a significant work but that's almost unrelated to HSearch codebase. It's more related to Avro.

Out of Date

Details

Assignee

Reporter

Components

Affects versions

Priority

Created August 6, 2011 at 2:20 PM
Updated September 11, 2011 at 6:21 PM
Resolved August 26, 2011 at 5:26 PM