You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using an SSE stream to synchronize a table of snapshots from a compacted Kafka topic, each SSE event needs to convey the message key so that updates for the same key can be easily detected to replace the previous values per distinct key.
Therefore we need to expose the message key in the SSE event, and the message key may not be present in the payload of the message.
Given that custom SSE event fields are not exposed to browser clients, it makes sense to encode that message key into the id field instead.
We already use the id field for last-event-id during stream reconnect, but we can extend it to include the message key (base64) and make it easily parseable by the client as JSON.
id: ["<base64(key)>","<progress>/<etag>"]
Then clients can do JSON.parse(event. lastEventId)[0] to get the message key.
Not every sse-kafka mapping will be used with compacted Kafka topics (which always require a message key for compaction), so there is a possibility that the message key could be null.
In this case we can choose to encode the id field as follows:
id: [null, "<progress>/<etag>"]
Then JSON.parse(event.id)[0] would return null for the message key, which is accurate.
Additionally, the message payload may already contain enough information to extract the message key, making it unnecessary to include an opaque message key in the SSE event id field. This should be captured in the binding configuration .
When message key is omitted from the event id field, then no change in behavior.
id: <progress>/<etag>
Note: see #26 regarding <etag> in above id field syntax.
The text was updated successfully, but these errors were encountered:
When using an
SSE
stream to synchronize a table of snapshots from a compacted Kafka topic, eachSSE
event needs to convey the message key so that updates for the same key can be easily detected to replace the previous values per distinct key.Therefore we need to expose the message key in the
SSE
event, and the message key may not be present in the payload of the message.Given that custom SSE event fields are not exposed to browser clients, it makes sense to encode that message key into the
id
field instead.We already use the
id
field forlast-event-id
during stream reconnect, but we can extend it to include the message key (base64) and make it easily parseable by the client as JSON.Then clients can do
JSON.parse(event. lastEventId)[0]
to get the message key.Not every
sse-kafka
mapping will be used with compacted Kafka topics (which always require a message key for compaction), so there is a possibility that the message key could benull
.In this case we can choose to encode the
id
field as follows:Then
JSON.parse(event.id)[0]
would returnnull
for the message key, which is accurate.Additionally, the message payload may already contain enough information to extract the message key, making it unnecessary to include an opaque message key in the
SSE
eventid
field. This should be captured in the binding configuration .When message key is omitted from the event
id
field, then no change in behavior.Note: see #26 regarding
<etag>
in aboveid
field syntax.The text was updated successfully, but these errors were encountered: