Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support key in event id field for sse-kafka binding #27

Closed
jfallows opened this issue May 4, 2022 · 1 comment · Fixed by #31
Closed

Support key in event id field for sse-kafka binding #27

jfallows opened this issue May 4, 2022 · 1 comment · Fixed by #31
Assignees
Labels
enhancement New feature or request

Comments

@jfallows
Copy link
Contributor

jfallows commented May 4, 2022

When using an SSE stream to synchronize a table of snapshots from a compacted Kafka topic, each SSE event needs to convey the message key so that updates for the same key can be easily detected to replace the previous values per distinct key.

Therefore we need to expose the message key in the SSE event, and the message key may not be present in the payload of the message.

Given that custom SSE event fields are not exposed to browser clients, it makes sense to encode that message key into the id field instead.

We already use the id field for last-event-id during stream reconnect, but we can extend it to include the message key (base64) and make it easily parseable by the client as JSON.

id: ["<base64(key)>","<progress>/<etag>"]

Then clients can do JSON.parse(event. lastEventId)[0] to get the message key.

Not every sse-kafka mapping will be used with compacted Kafka topics (which always require a message key for compaction), so there is a possibility that the message key could be null.

In this case we can choose to encode the id field as follows:

id: [null, "<progress>/<etag>"]

Then JSON.parse(event.id)[0] would return null for the message key, which is accurate.

Additionally, the message payload may already contain enough information to extract the message key, making it unnecessary to include an opaque message key in the SSE event id field. This should be captured in the binding configuration .

When message key is omitted from the event id field, then no change in behavior.

id: <progress>/<etag>

Note: see #26 regarding <etag> in above id field syntax.

@jfallows jfallows added the enhancement New feature or request label May 4, 2022
@jfallows jfallows self-assigned this May 4, 2022
@akrambek
Copy link
Contributor

akrambek commented May 4, 2022

Yeah, I like it being optional because usually, the key should be part of the payload to avoid additional configuration.

vordimous pushed a commit to vordimous/zilla that referenced this issue Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants