Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance kafka binding to notify transition from historical to live messages #172

Closed
jfallows opened this issue Feb 16, 2023 · 0 comments · Fixed by #181
Closed

Enhance kafka binding to notify transition from historical to live messages #172

jfallows opened this issue Feb 16, 2023 · 0 comments · Fixed by #181
Assignees
Labels
enhancement New feature or request

Comments

@jfallows
Copy link
Contributor

jfallows commented Feb 16, 2023

When reading messages from kafka binding cache, the latest message in the Kafka partition might be higher than the most recently received message in the cache, for example if the remaining messages are still in flight.

Therefore, we know that reading historical messages that have already occurred will be completed when we receive that initially observed latest message in the cache.

However, due to filtering, a kafka binding cache client might not receive any messages at all, and be waiting for a new message to arrive.

In this case, it is useful for the kafka binding cache to detect it is up-to-date with all historical messages, but awaiting newer live messages.

This can be communicated per partition by sending a FLUSH reply frame with a KafkaFetchFlushEx indicating the updated progress offset at the initial latest offset.

For kafka binding merged clients, we can collect the FLUSH reply frames for each partition, and when all partitions have been accounted for, then we can send a FLUSH frame with KafkaMergedFlushEx indicating the progress offsets for all partitions, matching the initial latest offsets for all partitions received with the BEGIN reply frame and associated KafkaMergedBeginEx.

Receiving this FLUSH frame before any DATA frames will notify the kafka binding merged client of the transition from historical to live messages, while continuing to receive future messages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants