Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

query engine integration #2074

Merged
merged 1 commit into from
Jan 11, 2025
Merged

Conversation

joshua-spacetime
Copy link
Collaborator

@joshua-spacetime joshua-spacetime commented Dec 19, 2024

Description of Changes

This patch integrates the new query engine into the query code path.

What is implemented?

  1. Instantiating tuple-at-a-time iterators from a physical plan
  2. The addition of a new query crate which is the top level crate for invoking the query engine.
  3. Invoking the new engine for initial subscriptions, one off queries, and incremental update
  4. Benchmarks

API and ABI breaking changes

n/a

Expected complexity level and risk

4

The query engine is responsible for implementing the subscription logic, which is the only way to read data out of SpacetimeDB. As such, its correctness and performance is paramount.

Testing

Does the new engine preserve correctness? Does it also preserve performance?

We have already added tests to ensure we preserve semantics, with more being worked on currently. This patch updates the applicable benchmarks to ensure we maintain performance characteristics.

full-scan               time:   [10.988 ms 11.039 ms 11.098 ms]
                        change: [-24.628% -24.131% -23.622%] (p = 0.00 < 0.05)
                        Performance has improved.

full-join               time:   [92.660 µs 92.817 µs 92.976 µs]
                        change: [-34.859% -34.730% -34.595%] (p = 0.00 < 0.05)
                        Performance has improved.

query-indexes-multi     time:   [361.15 ns 361.59 ns 362.10 ns]
                        change: [-30.401% -30.266% -30.125%] (p = 0.00 < 0.05)
                        Performance has improved.

@joshua-spacetime joshua-spacetime self-assigned this Dec 19, 2024
@joshua-spacetime joshua-spacetime force-pushed the joshua/query-engine-integration branch 2 times, most recently from 3e4ae13 to eaa2861 Compare December 19, 2024 21:00
@joshua-spacetime joshua-spacetime force-pushed the joshua/query-engine-integration branch from 8a4e30b to bace373 Compare December 20, 2024 06:22
@bfops bfops added the release-any To be landed in any release window label Dec 30, 2024
@bfops bfops linked an issue Dec 30, 2024 that may be closed by this pull request
@joshua-spacetime joshua-spacetime force-pushed the joshua/query-engine-integration branch 6 times, most recently from cfb0a7b to 17cceac Compare January 9, 2025 07:14
Copy link
Contributor

@mamcx mamcx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@joshua-spacetime joshua-spacetime force-pushed the joshua/query-engine-integration branch from 17cceac to 5d77cb4 Compare January 10, 2025 22:39
Integrate the new query engine into the codebase for

1. Initial subscriptions
2. One off queries
3. Incremental updates
@joshua-spacetime joshua-spacetime force-pushed the joshua/query-engine-integration branch from bfe8252 to ab3065e Compare January 10, 2025 23:35
@joshua-spacetime joshua-spacetime added this pull request to the merge queue Jan 11, 2025
Merged via the queue into master with commit 3798f46 Jan 11, 2025
8 checks passed
@joshua-spacetime joshua-spacetime deleted the joshua/query-engine-integration branch January 11, 2025 00:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-any To be landed in any release window
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Integrate query engine
4 participants