Clean up benchmark .spacetime in CI

[kazimuth]

Description of Changes

I thought we had already done this.

Expected complexity level and risk

0

Testing

I am about to run CI by clicking the "Create Pull Request" button...

[kazimuth]

benchmarks please

[github-actions]

Callgrind benchmark results

Callgrind Benchmark Report

These benchmarks were run using callgrind,
an instruction-level profiler. They allow comparisons between sqlite (sqlite), SpacetimeDB running through a module (stdb_module), and the underlying SpacetimeDB data storage engine (stdb_raw). Callgrind emulates a CPU to collect the below estimates.

Measurement changes larger than five percent are in bold.

In-memory benchmarks

callgrind: empty transaction

db	total reads + writes	old total reads + writes	Δrw	estimated cycles	old estimated cycles	Δcycles
stdb_raw	6399	6399	0.00%	6523	6523	0.00%
sqlite	5579	5579	0.00%	6019	6019	0.00%

callgrind: filter

db	schema	indices	count	preload	_column	data_type	total reads + writes	old total reads + writes	Δrw	estimated cycles	old estimated cycles	Δcycles
stdb_raw	u32_u64_str	no_index	64	128	1	u64	76593	76593	0.00%	77005	76981	0.03%
stdb_raw	u32_u64_str	no_index	64	128	2	string	119091	119091	0.00%	119763	119779	-0.01%
stdb_raw	u32_u64_str	btree_each_column	64	128	2	string	25083	25083	0.00%	25655	25627	0.11%
stdb_raw	u32_u64_str	btree_each_column	64	128	1	u64	24051	24051	0.00%	24419	24451	-0.13%
sqlite	u32_u64_str	no_index	64	128	2	string	144695	144695	0.00%	146257	146261	-0.00%
sqlite	u32_u64_str	no_index	64	128	1	u64	124044	124044	0.00%	125338	125338	0.00%
sqlite	u32_u64_str	btree_each_column	64	128	1	u64	131361	131361	0.00%	132807	132811	-0.00%
sqlite	u32_u64_str	btree_each_column	64	128	2	string	134494	134494	0.00%	136100	136100	0.00%

callgrind: insert bulk

db	schema	indices	count	preload	total reads + writes	old total reads + writes	Δrw	estimated cycles	old estimated cycles	Δcycles
stdb_raw	u32_u64_str	unique_0	64	128	878247	875757	0.28%	932799	899593	3.69%
stdb_raw	u32_u64_str	btree_each_column	64	128	1024285	1023785	0.05%	1065961	1064997	0.09%
sqlite	u32_u64_str	unique_0	64	128	398320	398320	0.00%	415178	415178	0.00%
sqlite	u32_u64_str	btree_each_column	64	128	983637	983637	0.00%	1020525	1020529	-0.00%

callgrind: iterate

db	schema	indices	count	total reads + writes	old total reads + writes	Δrw	estimated cycles	old estimated cycles	Δcycles
stdb_raw	u32_u64_str	unique_0	1024	153726	153726	0.00%	153866	153866	0.00%
stdb_raw	u32_u64_str	unique_0	64	16751	16751	0.00%	16879	16875	0.02%
sqlite	u32_u64_str	unique_0	1024	1067255	1067273	-0.00%	1070663	1070681	-0.00%
sqlite	u32_u64_str	unique_0	64	76207	76207	0.00%	77445	77445	0.00%

callgrind: serialize_product_value

count	format	total reads + writes	old total reads + writes	Δrw	estimated cycles	old estimated cycles	Δcycles
64	json	47528	47528	0.00%	50282	50282	0.00%
64	bsatn	25509	25509	0.00%	27821	27821	0.00%
16	bsatn	8200	8200	0.00%	9628	9628	0.00%
16	json	12188	12188	0.00%	14194	14194	0.00%

callgrind: update bulk

db	schema	indices	count	preload	total reads + writes	old total reads + writes	Δrw	estimated cycles	old estimated cycles	Δcycles
stdb_raw	u32_u64_str	unique_0	1024	1024	20480022	20084936	1.97%	21091408	20662240	2.08%
stdb_raw	u32_u64_str	unique_0	64	128	1284830	1284386	0.03%	1326720	1359554	-2.42%
sqlite	u32_u64_str	unique_0	1024	1024	1802182	1802182	0.00%	1811362	1811362	0.00%
sqlite	u32_u64_str	unique_0	64	128	128528	128528	0.00%	131492	131492	0.00%

On-disk benchmarks

callgrind: empty transaction

db	total reads + writes	old total reads + writes	Δrw	estimated cycles	old estimated cycles	Δcycles
stdb_raw	6404	6404	0.00%	6544	6544	0.00%
sqlite	5621	5621	0.00%	6121	6121	0.00%

callgrind: filter

db	schema	indices	count	preload	_column	data_type	total reads + writes	old total reads + writes	Δrw	estimated cycles	old estimated cycles	Δcycles
stdb_raw	u32_u64_str	no_index	64	128	1	u64	76598	76598	0.00%	76978	76982	-0.01%
stdb_raw	u32_u64_str	no_index	64	128	2	string	119096	119096	0.00%	119728	119820	-0.08%
stdb_raw	u32_u64_str	btree_each_column	64	128	2	string	25089	25088	0.00%	25561	25572	-0.04%
stdb_raw	u32_u64_str	btree_each_column	64	128	1	u64	24056	24056	0.00%	24400	24400	0.00%
sqlite	u32_u64_str	no_index	64	128	1	u64	125965	125965	0.00%	127543	127547	-0.00%
sqlite	u32_u64_str	no_index	64	128	2	string	146616	146616	0.00%	148450	148446	0.00%
sqlite	u32_u64_str	btree_each_column	64	128	2	string	136616	136616	0.00%	138724	138724	0.00%
sqlite	u32_u64_str	btree_each_column	64	128	1	u64	133457	133457	0.00%	135341	135341	0.00%

callgrind: insert bulk

db	schema	indices	count	preload	total reads + writes	old total reads + writes	Δrw	estimated cycles	old estimated cycles	Δcycles
stdb_raw	u32_u64_str	unique_0	64	128	826870	827011	-0.02%	880656	880749	-0.01%
stdb_raw	u32_u64_str	btree_each_column	64	128	975779	974099	0.17%	1045965	1044871	0.10%
sqlite	u32_u64_str	unique_0	64	128	415857	415857	0.00%	431871	431875	-0.00%
sqlite	u32_u64_str	btree_each_column	64	128	1021898	1021898	0.00%	1057614	1057614	0.00%

callgrind: iterate

db	schema	indices	count	total reads + writes	old total reads + writes	Δrw	estimated cycles	old estimated cycles	Δcycles
stdb_raw	u32_u64_str	unique_0	1024	153731	153731	0.00%	153855	153855	0.00%
stdb_raw	u32_u64_str	unique_0	64	16756	16756	0.00%	16880	16880	0.00%
sqlite	u32_u64_str	unique_0	1024	1070323	1070323	0.00%	1074097	1074097	0.00%
sqlite	u32_u64_str	unique_0	64	77973	77973	0.00%	79303	79303	0.00%

callgrind: serialize_product_value

count	format	total reads + writes	old total reads + writes	Δrw	estimated cycles	old estimated cycles	Δcycles
64	json	47528	47528	0.00%	50282	50282	0.00%
64	bsatn	25509	25509	0.00%	27821	27821	0.00%
16	bsatn	8200	8200	0.00%	9628	9628	0.00%
16	json	12188	12188	0.00%	14194	14194	0.00%

callgrind: update bulk

db	schema	indices	count	preload	total reads + writes	old total reads + writes	Δrw	estimated cycles	old estimated cycles	Δcycles
stdb_raw	u32_u64_str	unique_0	1024	1024	18999658	19002577	-0.02%	19634924	19644985	-0.05%
stdb_raw	u32_u64_str	unique_0	64	128	1237372	1237396	-0.00%	1308478	1309094	-0.05%
sqlite	u32_u64_str	unique_0	1024	1024	1809743	1809761	-0.00%	1818455	1818485	-0.00%
sqlite	u32_u64_str	unique_0	64	128	132654	132654	0.00%	135834	135834	0.00%

[github-actions]

Criterion benchmark results

Criterion benchmark report

YOU SHOULD PROBABLY IGNORE THESE RESULTS.

Criterion is a wall time based benchmarking system that is extremely noisy when run on CI. We collect these results for longitudinal analysis, but they are not reliable for comparing individual PRs.

Go look at the callgrind report instead.

empty

db	on disk	new latency	old latency	new throughput	old throughput
sqlite	💿	-	421.2±4.72ns	-	-
sqlite	🧠	-	411.6±2.68ns	-	-
stdb_raw	💿	775.9±1.52ns	774.9±1.22ns	-	-
stdb_raw	🧠	774.4±0.28ns	774.2±1.02ns	-	-

insert_1

db	on disk	schema	indices	preload	new latency	old latency	new throughput	old throughput

insert_bulk

db	on disk	schema	indices	preload	count	new latency	old latency	new throughput	old throughput
sqlite	💿	u32_u64_str	btree_each_column	2048	256	-	586.9±0.86µs	-	1703 tx/sec
sqlite	💿	u32_u64_str	unique_0	2048	256	-	152.7±0.51µs	-	6.4 Ktx/sec
sqlite	💿	u32_u64_u64	btree_each_column	2048	256	-	472.4±0.65µs	-	2.1 Ktx/sec
sqlite	💿	u32_u64_u64	unique_0	2048	256	-	137.7±0.65µs	-	7.1 Ktx/sec
sqlite	🧠	u32_u64_str	btree_each_column	2048	256	-	447.9±0.87µs	-	2.2 Ktx/sec
sqlite	🧠	u32_u64_str	unique_0	2048	256	-	123.6±0.58µs	-	7.9 Ktx/sec
sqlite	🧠	u32_u64_u64	btree_each_column	2048	256	-	371.0±0.65µs	-	2.6 Ktx/sec
sqlite	🧠	u32_u64_u64	unique_0	2048	256	-	109.3±1.05µs	-	8.9 Ktx/sec
stdb_raw	💿	u32_u64_str	btree_each_column	2048	256	585.8±47.91µs	596.7±23.27µs	1706 tx/sec	1675 tx/sec
stdb_raw	💿	u32_u64_str	unique_0	2048	256	496.3±27.93µs	483.7±35.95µs	2014 tx/sec	2.0 Ktx/sec
stdb_raw	💿	u32_u64_u64	btree_each_column	2048	256	330.1±15.77µs	320.2±10.39µs	3.0 Ktx/sec	3.0 Ktx/sec
stdb_raw	💿	u32_u64_u64	unique_0	2048	256	343.6±12.97µs	331.4±19.71µs	2.8 Ktx/sec	2.9 Ktx/sec
stdb_raw	🧠	u32_u64_str	btree_each_column	2048	256	298.4±0.81µs	296.2±0.16µs	3.3 Ktx/sec	3.3 Ktx/sec
stdb_raw	🧠	u32_u64_str	unique_0	2048	256	230.7±0.51µs	227.5±0.44µs	4.2 Ktx/sec	4.3 Ktx/sec
stdb_raw	🧠	u32_u64_u64	btree_each_column	2048	256	233.1±0.39µs	234.6±0.12µs	4.2 Ktx/sec	4.2 Ktx/sec
stdb_raw	🧠	u32_u64_u64	unique_0	2048	256	208.2±0.14µs	207.1±0.29µs	4.7 Ktx/sec	4.7 Ktx/sec

iterate

db	on disk	schema	indices	new latency	old latency	new throughput	old throughput
sqlite	💿	u32_u64_str	unique_0	-	23.9±0.02µs	-	40.8 Ktx/sec
sqlite	💿	u32_u64_u64	unique_0	-	21.0±0.28µs	-	46.5 Ktx/sec
sqlite	🧠	u32_u64_str	unique_0	-	21.4±0.08µs	-	45.6 Ktx/sec
sqlite	🧠	u32_u64_u64	unique_0	-	18.3±0.03µs	-	53.3 Ktx/sec
stdb_raw	💿	u32_u64_str	unique_0	4.9±0.00µs	4.9±0.00µs	199.5 Ktx/sec	200.4 Ktx/sec
stdb_raw	💿	u32_u64_u64	unique_0	4.8±0.00µs	4.8±0.00µs	204.0 Ktx/sec	204.4 Ktx/sec
stdb_raw	🧠	u32_u64_str	unique_0	4.9±0.00µs	4.9±0.00µs	200.1 Ktx/sec	200.3 Ktx/sec
stdb_raw	🧠	u32_u64_u64	unique_0	4.8±0.00µs	4.8±0.00µs	204.3 Ktx/sec	204.5 Ktx/sec

find_unique

db	on disk	key type	preload	new latency	old latency	new throughput	old throughput

filter

db	on disk	key type	index strategy	load	count	new latency	old latency	new throughput	old throughput
sqlite	💿	string	index	2048	256	-	67.6±0.11µs	-	14.5 Ktx/sec
sqlite	💿	u64	index	2048	256	-	63.1±0.27µs	-	15.5 Ktx/sec
sqlite	🧠	string	index	2048	256	-	64.0±0.19µs	-	15.3 Ktx/sec
sqlite	🧠	u64	index	2048	256	-	57.4±0.11µs	-	17.0 Ktx/sec
stdb_raw	💿	string	index	2048	256	5.1±0.00µs	5.2±0.01µs	191.9 Ktx/sec	188.7 Ktx/sec
stdb_raw	💿	u64	index	2048	256	5.0±0.00µs	5.1±0.00µs	195.7 Ktx/sec	192.8 Ktx/sec
stdb_raw	🧠	string	index	2048	256	5.1±0.00µs	5.2±0.00µs	192.2 Ktx/sec	189.1 Ktx/sec
stdb_raw	🧠	u64	index	2048	256	5.0±0.01µs	5.1±0.00µs	195.6 Ktx/sec	192.7 Ktx/sec

serialize

schema	format	count	new latency	old latency	new throughput	old throughput
u32_u64_str	bflatn_to_bsatn_fast_path	100	3.6±0.00µs	3.6±0.00µs	26.5 Mtx/sec	26.5 Mtx/sec
u32_u64_str	bflatn_to_bsatn_slow_path	100	3.6±0.01µs	3.6±0.02µs	26.5 Mtx/sec	26.6 Mtx/sec
u32_u64_str	bsatn	100	15.5±0.12ns	15.5±0.10ns	6.0 Gtx/sec	6.0 Gtx/sec
u32_u64_str	bsatn	100	2.4±0.00µs	2.4±0.01µs	39.6 Mtx/sec	39.5 Mtx/sec
u32_u64_str	json	100	5.1±0.04µs	5.1±0.04µs	18.8 Mtx/sec	18.8 Mtx/sec
u32_u64_str	json	100	8.2±0.04µs	8.2±0.05µs	11.6 Mtx/sec	11.7 Mtx/sec
u32_u64_str	product_value	100	1020.6±0.38ns	1018.6±0.96ns	93.4 Mtx/sec	93.6 Mtx/sec
u32_u64_u64	bflatn_to_bsatn_fast_path	100	1016.6±14.48ns	1006.1±11.65ns	93.8 Mtx/sec	94.8 Mtx/sec
u32_u64_u64	bflatn_to_bsatn_slow_path	100	2.8±0.07µs	2.8±0.03µs	34.2 Mtx/sec	34.0 Mtx/sec
u32_u64_u64	bsatn	100	14.8±0.02ns	14.8±0.04ns	6.3 Gtx/sec	6.3 Gtx/sec
u32_u64_u64	bsatn	100	1744.5±20.36ns	1734.0±21.74ns	54.7 Mtx/sec	55.0 Mtx/sec
u32_u64_u64	json	100	3.3±0.03µs	3.1±0.04µs	28.6 Mtx/sec	30.4 Mtx/sec
u32_u64_u64	json	100	6.0±0.27µs	5.9±0.03µs	15.9 Mtx/sec	16.1 Mtx/sec
u32_u64_u64	product_value	100	1016.0±4.25ns	1015.7±0.77ns	93.9 Mtx/sec	93.9 Mtx/sec
u64_u64_u32	bflatn_to_bsatn_fast_path	100	757.9±2.25ns	758.3±5.27ns	125.8 Mtx/sec	125.8 Mtx/sec
u64_u64_u32	bflatn_to_bsatn_slow_path	100	2.8±0.00µs	2.8±0.01µs	34.3 Mtx/sec	34.1 Mtx/sec
u64_u64_u32	bsatn	100	1748.1±21.46ns	1740.4±23.82ns	54.6 Mtx/sec	54.8 Mtx/sec
u64_u64_u32	bsatn	100	690.2±0.93ns	705.4±0.84ns	138.2 Mtx/sec	135.2 Mtx/sec
u64_u64_u32	json	100	3.5±0.05µs	3.2±0.04µs	27.6 Mtx/sec	30.0 Mtx/sec
u64_u64_u32	json	100	5.9±0.00µs	5.9±0.04µs	16.3 Mtx/sec	16.3 Mtx/sec
u64_u64_u32	product_value	100	1014.6±0.73ns	1014.1±0.59ns	94.0 Mtx/sec	94.0 Mtx/sec

stdb_module_large_arguments

arg size	new latency	old latency	new throughput	old throughput
64KiB	101.9±6.97µs	104.6±6.28µs	-	-

stdb_module_print_bulk

line count	new latency	old latency	new throughput	old throughput
1	52.7±5.75µs	56.5±5.19µs	-	-
100	600.8±4.44µs	602.3±4.61µs	-	-
1000	3.5±0.34ms	5.4±0.03ms	-	-

remaining

name	new latency	old latency	new throughput	old throughput
special/db_game/circles/load=10	46.0±9.61ms	53.0±1.29ms	-	-
special/db_game/circles/load=100	34.6±0.05ms	38.5±2.71ms	-	-
special/db_game/ia_loop/load=500	149.9±0.84ms	150.6±1.07ms	-	-
special/db_game/ia_loop/load=5000	5.3±0.02s	5.3±0.03s	-	-
sqlite/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256	-	53.7±0.28µs	-	18.2 Ktx/sec
sqlite/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256	-	46.1±0.11µs	-	21.2 Ktx/sec
sqlite/🧠/update_bulk/u32_u64_str/unique_0/load=2048/count=256	-	39.1±0.24µs	-	25.0 Ktx/sec
sqlite/🧠/update_bulk/u32_u64_u64/unique_0/load=2048/count=256	-	34.9±0.18µs	-	28.0 Ktx/sec
stdb_module/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256	1283.4±8.16µs	1268.4±12.89µs	779 tx/sec	788 tx/sec
stdb_module/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256	1019.4±7.27µs	1009.6±7.88µs	980 tx/sec	990 tx/sec
stdb_raw/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256	625.8±17.86µs	622.0±16.94µs	1598 tx/sec	1607 tx/sec
stdb_raw/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256	485.6±8.35µs	415.2±11.69µs	2.0 Ktx/sec	2.4 Ktx/sec
stdb_raw/🧠/update_bulk/u32_u64_str/unique_0/load=2048/count=256	369.1±0.85µs	360.6±0.20µs	2.6 Ktx/sec	2.7 Ktx/sec
stdb_raw/🧠/update_bulk/u32_u64_u64/unique_0/load=2048/count=256	327.5±0.19µs	326.7±0.20µs	3.0 Ktx/sec	3.0 Ktx/sec

[bfops]

I thought we had already done this.

We didn't end up merging the previous attempt to do this (#1445) because it didn't fix the benchmarks at that time.