[Bug]Fix the collect distinct bug #5000
Conversation
neuyilan
force-pushed
the
dev/qhl/collect_distinct_bug_fix
branch
from
a2df38d to
25df878
Compare
| Arrays.stream(aggregators).forEach(FieldAggregator::reset); | ||
| } | ||
|
|
||
| @Override | ||
| public void add(KeyValue kv) { | ||
| if (initialKv == null) { | ||
| initialKv = kv; |
There was a problem hiding this comment.
Is the reason for extracting initialKv separately to handle the case where there is only one kv and it is a retract? What if there are multiple kvs, but all of them are retracts? It seems like it will still return an INSERT record? It looks like this situation existed before?
There was a problem hiding this comment.
Yes, you are right.
If there are multiple kvs, but all of them are retracts, we still return as INSERT record now.
Do we need to deal with this situation?
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/ReducerMergeFunctionWrapper.java
Show resolved
Hide resolved
|
Hi @neuyilan and @zhongyujiang , I understand this bug, and currently we haven't considered the input situation. Generally speaking, the input is a single piece of data, and we even use the ARRAY function to assemble an array of individual element. But I feel that the changes are too big, and in order to fix this bug, some difficult to understand code has been introduced. This may be a current limitation, you can consider changing the document to clarify it. |
Purpose
Fix #4999
Tests
testAggWithDistinctFirstDuplicate
API and Format
no
Documentation
no