[Proposal] Cherry-pick commits from Greenplum to Cloudberry Database

[tuhaihe]

Proposers

Max Yang (@my-ship-it)
Dianjin Wang (@tuhaihe)

Proposal Status

In Progress

Abstract

This proposal aims to make the process of cherry-picking commits from Greenplum to Cloudberry more transparent and easier to track. It also aims to open it to more interested community contributors who would like to join.

Motivation

The Cloudberry Database project was started in 2022, using Greenplum 7.0.0-beta.3 as the codebase before it was made open source. We then focused on developing features for the Cloudberry Database and had slow progress in cherry-picking commits from the upstream Greenplum. When Greenplum became closed-source, cherry-picking work became more important. We found that a few users had adopted the latest open-source Greenplum version 7.1.0 or deployed the latest Greenplum's main branch in their production. We also noticed some minor operation behavior changes and bugs are already fixed in the open-source Greenplum. Therefore, we need to cherry-pick all the necessary commits from the open-source Greenplum to Cloudberry to keep up with the upstream changes for better compatibility between Greenplum and Cloudberry.

Implementation

We started this work in the past few months, but it's not open and transparent enough with the community members. To improve cherry-pick tracking, we have created a GitHub project for public view.
- GitHub project: https://github.com/orgs/cloudberrydb/projects/9

Now, there are nearly 3,000 commits that need to be cherry-picked from Greenplum to Cloudberry. You can check the following document for an overview.

• Google Docs: https://docs.google.com/spreadsheets/d/1Ks6kS0cyk_sG5hIpGXZEmee786h9KVN0Rrw0ljgfY8E/edit?usp=sharing (If you already helped cherry-pick some commits, please comment on them to avoid repetitive cherry-picks.)

How to cherry-pick

When cherry-picking, it's important not to combine multiple original upstream commits into one. Keep the commit history unchanged and do not alter the original author information and commit messages.

If you need to make changes due to conflicts or for Cloudberry, create a new commit yourself instead of editing the original cherry-picked commits directly.

The rest of the process follows the normal pull request workflow. When merging the pull request, choose "rebase and merge" instead of the "squash and merge" option.

Eg:

• Cherry pick from GPDB (02/07/2022-02/28/2022) #432
• Cherry-pick Resgroup related code from GreenPlum [Mar 2, 2022 - Feb 7, 2023] #448

Rollout/Adoption Plan

We would like to have a flexible cherry-pick strategy, allowing you to cherry-pick the commits by features (like Resource Group V2, gptoolkit, etc.), by the bug fix priority (Urgent issues are first), or on a monthly interval (sort from oldest to newest). If you would like to contribute, please feel free to share your comments in this discussion.

Moreover, it would be better to add the label cherry-pick to your pull request on cherry-picks.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[jianlirong]

Wow, this is huge, nearly 3000 commits needed to be cherry-picked. We need more community contributors to join us on this work. It's also an excellent chance for database kernel developers who are new to Cloudberry to gain hands-on experience. By cherry-picking GPDB commits one by one, you can gradually become familiar with Cloudberry's codebase.

[tuhaihe]

About ~300 commits are related to Docs (/gpdb-docs or /doc). Will remove them from the list in the following weeks.

[reshke]

ok, i will take a look and do some cherry-picks
https://docs.google.com/spreadsheets/d/1Ks6kS0cyk_sG5hIpGXZEmee786h9KVN0Rrw0ljgfY8E/edit?disco=AAABX3L4Fno
i did start with first unmegred.

[tuhaihe]

Hi all, you can find the updates on cherry-picking commits from GP to Cloudberry here: https://lists.apache.org/thread/bf4n0p6jt8x2wnsmgwqwmqqboy4kq0st.