branch-4.1: [feat](compaction) Support ADMIN COMPACT TABLE type='FULL' and enable it in cloud mode #62596#62675
Closed
github-actions[bot] wants to merge 1 commit intobranch-4.1from
Closed
branch-4.1: [feat](compaction) Support ADMIN COMPACT TABLE type='FULL' and enable it in cloud mode #62596#62675github-actions[bot] wants to merge 1 commit intobranch-4.1from
github-actions[bot] wants to merge 1 commit intobranch-4.1from
Conversation
… it in cloud mode (#62596) ## Summary Unify the ``ADMIN COMPACT TABLE ... WHERE type = '...'`` SQL entry so that BASE, CUMULATIVE and FULL all work on both the local and cloud (storage-compute split) engines. Before this change, FULL was rejected in FE and cloud mode was blocked outright, forcing users to curl each BE's HTTP API to run manual compaction. ## What changes ### FE - ``AdminCompactTableCommand``: add ``FULL`` to ``CompactionType``, relax ``analyzeWhere``, map ``FULL -> \"full\"`` in ``getCompactionType``, and drop the cloud-mode gate so the command dispatches via the normal Agent RPC path. - ``CloudEnv.compactTable``: override ``Env.compactTable`` to walk partitions/indices/tablets, resolve the primary BE via ``Replica.getBackendId()``, skip tablets whose compute group has no available BE, and raise ``DdlException`` if nothing was dispatched. ### BE - ``submit_table_compaction_callback``: strictly match ``base``/``cumulative``/``full`` instead of silently downgrading any non-``\"base\"`` value to ``CUMULATIVE_COMPACTION``. FULL goes through ``submit_compaction_task(force=false, eager=true, trigger_method=MANUAL)`` after setting ``last_full_compaction_schedule_time``, matching the HTTP API default. - ``cloud_submit_table_compaction_callback``: new callback mirroring the local one but calling ``CloudStorageEngine::submit_compaction_task``; honors the cloud HTTP convention that base/cumu need ``sync_delete_bitmap`` while full does not. - ``agent_server::cloud_start_workers``: register the previously stubbed-out ``SUBMIT_TABLE_COMPACTION`` worker (drops the ``// TODO(plat1ko)`` comment). ### Thrift / wire compatibility ``TCompactionReq.type`` remains ``optional string``, so no IDL change. A new FE against an old BE would see ``\"full\"`` silently downgraded (local) or dropped (cloud), so the release order is **all BEs upgraded first, then FE**. FE rollback can lead BE since the FE simply stops sending ``\"full\"``. ## Test plan - [x] ``mvn -pl fe-core -am compile`` passes. - [x] New regression suite ``regression-test/suites/compaction/test_admin_compact_table.groovy`` covers base / cumulative / full end-to-end on the local engine, plus negative cases (unknown type, missing WHERE). - [x] New regression suite ``regression-test/suites/cloud_p0/compaction/test_cloud_admin_compact_table.groovy`` covers the same matrix on the cloud engine. - [x] Design document and code reviewed independently by Codex (two rounds).
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #62596