Moved gp_relsizes_stats extention from Greenplum by Vlasdislav · Pull Request #1757 · apache/cloudberry

Vlasdislav · 2026-05-19T23:45:30Z

What does this PR do?

This PR ports the gp_relsizes_stats extension from the standalone open-gpdb/gp_relsizes_stats repository into the Cloudberry monorepo under gpcontrib/.

The extension collects and stores statistics about table and file sizes on master and segment hosts. It supports automatic collection via a Background Worker and manual one-shot collection via relsizes_stats_schema.relsizes_collect_stats_once().

Changes made during porting:

Makefile — added dual-mode build support: USE_PGXS=1 for standalone builds (original behavior) and in-tree build via $(top_builddir)/src/Makefile.global + contrib-global.mk for building inside the Cloudberry source tree
gpcontrib/Makefile — added gp_relsizes_stats to recurse_targets for both release and debug (enable_debug_extensions=yes) build configurations
README.md — Updated
rebuild_in_docker.sh — updated default PG_CONFIG path from Greenplum to Cloudberry (/usr/local/cloudberry-db/bin/pg_config) and updated default container directory path accordingly

Type of Change

Bug fix (non-breaking change)
New feature (non-breaking change)
Breaking change (fix or feature with breaking changes)
Documentation update

Breaking Changes

Not applicable.

Test Plan

Integration tests added/updated — regression tests in test/sql/gp_relsizes_stats.sql and test/sql/grants.sql are included from the upstream repository
Passed make installcheck
Passed make -C src/test installcheck-cbdb-parallel

Impact

Performance:
No impact on existing functionality. The background worker is disabled by default (gp_relsizes_stats.enabled = false).

User-facing changes:
New extension gp_relsizes_stats available in gpcontrib. Must be added to shared_preload_libraries to enable the background worker.

Dependencies:
None. Uses standard PostgreSQL/Cloudberry APIs only.

Checklist

Followed contribution guide
Added/updated documentation
Reviewed code for security implications
Requested review from cloudberry committers

Additional Context

Upstream repository: https://github.com/open-gpdb/gp_relsizes_stats

…t of ignored

…ototypes for load_data and write_data in table functions

- give ignored_db_names as array of datum (not as List) - add names of ignored dbs to sql-query and remove ignore-check loop

- rename some varables - remove unused variable

- remove recursive logic (base/dboid directory can not contain other directories - rename new_path -> file_path (because it can be just path to file)

- remove constant len buffers for query - make buffer bigger for query when write into it

- remove constant len buffer for query (in get_file_sizes_for_databases) - remove MAX_QUERY_SIZE constant

- dynamically allocate buffer for data_dir - remove all warnings during building

- remove some commented line - apply clang-format

Fix problem with PG_RE_THROW() cycling and rework comments in code

The function made pauses for gp_relsizes_stats.file_naptime (1ms) between lstat calls. It took 1000s to process one million files in database directory. Remove the pauses for relsizes_collect_stats_once. Save the pauses for the case when gp_relsizes_stats_worker gets statistics on files. Increase timeout for database_relsizes_collector_worker from 30 seconds to 5 hours. Additionally, fix the dboid argument type: it should be OID (unsigned), but not INTEGER (signed).

Fixed typo in column name in README.

…_stats_once Speedup relsizes_collect_stats_once

In the presence of partitioned tables, the size of their files was taken into account twice when calculating the schema size. Add the own_file column to table_files to distinguish its own table files from files of its partitions. Use this new column to add the size of each file only once when schema size is calculated. In addition, replace UNION with UNION ALL to simplify the query plan, as duplicate rows are not possible.

Fix the calculation of a schema size when partitioned tables present

…worker When relsizes_collect_stats_once was called and shared_preload_libraries was empty, then segfault happend, because shared memory was not initialized. Now it's not necessary to fill shared_preload_libraries to run the tests. The get_databases_oids function is simplified - it fetchs OIDs only. Move common code from relsizes_collect_stats_once and relsizes_collect_stats to relsizes_collect_stats_once_internal. Remove shmem_startup_hook usage. Database name in the database worker name is replaced with OID to make the code simpler . Getting name by OID makes the code more complex, because we need to open a transaction.

Use worker argument instead of shared memory to pass database to the worker

Replace memset with initialization at declaration. This gives compiler more freedom to optimize the code. Replace GetConfigOptionByName with direct access to variable.

Workers run queries which Orca cannot build plan for or which Orca cannot build a better plan for. So use Postgres planner, because it is faster and there will be no log records about "Feature not supported".

Disable Orca in workers

The extenstion should work without errors when gp_relsizes_stats.so is built from this commit, but ALTER EXTENSION has not yet been executed. When relsizes_stats_schema.get_stats_for_database has one argument, the one argument only is passed to the function. Revert this commit after decision how to upgrade extensions is made.

truncate takes access exclusive lock access delete from table doesn't take lock now we can select statistics if collecting at the moment

Add privileges for user who creates gp_relsizes_stats to use all tables, views and functions from the extension. This user can grant this privileges to others. It is not necessary to grant EXECUTE on the functions, because users can execute them by default when they have the USAGE privilege on the schema.

Non-super users can use the extension right after they create it

…23caa7e897a2bb6d4e9' git-subtree-dir: gpcontrib/gp_relsizes_stats git-subtree-mainline: 61633d2 git-subtree-split: fa47d5b

github-actions

Hi, @Vlasdislav welcome!🎊 Thanks for taking the effort to make our project better! 🙌 Keep making such awesome contributions!

Copilot

Pull request overview

This PR ports the gp_relsizes_stats extension into the Cloudberry monorepo under gpcontrib/, integrating it into the in-tree build and adding regression tests.

Changes:

Adds the new gp_relsizes_stats extension (C code, SQL install/upgrade scripts, control file, docs).
Integrates the extension into gpcontrib/Makefile recursion targets and adds a dual-mode (PGXS vs in-tree) Makefile.
Adds regression tests and expected outputs for functionality and privilege behavior.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
gpcontrib/Makefile	Adds `gp_relsizes_stats` to the gpcontrib build recurse targets.
gpcontrib/gp_relsizes_stats/Makefile	Builds/installs the extension in-tree or via PGXS; wires up regression tests.
gpcontrib/gp_relsizes_stats/src/gp_relsizes_stats.c	Implements background worker orchestration and filesystem stats collection.
gpcontrib/gp_relsizes_stats/sql/gp_relsizes_stats--1.3.sql	Creates schema/tables/views and declares C functions.
gpcontrib/gp_relsizes_stats/sql/gp_relsizes_stats--1.2--1.3.sql	Upgrade script (grants).
gpcontrib/gp_relsizes_stats/sql/gp_relsizes_stats--1.1--1.2.sql	Upgrade script (view changes).
gpcontrib/gp_relsizes_stats/sql/gp_relsizes_stats--1.0--1.1.sql	Upgrade script (function signature change).
gpcontrib/gp_relsizes_stats/gp_relsizes_stats.control	Registers extension metadata (version, module path, trusted flag).
gpcontrib/gp_relsizes_stats/test/sql/grants.sql	Adds privileges/regression coverage for creator + granting to another role.
gpcontrib/gp_relsizes_stats/test/sql/gp_relsizes_stats.sql	Adds functional regression coverage for stats collection and size reporting.
gpcontrib/gp_relsizes_stats/test/expected/grants.out	Expected output for `grants.sql`.
gpcontrib/gp_relsizes_stats/test/expected/gp_relsizes_stats.out	Expected output for `gp_relsizes_stats.sql`.
gpcontrib/gp_relsizes_stats/README.md	Extension documentation.
gpcontrib/gp_relsizes_stats/LICENCE	Bundled Apache 2.0 license text for the extension.
gpcontrib/gp_relsizes_stats/.gitignore	Ignores local build artifacts in the extension directory.
gpcontrib/gp_relsizes_stats/.clang-format	Local formatting configuration for the extension sources.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    } s;
+} DbWorkerArg;
+
+static_assert(sizeof(Datum) == sizeof(DbWorkerArg), "Invalid size of structure in DbWorkerArg");


+        bool oid_nullable;
+
+        heap_deform_tuple(SPI_tuptable->vals[i], SPI_tuptable->tupdesc, &oid_datum, &oid_nullable);


+    if (create_transaction) {
+        SetCurrentStatementStartTimestamp();
+        StartTransactionCommand();
+    }
+
+    if (SPI_connect() < 0) {
+        error = "get_databases_oids: SPI_connect failed";
+        goto finish_transaction;
+    }
+    if (create_transaction) {
+        PushActiveSnapshot(GetTransactionSnapshot());
+        pgstat_report_activity(STATE_RUNNING, sql);
+    }
+


+    }
+    SPI_finish();
+finish_transaction:
+    PopActiveSnapshot();


+        pgstat_report_activity(STATE_RUNNING, sql_get_stats);
+        retcode = SPI_execute_with_args(sql_get_stats, 1,
+                              (Oid[]){INT4OID},
+                              (Datum[]){ObjectIdGetDatum(MyDatabaseId)},


+SELECT '\! cp "' || setting || '/pg_hba.conf" "'  || setting || '/pg_hba.conf.backup"' as cp_backup
+FROM pg_settings
+WHERE name = 'data_directory' \gset
+
+:cp_backup
+
+SELECT '\! echo "local all user1,user2 trust" >> ' || setting || '/pg_hba.conf' as add_users
+FROM pg_settings
+WHERE name = 'data_directory' \gset
+
+:add_users
+
+-- start_ignore
+\! gpstop -u
+-- end_ignore


+SELECT EXTRACT(EPOCH FROM LOCALTIMESTAMP(0)) t1 \gset
+
+SELECT relsizes_stats_schema.relsizes_collect_stats_once();
+
+SELECT (EXTRACT(EPOCH FROM LOCALTIMESTAMP(0)) - :t1) < 5;
+


+make && make install
+```
+
+### Confguration


+### Installation
+Install from source:
+```
+git clone git@github.com:open-gpdb/gp_relsizes_stats.git
+cd gp_relsizes_stats
+# Build it. Building would require GP installed nearby and sourcing greenplum_path.sh
+source <path_to_gp>/greenplum_path.sh
+make && make install


tuhaihe · 2026-05-21T09:19:42Z

+DATA            = $(wildcard sql/*--*.sql)
+REGRESS         = grants gp_relsizes_stats
+REGRESS_OPTS    = --inputdir=test/
+PGFILEDESC      = "gp_relsizes_stats - an extension to track table on-disc sizes in greenplum"


tuhaihe · 2026-05-21T09:16:22Z

Hi @Vlasdislav thanks for your work. The following are my comments for your reference:

Import the commit history to the gpcontrib/gp_relsizes_stats from the start for a better rebase:

git filter-repo --force --to-subdirectory-filter gpcontrib/gp_relsizes_stats

Rewrite some commits for a better clean & lean commit history
ASF license compliance. See more comments in the code.

tuhaihe · 2026-05-21T09:18:31Z

Please add the standard ASF license header. Can take this for a reference:

cloudberry/gpcontrib/gp_stats_collector/src/gp_stats_collector.c

Lines 1 to 26 in d4f437b

/*-------------------------------------------------------------------------

*

* Licensed to the Apache Software Foundation (ASF) under one

* or more contributor license agreements. See the NOTICE file

* distributed with this work for additional information

* regarding copyright ownership. The ASF licenses this file

* to you under the Apache License, Version 2.0 (the

* "License"); you may not use this file except in compliance

* with the License. You may obtain a copy of the License at

*

* http://www.apache.org/licenses/LICENSE-2.0

*

* Unless required by applicable law or agreed to in writing,

* software distributed under the License is distributed on an

* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

* KIND, either express or implied. See the License for the

* specific language governing permissions and limitations

* under the License.

*

* gp_stats_collector.c

*

* IDENTIFICATION

* gpcontrib/gp_stats_collector/src/gp_stats_collector.c

*

*-------------------------------------------------------------------------

*/

Yes, I have already added it locally

tuhaihe · 2026-05-21T09:19:04Z

I think we can remove this file.

tuhaihe · 2026-05-21T09:19:42Z

+DATA            = $(wildcard sql/*--*.sql)
+REGRESS         = grants gp_relsizes_stats
+REGRESS_OPTS    = --inputdir=test/
+PGFILEDESC      = "gp_relsizes_stats - an extension to track table on-disc sizes in greenplum"


tuhaihe · 2026-05-21T09:22:35Z

Could you enable the tests in the .github/workflows CI workflows files for this new extension? Then we can see if it can build/run well in the test env.

Smyatkin-Maxim and others added 30 commits March 18, 2024 12:06

Initial commit with working gp extension template

1165bbc

create function for reading list of databases ids without db from lis…

15665d8

…t of ignored

add create_trunkate function; write code for main function; create pr…

81ec437

…ototypes for load_data and write_data in table functions

save commit; add functions for segments

bf81310

version with getting file_sizes problems

bfe81d2

remove unused and debug functions

5150446

try to fix bug with another creation of HeapTuple - doesn't work

24e6bd7

try to automaticly generate tuple desc for return result (does not help)

3e77f8b

first working version; fixed incorrect memory context usage for tupstore

99f6c1b

remove unused function fields and variables

f5a080c

add clang-format config to gitignore

40fe032

make sql-style naming in .sql file

37ae025

rename data collecting function

af1e5ce

write function for creating and truncating special info tables

99a5af0

rework sql-queries and change code-style

7181191

remove goto after elog for error

7a0d23f

rework comments

ef22a64

change elog to ereport

0e6fa21

rework get_collectable_db_ids function

b5e2a37

- give ignored_db_names as array of datum (not as List) - add names of ignored dbs to sql-query and remove ignore-check loop

naming fix

5fc4da8

- rename some varables - remove unused variable

add more warning logging and write comments to functions

12e346b

move numbers in code under defines

103fae6

rework fill_file_sizes function

72c44d4

- remove recursive logic (base/dboid directory can not contain other directories - rename new_path -> file_path (because it can be just path to file)

apply format

81c4574

rework query creation

94b86b0

- remove constant len buffers for query - make buffer bigger for query when write into it

rework query creation

d720fc0

- remove constant len buffer for query (in get_file_sizes_for_databases) - remove MAX_QUERY_SIZE constant

rework data_dir storing way

90c5b49

- dynamically allocate buffer for data_dir - remove all warnings during building

minor changes

7bc162a

- remove some commented line - apply clang-format

create new schema for storing tables with plugin collected data

5e0e2bd

change distribution for segment_file_map table

4466c15

AndrewOvvv and others added 21 commits August 28, 2025 16:34

Merge pull request apache#18 from open-gpdb/ch1nn1y

0b2dfe3

Fix problem with PG_RE_THROW() cycling and rework comments in code

Fixed typo in column name

c611e1c

Merge pull request apache#19 from Odyvia42/patch-1

4b01159

Fixed typo in column name in README.

Remove uninformative lines from expected output

94e1009

Merge pull request apache#20 from open-gpdb/speedup__relsizes_collect…

f00648f

…_stats_once Speedup relsizes_collect_stats_once

Merge pull request apache#21 from open-gpdb/fix_schema_size

32705fa

Fix the calculation of a schema size when partitioned tables present

Merge pull request apache#22 from open-gpdb/remove_shared_mem

c6436bf

Use worker argument instead of shared memory to pass database to the worker

Refactoring (apache#23)

857b8e8

Replace memset with initialization at declaration. This gives compiler more freedom to optimize the code. Replace GetConfigOptionByName with direct access to variable.

Disable Orca in workers

454c11a

Workers run queries which Orca cannot build plan for or which Orca cannot build a better plan for. So use Postgres planner, because it is faster and there will be no log records about "Feature not supported".

Merge pull request apache#24 from open-gpdb/disable_orca

0047d14

Disable Orca in workers

fix typo (apache#26)

22caba7

do delete from table instead truncate

f906b49

truncate takes access exclusive lock access delete from table doesn't take lock now we can select statistics if collecting at the moment

Merge pull request apache#30 from open-gpdb/grant

fa47d5b

Non-super users can use the extension right after they create it

Add 'gpcontrib/gp_relsizes_stats/' from commit 'fa47d5b3f665c6613efa8…

51889a4

…23caa7e897a2bb6d4e9' git-subtree-dir: gpcontrib/gp_relsizes_stats git-subtree-mainline: 61633d2 git-subtree-split: fa47d5b

Feat: Moved gp_relsizes_stats extention from gp

910ecbb

Fix: adapt Makefile, fix uninitialized var

c6072c4

github-actions Bot reviewed May 19, 2026

View reviewed changes

tuhaihe self-requested a review May 20, 2026 06:13

leborchuk requested a review from Copilot May 21, 2026 07:52

Copilot started reviewing on behalf of leborchuk May 21, 2026 07:52 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

tuhaihe reviewed May 21, 2026

View reviewed changes

Fix: Refactoring, correction of typos, set Licence

6ee3710

Vlasdislav force-pushed the gp_relsizes_stats branch from 3514aa1 to 6ee3710 Compare May 21, 2026 16:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Moved gp_relsizes_stats extention from Greenplum#1757

Moved gp_relsizes_stats extention from Greenplum#1757
Vlasdislav wants to merge 116 commits into
apache:mainfrom
Vlasdislav:gp_relsizes_stats

Vlasdislav commented May 19, 2026 •

edited

Loading

Uh oh!

github-actions Bot left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

tuhaihe May 21, 2026

Uh oh!

tuhaihe commented May 21, 2026

Uh oh!

tuhaihe May 21, 2026

Uh oh!

Vlasdislav May 21, 2026

Uh oh!

tuhaihe May 21, 2026

Uh oh!

tuhaihe May 21, 2026

Uh oh!

tuhaihe May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

		bool oid_nullable;

		heap_deform_tuple(SPI_tuptable->vals[i], SPI_tuptable->tupdesc, &oid_datum, &oid_nullable);

	/*-------------------------------------------------------------------------
	*
	* Licensed to the Apache Software Foundation (ASF) under one
	* or more contributor license agreements. See the NOTICE file
	* distributed with this work for additional information
	* regarding copyright ownership. The ASF licenses this file
	* to you under the Apache License, Version 2.0 (the
	* "License"); you may not use this file except in compliance
	* with the License. You may obtain a copy of the License at
	*
	* http://www.apache.org/licenses/LICENSE-2.0
	*
	* Unless required by applicable law or agreed to in writing,
	* software distributed under the License is distributed on an
	* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	* KIND, either express or implied. See the License for the
	* specific language governing permissions and limitations
	* under the License.
	*
	* gp_stats_collector.c
	*
	* IDENTIFICATION
	* gpcontrib/gp_stats_collector/src/gp_stats_collector.c
	*
	*-------------------------------------------------------------------------
	*/

Conversation

Vlasdislav commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Type of Change

Breaking Changes

Test Plan

Impact

Checklist

Additional Context

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

tuhaihe May 21, 2026

Choose a reason for hiding this comment

Uh oh!

tuhaihe commented May 21, 2026

Uh oh!

tuhaihe May 21, 2026

Choose a reason for hiding this comment

Uh oh!

Vlasdislav May 21, 2026

Choose a reason for hiding this comment

Uh oh!

tuhaihe May 21, 2026

Choose a reason for hiding this comment

Uh oh!

tuhaihe May 21, 2026

Choose a reason for hiding this comment

Uh oh!

tuhaihe May 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Vlasdislav commented May 19, 2026 •

edited

Loading