Support Writing Iceberg metadata

[zhousun]

What feature are you requesting?

Currently pg_mooncake is wring delta metadata to the object store. But it would be ideal to also write iceberg version.

Why are you requesting this feature?

Fit better into the iceberg ecosystem.

What is your proposed implementation for this feature?

#[cxx::bridge]
mod ffi {
    extern "Rust" {
        fn DeltaInit();

        fn DeltaCreateTable(
            table_name: &CxxString,
            path: &CxxString,
            options: &CxxString,
            column_names: &CxxVector<CxxString>,
            column_types: &CxxVector<CxxString>,
        ) -> Result<()>;

        fn DeltaModifyFiles(
            path: &CxxString,
            options: &CxxString,
            file_paths: &CxxVector<CxxString>,
            file_sizes: &CxxVector<i64>,
            is_add_files: &CxxVector<i8>,
        ) -> Result<()>;
    }
}

Implement the corresponding functions for iceberg, with same semantics.
Catalog support for iceberg is more important than delta, but for MVP it is fine to just write to filesystem.

[umeshkacha]

is there any ETA for this?

[dpxcc]

It's on our shortlist, but we don't have a concrete ETA yet.

The challenges are:

1. There isn't a good library for writing Iceberg metadata, except by spinning up a JVM. We're hoping that iceberg-rust will soon make this easier so we can adopt it. Alternatively, we could manually write the metadata ourselves since we only use a small subset of the Iceberg spec, though that's not ideal.
2. Writing Iceberg metadata is more expensive than Delta Lake because it's based on snapshots. This would require batched writes and compaction to deliver a better user experience.

[zhousun]

@umeshkacha
Another big difference is Iceberg usually requires a catalog to work, which makes the setup a lot more complex for regular postgres user.
This is changing with S3 Table Bucket, I am working on a prototype with S3 Table Bucket.

[umeshkacha]

Sounds good Zhou. Thanks for the update. Any ETA? Thanks, Umesh

…

On Sat, Jan 18, 2025 at 1:05 AM Zhou Sun ***@***.***> wrote: @umeshkacha <https://github.com/umeshkacha> Another big difference is Iceberg usually requires a catalog to work, which makes the setup a lot more complex for regular postgres user. This is changing with S3 Table Bucket <https://www.mooncake.dev/blog/s3tables>, I am working on a prototype with S3 Table Bucket. — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE45HS3XTSP5PTXEKJKTL32LFLRTAVCNFSM6AAAAABRN7EV26VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOJZGA3DGMRQHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>