Decouple lakehouse writes from transaction lifecycle

[dpxcc]

What feature are you requesting?

Decouple lakehouse writes from transaction lifecycle

Why are you requesting this feature?

Each transaction creates a new Delta Lake log record, which is inefficient for small INSERTs

What is your proposed implementation for this feature?

• Add a new command, e.g. mooncake.export_table(), to manually trigger writes to the lakehouse
• Parquet files will continue to be uploaded to the lake when generated, as they are now
• Add a new metadata table to track changes that have not yet been exported to the lakehouse

[dentiny]

Hi I'm interested in this ticket. Could you please assign it to me?

If you think my progress slow, feel free to ping me any time, I will try to prioritize and expedite. Thank you!