v2.5.0

[xuchen-plus]

LakeSoul 2.5.0 Release Note

What's New

1. Python Reader supports PyTorch, PyArrow, Pandas, Ray, and distributed execution;
2. Support Spark Gluten Vectorized Engine;
3. Spark SQL supports Compaction, Rollback and other Call Procedures;
4. Flink CDC’s entire database synchronization supports MySQL, PostgreSQL, PolarDB, and Oracle;
5. Support streaming and batch export to MySQL, PostgreSQL, PolarDB, and Apache Doris;

更新内容

1. Python Reader 支持 PyTorch、PyArrow、Pandas、Ray，支持分布式执行；
2. 支持 Spark Gluten Vectorized Engine；
3. Spark SQL 支持 Compaction、Rollback 等 Call Procedures；
4. Flink CDC 整库同步支持 MySQL、PostgreSQL、PolarDB、Oracle；
5. 支持流式、批式出湖至 MySQL、PostgreSQL、PolarDB、Apache Doris；
6. 优化 NativeIO 性能；

What's Changed

• [Spark]rename MetaVersion at lakesoul-spark as SparkMetaVersion by @Ceng23333 in [Spark]rename MetaVersion at lakesoul-spark as SparkMetaVersion #353
• [Metadata]Replace table_info.table_schema with arrow kind schema (Backward Compatibility) by @Ceng23333 in [Metadata]Replace table_info.table_schema with arrow kind schema (Backward Compatibility) #354
• [Python][Dataset] Add Ray reading support by @codingfun2022 in [Python][Dataset] Add Ray reading support #355
• [Spark]optimize incremental read and fix compact operation cause column disorder bug by @F-PHantam in [Spark]optimize incremental read and fix compact operation cause column disorder bug #352
• [Rust] Create Rust CI by @Ceng23333 in [Rust] Create Rust CI #356
• [Rust][Metadata]Create Rust MetadataClient & add CI test cases by @Ceng23333 in [Rust][Metadata]Create Rust MetadataClient & add CI test cases #357
• [Rust][NativeIO]Use stable rustc for lakesoul-io feature default by @Ceng23333 in [Rust][NativeIO]Use stable rustc for lakesoul-io feature default #358
• [Python][Rust][Metadata] Update python metadata interface && Full arrow types test by @Ceng23333 in [Python][Rust][Metadata] Update python metadata interface && Full arrow types test #359
• [Spark] Spark Sql Support 'drop partition' Operation by @F-PHantam in [Spark] Spark Sql Support 'drop partition' Operation #360
• [Python]python deserialized schema from java by @Ceng23333 in [Python]python deserialized schema from java #361
• [Python] Fix wheel building; update version to 1.0.0b1 by @codingfun2022 in [Python] Fix wheel building; update version to 1.0.0b1 #362
• [Rust][Metadata]Asynchronized rust metadata method by @Ceng23333 in [Rust][Metadata]Asynchronized rust metadata method #365
• Add some rust test cases by @zhaishuangszszs in Add some rust test cases #364
• [Datafusion]Implement LakeSoul Catalog by @Ceng23333 in [Datafusion]Implement LakeSoul Catalog #366
• [Rust] add upsert test cases by @zhaishuangszszs in [Rust] add upsert test cases #367
• [Flink] update fury version to 0.4 by @xuchen-plus in [Flink] update fury version to 0.4 #368
• refine upsert test by @zhaishuangszszs in refine upsert test #369
• [Spark] support call sql syntax by @moresun in [Spark] support call sql syntax #370
• [Rust]DataFusion version upgraded to 33.0.0 by @Ceng23333 in [Rust]DataFusion version upgraded to 33.0.0 #372
• [Spark] Support Gluten Vectorized Engine by @xuchen-plus in [Spark] Support Gluten Vectorized Engine #374
• [Flink] Support oracle cdc source by @ChenYunHey in [Flink] Support oracle cdc source #375
• [NativeIO] Use rust block api in file read by @xuchen-plus in [NativeIO] Use rust block api in file read #377
• [Flink] Add export to external dbs for LakeSoul's tables by @ChenYunHey in [Flink] Add export to external dbs for LakeSoul's tables #376
• [Rust] Add LakeSoulHashTable Sink for DataFusion by @Ceng23333 in [Rust] Add LakeSoulHashTable Sink for DataFusion #382
• [NativeIO] Enable parquet rowgroup prefetch. Support s3 host style access by @xuchen-plus in [NativeIO] Enable parquet rowgroup prefetch. Support s3 host style access #384
• [Rust]fix hash value to spark_murmur3 by @Ceng23333 in [Rust]fix hash value to spark_murmur3 #385
• [BugFix]Fails when create table with nullable hash colmun by @Ceng23333 in [BugFix]Fails when create table with nullable hash colmun #387
• [Flink] Add Jdbc cdc sources and sinks by @ChenYunHey in [Flink] Add Jdbc cdc sources and sinks #381
• [Python] fix python meta config parse logic by @xuchen-plus in [Python] fix python meta config parse logic #388
• [Project/Doc] Bump version to 2.5.0 and update docs by @xuchen-plus in [Project/Doc] Bump version to 2.5.0 and update docs #389
• Bump postcss from 8.4.23 to 8.4.33 in /website by @dependabot in Bump postcss from 8.4.23 to 8.4.33 in /website #396
• Bump @babel/traverse from 7.21.5 to 7.23.7 in /website by @dependabot in Bump @babel/traverse from 7.21.5 to 7.23.7 in /website #393
• Bump follow-redirects from 1.15.2 to 1.15.4 in /website by @dependabot in Bump follow-redirects from 1.15.2 to 1.15.4 in /website #399
• Bump org.apache.avro:avro from 1.11.0 to 1.11.3 in /lakesoul-spark by @dependabot in Bump org.apache.avro:avro from 1.11.0 to 1.11.3 in /lakesoul-spark #394
• Bump com.google.guava:guava from 30.1.1-jre to 32.0.0-jre in /lakesoul-presto by @dependabot in Bump com.google.guava:guava from 30.1.1-jre to 32.0.0-jre in /lakesoul-presto #395
• [Rust] Update arrow rs dependencies by @xuchen-plus in [Rust] Update arrow rs dependencies #400

New Contributors

• @zhaishuangszszs made their first contribution in Add some rust test cases #364

Full Changelog: v2.4.1...v2.5.0

---

This discussion was created from the release v2.5.0.