Dask and delta-rs integeration
This release builds a wrapper around the Rust package called delta-rs
and uses dask for parallel reading.
Features:
- Reads the parquet files based on delta logs parallelly using the dask engine
- Supports all three filesystems like s3, azurefs, gcsfs
- Supports some delta features like
- Time Travel
- Schema evolution
- parquet filters
- row filter
- partition filter
- Query Delta commit info - History
- vacuum the old/ unused parquet files
- load different versions of data using DateTime.