Release DataFusion 44.0.0

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Last release was https://crates.io/crates/datafusion/43.0.0 November 8th, 2024 so next major release would be around December 8, 2024

• Voting Thread: https://lists.apache.org/thread/78zxfpn7fh1csz6fdolpgh3lb4ltxngp

Prior release tickets:

• 43.0.0 Release DataFusion 43.0.0 #12470

Items to possibly fix before release

• Regression: Invalid comparison operation: Utf8 == Utf8View error during LEFT ANTI JOIN #13510
• [Epic] A Collection of Additional UTF8View support tickets #13504

Improved upgrade experience:

• Make migration to Signature::nullary in 44.0.0 easier / less confusing #13763
• Make DocBuilder migration in 44.0.0 easier #13764
• Making the recursive dependency an optional feature #13766
• Deprecate ScalarUDFImpl::return_type #13717

Downstream project testing:

• datafusion python: chore: Upgrade to DataFusion 44 datafusion-python#972
• comet: Test DataFusion 44.0.0 with Comet #13835
• delta.rs: Test DataFusion 44.0.0 with delta.rs #13834
• Test DataFusion 44.0.0 with Sail #13855

[alamb]

I would love to get this one in:

• Perf: Allow User defined functions to potentially reuse their argument arrays (to avoid new allocations) #13516

Some things to highlight (that are already merged)

• Support vectorized append and compare for multi group by #12996 from @Rachelint as well

[alamb]

Also this hopefully:

• Add SessionConfig reference to ScalarFunctionArgs #13519

[alamb]

@Omega359 and @andygrove suggested #13525 (comment) that for this release we

we file tickets as part of every release to test DF with latest versions of common external dependencies (datafusion-python, iceberg, lance, delta, etc) as part of the gating criteria for a release. Perhaps this would be as simple as changing the DF version in those projects and running their test suite. Any blockers that are found in DF have tickets filed and are fixed, other blockers in the respective projects have tickets filed in those projects

Perhaps for 44.0.0 we can try with own subprojects (Ballista, Comet, DF Python, DF Ray)

[alamb]

I started gathering a list of items we think we should fix prior to the release in the description

[alamb]

Does anyone have any opinion about holding the DataFusion 44 release for the next major arrow release?

• Release arrow-rs / parquet major version 54.0.0 (December 2024) arrow-rs#6342

That would fix a bunch of the StringView issues, but would likely delay DataFusion 44 for a few more weeks

[Omega359]

It's soon to be the holiday season so I'm all for cooking the release a bit more.

[alamb]

I would personally love to see DataFusion 44.0.0 be lauded as "super stable" and have few upgrade issues (we would largely achieve this by testing upgrades with other projects prior to release)

The downside is that the changeset in DataFusion 44.0.0 might be larger

[andygrove]

I'd like to see an upgrade guide for the 44 release (and am willing to take the lead on this). I am trying to upgrade Comet now and am running into some issues.

I am +1 for waiting for the next Arrow release.

[andygrove]

I filed #13702

[andygrove]

How would everyone feel about increasing the length of release votes from 3 days to 7 days to give downstream projects more time to test the release?

Sometimes the vote starts on a Friday and passes on a Monday and we can't expect everyone to be working weekends.

[alamb]

How would everyone feel about increasing the length of release votes from 3 days to 7 days to give downstream projects more time to test the release?

Sometimes the vote starts on a Friday and passes on a Monday and we can't expect everyone to be working weekends.

In my opinion, we would ideally do the testing before we make the release candidate (as the overhead of making RCs is non trivial). However, I am not opposed to extending the voting timeline if that would get us more testing time

[andygrove]

Testing before we create the RC makes sense

[andygrove]

I'm still working on updating Comet to work with latest DF from main (upgrading from 43.0.0). This is quite a disruptive upgrade for us so far.

[alamb]

I have been thinking about this upgrade and the next arrow release

• Release arrow-rs / parquet major version 54.0.0 (December 2024) arrow-rs#6342

Specifically I think we need this upgrade in order to resolve some of the issues in 44 related to string view

• [Epic] A Collection of Additional UTF8View support tickets #13504

However, what I would like to propose is:

1. Plan to release DataFusion 44.0.0 without also upgrading arrow-rs

The downside of this plan is

1. it will take longer to get the arrow upgrade (and this the string view fixes) throughout the ecosystem

The upsides are

1. It will reduce the complexity of the DataFusion 44 upgrade
2. it will give us a longer period to test the upgrade integration DataFusion

If someone is critically waiting on the string view fixes, we can also contemplate making another arrow incremental release

[alamb]

I am also going to coordinate with the delta-rs folks to try and test upgrading prior to release:

• chore: upgrade to datafusion 43 delta-io/delta-rs#2886 (comment)

[timsaucer]

Starting an issue and branch in datafusion-python : apache/datafusion-python#972

[rluvaton]

I'll give my 2 cents from my experience regarding how other popular project handle this issue

In Node.js (I'm a core collaborator) before each release we run the tests of the most popular packages with the new node version and making sure we are not breaking them, if we are we let them know. this is really helpful to avoid breaking the entire world

this process is a CITGM, you can watch this talk for more info

[andygrove]

I am +1 for releasing DF 44 without the next major version of Arrow. However, there are some bug fixes and performance improvements in Arrow that I would like to use so would like to see a new minor release (and am happy to help with this if needed).

[mbwhite]

Hi - would like to get #13647 out as soon as is practical - a small fix but does unblock my use of
DataFusion for Substrait.

Thanks!

[alamb]

I am +1 for releasing DF 44 without the next major version of Arrow. However, there are some bug fixes and performance improvements in Arrow that I would like to use so would like to see a new minor release (and am happy to help with this if needed).

• I have filed Release arrow-rs / parquet minor version 53.4.0 (Jan 2025) arrow-rs#6887 to track this potential release

[alamb]

I am looking at #13510 now, and then plan to make a PR with release notes, verison update, etc

Update:

• Regression: Invalid comparison operation: Utf8 == Utf8View error during LEFT ANTI JOIN #13510 has a workaround and is no worse than in 43 (though it is a regression since 42) so in my opinion this is not a blocker for the 44 release (it would be nice to fix though)

[andygrove]

@alamb The miri issue does not seem to be resolved for Comet - #13835 (comment)

Is there anything I should have done other than set default-features=false on all DF crates?

[alamb]

@alamb The miri issue does not seem to be resolved for Comet - #13835 (comment)

Is there anything I should have done other than set default-features=false on all DF crates?

That sounds like it is the right thing to me 🤔

• Since @buraksenn did the work to make this optional in [minor] make recursive package dependency optional  #13778, perhaps they have some idea?

[alamb]

Since @buraksenn did the work to make this optional in [minor] make recursive package dependency optional #13778, perhaps they have some idea?

Update I think I know what is wrong -- working on a fix now - #13887

[alamb]

I think all we are waiting on now is someone to review / approve (or fix in some other way) the recrusive-protection feature fix:

• Fix recursive-protection feature flag #13887

[alamb]

Update, when preparing to make a final RC I found another issue:

• swap_hash_join is no longer public so comet doesn't compile #13898

I have a proposed fix here:

• Fix visibility of swap_hash_join to be pub  #13899

[alamb]

Ok, we are clear for 🚀 from what I can tell -- the comet upgrade is working. I will merge the following PR

• Prepare for 44.0.0 release: version and changelog #13882 (thanks @xudong963 and @phillipleblanc for the review 🙏 )
  And then make a Release Candidate

[alamb]

I made a release candidate: https://lists.apache.org/thread/78zxfpn7fh1csz6fdolpgh3lb4ltxngp

[shehabgamin]

I made a release candidate: https://lists.apache.org/thread/78zxfpn7fh1csz6fdolpgh3lb4ltxngp

I'll re-test Sail with the release candidate tonight!

[shehabgamin]

Test 2 complete! Smooth Sailing testing commit 073a3b1 from DataFusion Release Candidate branch.

Test details here: #13855 (comment)

[alamb]

Thanks @shehabgamin

[timsaucer]

If we can make a new RC I would recommend cherry picking this fix for initcap: #13909

[alamb]

As @timsaucer mentions above (and on the mailing list): https://lists.apache.org/thread/34xy1nm47tkrr8lszxzwcrlf5b47h333

There was a regression introduced into initcap that has been sinced fixed: #13909

I will make a new RC

[alamb]

have created a new RC and started a new vote thread: https://lists.apache.org/thread/p0hyr78zs03f3snj0mh3vqm6c3kjwshn

[alamb]

The release has been approved and published to crates.io: https://lists.apache.org/thread/7w64xqzn71plvmxy61v6px8bb994kr79 🎉

https://crates.io/crates/datafusion/44.0.0

I have also made a small PR to improve the docs for the next release:

• Update release instructions for 44.0.0 #13959

[niebayes]

@alamb Hi, is the release for datafusion-cli delayed?

[alamb]

@alamb Hi, is the release for datafusion-cli delayed?

Good catch @niebayes -- I forgot to push this to crates.io. I have done so now and datafusion-cli 44.0.0 is available:

https://crates.io/crates/datafusion-cli/44.0.0

[alamb]

Here is a ticket to run ClickBench on DataFusion 44:

• Update ClickBench benchmarks with DataFusion 44.0.0 #13983

[alamb]

Filed ticket for the next relese:

• Release DataFusion 45.0.0 #14008