Build time regression

[waynexia]

Is your feature request related to a problem or challenge?

We observed a huge increase after upgrading datafusion GreptimeTeam/greptimedb#5417. I run a script to test the build time change day by day since v38.0, and here is the result:

The build time keeps increasing and is almost doubled since v38.0. Given the codebase keeps adding new code, it's expected to see a trend of increasing, but there are still some obvious "platforms" and "jumps", which might be abnormal.

This is the raw data:

build-times.csv

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[waynexia]

A few more details:

The build command I use is cargo build --release --timings --lib --quiet

The latest timing file is (rename it to HTML, restriction from GitHub): Cargo Timing Jan 23 2025.txt

And screenshot:

The datafusion crate (core) spends time for single thread compiling, which would be a major reason for slowing down the build time. I can observe this bottleneck in the downstream greptimedb project as well.

[jayzhan211]

After physical-optimizer, datasource could be a potential target to move out of core.

[findepi]

datasource could be a potential target to move out of core.

related

• [Epic] Extract catalog functionality from the core to make it more modular #10782

The build time keeps increasing and is almost doubled since v38.0. Given the codebase keeps adding new code, it's expected to see a trend of increasing

Did the amount of code double since v38? i doubt that, but @waynexia can you maybe chart build time against amount of code?
Also, can you make sure there is no bias in the measurement? If you build in reverse order and run cargo clean between each steps, will you get the same results? (i believe we regressed on the build time, but let's double check facts before jumping to conclusions)

and "jumps", which might be abnormal.

The biggest jump is at the beginning of August 2024.
Can you perhaps bisect those few days to determine the particular change that caused this?

Is the X axis position based on Commit Date of the current tip commit?

[alamb]

After physical-optimizer, datasource could be a potential target to move out of core.

Yes, 100% splitting out datasource is my next thing I would love to see (and I think it will help build times massively)

Thank you @waynexia for working on this project. It will be appreciated by all 🙏

As @jayzhan211 says, Once we complete this epic (@buraksenn and @berkaysynnada are pretty close)

• [EPIC] Extract remaining physical optimizer out of core #11502

Then I can help organize an effort to break out the data sources

[alamb]

100% making build time better would be really appreciated

[waynexia]

Looking forward to the ongoing refactors!

Also, can you make sure there is no bias in the measurement? If you build in reverse order and run cargo clean between each steps, will you get the same results? (i believe we regressed on the build time, but let's double check facts before jumping to conclusions)

Here is the script, I removed the target after every build:

script

#!/bin/bash

# Configuration
OUTPUT_FILE="build-times.csv"
START_DATE=$(date -d "8 months ago" +%Y-%m-%d)
END_DATE=$(date +%Y-%m-%d)

# Initialize CSV output
echo "date,commit_hash,build_time_seconds,success" > "$OUTPUT_FILE"


# Iterate day-by-day
current_date="$START_DATE"
while [[ "$current_date" < "$END_DATE" ]]; do
  # Find latest commit for the day
  git checkout main
  commit_hash=$(git log --until="$current_date 23:59" --first-parent -1 --format=%H)

  if [[ -n "$commit_hash" ]]; then
    echo "Processing $current_date ($commit_hash)..."
    
    # Checkout commit (detached HEAD)
    git checkout --force "$commit_hash" > /dev/null 2>&1
    
    # Clean and build
    rm -rf target/release
    cargo metadata --quiet > /dev/null
    start_time=$(date +%s)
    cargo build --release --timings --lib --quiet
    build_exit_code=$?
    end_time=$(date +%s)
    build_time=$((end_time - start_time))

    # Record results
    echo "$current_date,$commit_hash,$build_time,$build_exit_code" >> "$OUTPUT_FILE"
  else
    echo "No commit found for $current_date"
  fi

  # Move to next day
  current_date=$(date -d "$current_date + 1 day" +%Y-%m-%d)
done

# Return to original branch
git checkout main 

[waynexia]

I located the biggest jump happens on #11681, commit 3438b35. But I can't tell the reason. I go through it and it's just a normal logic change patch, no new dep or feature was introduced 🤔

[alamb]

I located the biggest jump happens on #11681, commit 3438b35. But I can't tell the reason. I go through it and it's just a normal logic change patch, no new dep or feature was introduced 🤔

I wonder if the new analyzer rule caused the jump 🤔 (datafusion/optimizer/src/analyzer/expand_wildcard_rule.rs)

One theory could be that the tree node walking code is substantial and slow to compile (it is a lot of templates 🤔 )

However, a new analyzer rule in datafusion-optimizer doesn't explain longer compile times in the datafusion crate 🤔

[waynexia]

After removing the WildcardOptions (by replacing it with an empty structure) I can see the build time drops. Removing the rule itself and the change in core doesn't help. It looks like the change to Expr is the root cause, though I don't know the reason...

One guess is that both Expr and LogicalPlan are key enums used in many places, especially many type parameters (like the TreeNode). Thus a slight change to them will be amplified by a lot.

I have some thoughts on this specific case, like putting the expansion somewhere else, or changing some Fn type parameters to trait objects. But I can't tell how good it will be, and the balance between runtime overhead and build time consumption. Or we can wait for those refactors until we figure out why WildcardOptions causes that.