Is git_tracked_only supposed to improve hashing performance?

[kostyay]

Hi
We have a huge amount of tasks in our repo, more than 3000 to be exact.
The initial stage which consists of going over each task and calculating the file digest takes a lot of time (up to 5 minutes in some cases).
We are not currently using git_tracked_only but I'm wondering if that will improve performance?
Baur is executed in our CI, so everything there is already a part of git.
Ultimately I wonder if comparing to master and building only directories that changed compared to master can speed things up for us as usually developers work only on 1 app at a time.
Any ideas?

[fho]

Hello Kostyay,

5min is very long.

We are not currently using git_tracked_only but I'm wondering if that will improve performance?

Depends :-)

git_tracked_only=true will filter out input files that are not tracked by git. If you have temporary files in the repository directories that are not tracked by git but matched by some input file specifications, those would be hashed unnecessarily.
If that is not the case, it won't improve performance.
(Filtering out untracked files actually results in a bit more worked but the performance impact is negligible)

Are you using a baur version > 3.3? Since version 3.3 digests of tracked files are read from the git repository instead of calculated.

I recommend to run baur (status, run or ls-inputs) with --verbose and:

1. Check if the digests can be read from git during the CI runs.
   Please verify that a message like
   gitobjectid: loaded 46689 object IDs from git repository [...].
   shows up.
   Then check that messages like gitobjectid: object id not in cache, calculating ID for [...] are not printed prevalently for most of the files.
   Additionally I recommend to run baur with the new --require-clean-git-worktree parameter in CI, to ensure that resolving input files to untracked/modified ones result in a failure.
2. If the digests can be read from the git repo, gather from the verbose output which operations cost most of the time.

Ultimately I wonder if comparing to master and building only directories that changed compared to master can speed things up for us as usually developers work only on 1 app at a time.

Yes, that reminds me of https://github.com/mbtproject/mbt. It should be faster and might be sufficient for your scenario.