Can one of the admins verify this patch?

Admins can comment ok to test to allow this one PR to run or add to allowlist to allow all future PRs from the same author to run.

c.f. #10898 #10905

@martindurant @crusaderky

Here's a very minimal reproducer involving awkward array:

import json
import awkward as ak

from dask.base import tokenize, normalize_token

a = ak.Array([1, 2, 3, 4, 5])

layout, report = ak.typetracer.typetracer_with_report(a.layout.form_with_key())

b = ak.Array(layout)

print(report.data_touched, report.shape_touched)

d = {"what": b}

normalized = sorted(d.items(), key=lambda x: str(normalize_token(x)))

print(report.data_touched, report.shape_touched)

normalized = sorted(d.items(), key=str)

print(report.data_touched, report.shape_touched)

resulting in:

(coffea-dev-py311) lgray@visitor-122024646 coffea % python form_madness_repr_minimal.py
[] []
[] []
['node0'] ['node0', 'node0']

Which means str is intrusive to these objects while normalize_token itself is not, and could be instead handled through __dask_tokenize__. This PR makes it so that __dask_tokenize__ is the sole entrypoint to the tokenize machinery.

This should aid in the deterministic keys effort in the long run.

This appears to cause tokenization of SubgraphCallables that are exactly equal to now become equal.
I'm not sure that's desired or not in this phase of development.

Darn, this methodology has the unfortunate outcome that we are sensitive to the history of the graph. So if you start from a different graph and optimize it, that is not the same as constructing an optimized graph by hand.

This is why the SubgraphCallable tests are failing.

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

     15 files  ± 0       15 suites  ±0   3h 11m 48s ⏱️ - 6m 0s
 12 989 tests + 1   12 059 ✅ ±0     930 💤 + 1  0 ❌ ±0 
160 522 runs  +15  144 030 ✅  - 4  16 492 💤 +19  0 ❌ ±0 

Results for commit f1e196f. ± Comparison against base commit 7726d31.

♻️ This comment has been updated with latest results.

Hmm not fixed by including #10883 (actually this issue is broken further by that PR).

OK - this seems to get it all in reasonable shape, doesn't break any existing rules, and now has a test that shows what the problem is.

After getting it all in shape I tried merging #10883 again and that PR still causes the test to break.

@lgray I'd be tempted to remove the test dependency on awkward, and instead make a mock type that throws an error in __str__.

Superseded by #10919