Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RebenchDB considers failed benchmarks as having runtimes of 0 seconds #224

Open
OctaveLarose opened this issue Jan 9, 2025 · 0 comments

Comments

@OctaveLarose
Copy link

...which mistakenly makes benchmarks appear lightning fast, when in fact they've just failed.

See:
https://rebench.stefan-marr.de/Are-We-Fast-Yet/compare/7da4741e429375bcac6ebe2a4a5977827b599339..affb91df35f834dbc67c6e35fb71ce11ee41c965#exe-comp-somrs-interp

Data: https://rebench.stefan-marr.de/Are-We-Fast-Yet/data/5045.csv.gz

CD, DeltaBlue and Havlak fail on the BC interpreter, and are reported as having a runtime of 0 seconds.

There seems to be some sort of odd behavior in the first place where they hang and kill themselves somehow, so not a normal abort, which I will investigate. Still, RebenchDB shouldn't report them as having a 0 seconds runtime like it currently does.

Importantly, they are not present in the data file at all (as far as I can tell), so it's a RebenchDB bug and not a Rebench bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant