v3.0.2

What's Changed

Bug Fixes:

• Bug: Monitor on/off should not be part of State by @jeremyfowers in #206
• Fix help menu by @jeremyfowers in #207
• Prevent onnx 1.16.2 from being deployed by @jeremyfowers in #208

Full Changelog: v3.0.1...v3.0.2

v3.0.1

What's Changed

• Fix load_build bug by @jeremyfowers in #204

Full Changelog: v3.0.0...v3.0.1

v3.0.0

Summary

This release is the culmination of a month of work in the refresh branch to re-architect TurnkeyML for the future.

The high-level changes are:

• Stages have been replaced by Tools, which each have their own arguments parser to enable maximum configurability
• Sequences are no longer hardcoded lists of Stages - now users can enter any list of Tools they like
• Everything is a Tool: model discovery, benchmarking, and even cache management functionality (report, list, delete, etc.) are all now modular Tools.
• Plugins now primarily add new Tools. These new Tools can do a lot more than Stages used to be able to, such as:
  • Add a new frontend into the sequence, as a replacement for model discovery. For example, load an LLM checkpoint.
  • Replace the backend of the sequence with something other than our standard benchmarking flow. For example, run an accuracy test.
  • Add utility functions, such as managing the cache or built-in models.
• There is a brand new command line interface and API to expose the new way of doing things.
• The tools documentation has been completely re-written.

Breaking Changes

There are almost too many to list since the whole project has been re-architected. Here are the major things to be aware of:

• Stage classes need to be ported to on to the Tool class.
  • Any reference to stage should be changed to tool, for example the stage_status:: stat is now tool_status::
  • The names of most Tools have been renamed to fit our new naming scheme (see contribution guide).
• There is no longer a notion of benchmark_status, since benchmarking is now a Tool. It's now tool_status::benchmark.
• The build_model() API has been removed. Instead, Tools should be called as their own standalone APIs.
• The benchmark_files() API has been re-named to evaluate_files().
• The repository directory structure has been redone, which changes a lot of imports.
  • For example, turnkeyml.state has State and load_state().
  • See the code organization doc for the full set of changes.

v2.0.3

What's Changed

• Specify how errors should be handled during decoding by @danielholanda in #148

Full Changelog: v2.0.2...v2.0.3

v2.0.2

What's Changed

• Do not set benchmarking error status when a benchmark is skipped by @jeremyfowers in #146
• Relax ORT requirements by @danielholanda in #147

Full Changelog: v2.0.1...v2.0.2

v2.0.1

What's Changed

This is a hotfix release for this one bug fix:

• Ensure child subprocesses are killed on timeout by @danielholanda in #144

v2.0.0

What's Changed

This update improves TKML's support for LLMs and significantly improves data quality for mass-benchmarking scenarios.

LLM Support:

• Add LLaMA2 models by @jeremyfowers in #130
• Add phi2 to transformers corpus by @jeremyfowers in #135
• Fix LLM support for ONNX benchmarking by @jeremyfowers in #134

Data Quality Improvements:

• Add cache warmup for OnnxRuntime Benchmarking by @ramkrishna2910 in #133
• Save the exception into error_log when the subprocess is killed by @jeremyfowers in #139
• Save device name stat prior to running stages by @jeremyfowers in #138
• Capture PyTorch benchmarking outputs into a log file by @jeremyfowers in #121

Quality of Life

• Set a default benchmarking timeout by @danielholanda in #124

Bug fixes

• Ensure that turnkey cache benchmark can also target individual builds by @danielholanda in #123
• Update status.py to safely access eval stats for status printout by @jeremyfowers in #122
• Avoid parameter count overflow on onnx models by @danielholanda in #126
• Fix hang on OEM info read by @ramkrishna2910 in #128

Breaking Changes

• Plugin interface: BaseRT.device_name() is now a static method instead of a property.

Full Changelog: v1.1.3...v2.0.0

v1.1.3

Summary

This update improves turnkey's mass-benchmark abilities, particularly for LLMs and large pre-compiled caches.

Improvements

• New features:
  • Quickly benchmark entire caches by @jeremyfowers in #112
  • Configurable verbosity for turnkey status by @jeremyfowers in #116
• Data quality:
  • turnkey arguments and timestamp are stored in the stats and report @jeremyfowers in #105
  • Record the contents of error logs into stats and report by @jeremyfowers in #111
  • torch-eager and torch-compiled runtimes now perform cache warmup and can be extended to implement other torch-based runtimes (e.g., IPEX) by @jeremyfowers in #119

Bug Fixes

• Fixed confusing error reporting in no-build runtimes (e.g., PyTorch, IPEX, etc.) by @jeremyfowers in #106
• Use the correct 6B parameter checkpoint for GPT-J instead of the "tiny" one by @jeremyfowers in #108
• Fix FLOPs counting for large models by @pcolange in #109

PyPI: https://pypi.org/project/turnkeyml/1.1.3/
Full Changelog: v1.1.2...v1.1.3