Skip to content
This repository has been archived by the owner on Apr 8, 2024. It is now read-only.

Adding perf metrics to each node in both single and multi node scripts #192

Merged
merged 20 commits into from
Dec 9, 2021

Conversation

jfomhover
Copy link
Contributor

@jfomhover jfomhover commented Dec 8, 2021

This PR connects the dots between multiple precedent PRs:

It implements the performance metrics in single node and multi node scripts. Those metrics represent performance summary per node. When running on distributed, we use step to store metrics under one metric name for all nodes.

This is what those metrics look like in AzureML:
image

Here's an example of multinode perf (extract)
image

@jfomhover jfomhover temporarily deployed to mlops December 8, 2021 23:29 Inactive
@jfomhover jfomhover temporarily deployed to mlops December 8, 2021 23:30 Inactive
@jfomhover jfomhover temporarily deployed to mlops December 9, 2021 00:44 Inactive
@github-actions
Copy link

github-actions bot commented Dec 9, 2021

Unit Test Results for Build

  1 files  ±0    1 suites  ±0   22s ⏱️ +7s
82 tests ±0  82 ✔️ ±0  0 💤 ±0  0 ±0 

Results for commit 5493c75. ± Comparison against base commit c7298fa.

♻️ This comment has been updated with latest results.

@jfomhover jfomhover temporarily deployed to mlops December 9, 2021 05:41 Inactive
@jfomhover jfomhover temporarily deployed to mlops December 9, 2021 05:43 Inactive
@jfomhover jfomhover temporarily deployed to mlops December 9, 2021 05:45 Inactive
@jfomhover jfomhover temporarily deployed to mlops December 9, 2021 05:52 Inactive
@jfomhover jfomhover changed the title Add perf metrics to single node Adding perf metrics to each node in both single and multi node scripts Dec 9, 2021
@jfomhover jfomhover added this to the Standardization milestone milestone Dec 9, 2021
@jfomhover jfomhover added the enhancement New feature or request label Dec 9, 2021
@jfomhover jfomhover temporarily deployed to mlops December 9, 2021 06:02 Inactive
@jfomhover jfomhover marked this pull request as ready for review December 9, 2021 06:04
@jfomhover jfomhover temporarily deployed to mlops December 9, 2021 18:20 Inactive
@github-actions
Copy link

github-actions bot commented Dec 9, 2021

Code Coverage

Package Line Rate Branch Rate Complexity
common 84% 0% 0
scripts 100% 0% 0
scripts.data_processing.generate_data 93% 0% 0
scripts.data_processing.lightgbm_data2bin 95% 0% 0
scripts.data_processing.partition_data 92% 0% 0
scripts.inferencing.custom_win_cli 65% 0% 0
scripts.inferencing.lightgbm_c_api 35% 0% 0
scripts.inferencing.lightgbm_python 95% 0% 0
scripts.inferencing.treelite_python 94% 0% 0
scripts.model_transformation.treelite_compile 92% 0% 0
scripts.sample 93% 0% 0
scripts.training.lightgbm_python 80% 0% 0
Summary 81% (1057 / 1303) 0% (0 / 0) 0

@jfomhover jfomhover merged commit 65b5619 into main Dec 9, 2021
@jfomhover jfomhover deleted the jfomhover/singlenodeperfintegration branch December 9, 2021 20:45
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add system metrics (cpu utilization, gpu utilization, memory utilization, etc).
2 participants