Todo for MVP

[SandersAaronD]

This is a checklist for our MVP. Essential is essential, nice to have is nice to have; if any individual "nice to have" blocks us for an extended period we probably prefer to launch so that we can get feedback because it is possible to likely that early adopters will have asks that will be 1) easy to deliver and 2) not things we will think of otherwise. Nice to haves marked with an asterisk are things nobody else does.

Essential

Preview Give feedback

1. Instrumenting code and infrastructure entirely via a python exporter that requires a minimum of steps to install (install, import, login, init)
2. Tracking user-defined logs and metrics, split up into “runs”
3. Visualizing those from the UI
4. Successfully monitor "processes" grouped under "groups" grouped under "projects" with UI drill down similar to k6
5. Ensuring that this continues to work well for distributed training

Later

Preview Give feedback

1. PyTorch code profiling for the GPU*
2. One-click alerting on training run status (e.g. failed, crashed or timed out runs)*
3. Implement tracing for distributed jobs to keep track of their network usage and timing*
4. GPU monitoring via DCGM exporter (or: by taking code from DCGM exporter, which is OSS)
5. Making sure system monitoring includes all metrics in node exporter
6. Ensuring it is easy to download data for analysis
7. Correlating the run with the underlying infrastructure (e.g. cluster managers like Kubernetes and Slurm)
8. Ensuring that this can integrate directly with Grafana agent where it is installed to improve reliability
9. Provide a way to link between training runs and resource usage