Skip to content

Commit

Permalink
Fix typo in optimization-level.md (#114)
Browse files Browse the repository at this point in the history
**Description**
communcation -> communication
  • Loading branch information
eltociear authored Nov 1, 2023
1 parent 3a4ba14 commit 914c0e0
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/user-tutorial/optimization-level.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ id: optimization-level

# Optimization Level

Currently MS-AMP supports three optimization levels: O1 and O2 and O3. The three levels gradually incorporate 8-bit collective communcation, optimizer and distributed parallel training in an incremental manner. Users can directly set O1/O2 using `msamp.initialize` and set O3 in config file when using DeepSpeed.
Currently MS-AMP supports three optimization levels: O1 and O2 and O3. The three levels gradually incorporate 8-bit collective communication, optimizer and distributed parallel training in an incremental manner. Users can directly set O1/O2 using `msamp.initialize` and set O3 in config file when using DeepSpeed.

- O1: We found that directly transitioning weight gradients from FP32 to FP8 in the Transformer Engine leads to a decrease in accuracy. However, this issue is resolved in O1 through the implementation of FP8 for weight gradients and AllReduce communication. This optimization also has the added benefits of saving GPU memory and reducing communication bandwidth.

Expand Down

0 comments on commit 914c0e0

Please sign in to comment.