Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable fused topK_softmax kernel for hip #14

Merged
merged 4 commits into from
May 16, 2024
Merged

Conversation

divakar-amd
Copy link

No description provided.

@divakar-amd divakar-amd added the enhancement New feature or request label May 9, 2024
@divakar-amd divakar-amd self-assigned this May 9, 2024
@divakar-amd divakar-amd force-pushed the fused_topK_softmax branch from db178b3 to 428609c Compare May 9, 2024 22:05
@divakar-amd divakar-amd marked this pull request as draft May 9, 2024 22:12
@divakar-amd divakar-amd force-pushed the fused_topK_softmax branch from 428609c to 9f6113f Compare May 9, 2024 22:26
@gshtras
Copy link
Collaborator

gshtras commented May 10, 2024

Measuring performance (latency) on the new fused implementation

@divakar-amd
Copy link
Author

Tested on rocm/pytorch-private:vllm_0.4.0_ROCm6.1_public_20.04_PyTorch2.3_tunable
Enabling fused topK_softmax gives about 4-5% gain.
image

@divakar-amd divakar-amd marked this pull request as ready for review May 10, 2024 22:12
@shajrawi
Copy link
Collaborator

Once approved we need to also do an upstream PR

Copy link

@hthangirala hthangirala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@hthangirala hthangirala merged commit e3ae076 into main May 16, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants