NVIDIA Blackwell codegen

[johnnynunez]

Blackwell B100/B200 codegen 10.0
Blackwell RTX 50 codegen 12.0

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[youkaichao]

does it have some requirements on the nvcc version?

[johnnynunez]

does it have some requirements on the nvcc version?

yes, cuda 12.7 for B100/B200 and cuda 12.8 for RTX50.
Flash attention: Dao-AILab/flash-attention#1436

in pytorch is greater than 12.6

[tlrmchlsmth]

does it have some requirements on the nvcc version?

yes, cuda 12.7 for B100/B200 and cuda 12.8 for RTX50. Flash attention: Dao-AILab/flash-attention#1436

in pytorch is greater than 12.6

Thanks, I saw the flash attention changes -- good to know about the B100/B200 vs RTX difference there

[johnnynunez]

does it have some requirements on the nvcc version?

yes, cuda 12.7 for B100/B200 and cuda 12.8 for RTX50. Flash attention: Dao-AILab/flash-attention#1436
in pytorch is greater than 12.6

Thanks, I saw the flash attention changes -- good to know about the B100/B200 vs RTX difference there

I know more information:
10.0 blackwell b100/b200
10.1 blackwell thor
11.0 blackwell digits? only this I have to ask it
12.0 blackwell rtx50

[johnnynunez]

@tlrmchlsmth check it

[youkaichao]

does it have some requirements on the nvcc version?

I mean, will it break old nvcc if old nvcc cannot recognize these arch numbers?

[johnnynunez]

does it have some requirements on the nvcc version?

I mean, will it break old nvcc if old nvcc cannot recognize these arch numbers?

Yes, old nvcc can’t generate code for blackwell. I tried on rtx5090 using 12.6.3 and it’s not working(“unsupported architecture”). But new desktop blackwell has hopper capabilities like cuda thread clusters so it should work at the same 9.0 codegen.

this means that flash attention v3 it should be compatible and the upcoming versions