[V1][Spec Decode] Ngram Spec Decode

[LiuXiaoxuanPKU]

This PR tries to add ngram spec decode to V1.
Major changes:

1. Since we only implement the ngram spec decode, we did not add another scheduler for running the drafting method. We always check if we need to do ngram lookup before calling the scheduler.
2. Add a new field _spec_token_ids in Request to track speculated tokens.
3. Changes to model_runner:
   3.1 Change the _prepare_input to also return the logits of speculated tokens.
   3.2 Change the _prepare_input to add speculated tokens as input tokens.
   3.3 Change the execute_model to generate multiple tokens per call. Concretely, it will add more than one tokens to input_batch and req_state.
4. We only perform spec decode for requests in the running queue.
5. We only support greedy decoding for now.

What is missing

• Add tests.
• Stop checking for spec decode, where mutiple tokens are generated in a single step.
• For the ngram lookup logic, currently I just append dummy tokens directly instead of performing the lookup. We can move v0's lookup logic here.
• Check the correctness of this PR with prefix caching. <-- Currently no guarantee.
• Check the correctness of this PR with chunked prefill. <-- Should work. The behavior is: we will not do spec decode for the first chunk, but we will perform spec decode for the 2nd, 3rd chunks and decoding tokens.
• Scheduling guarantee.
• Style.

Tasks out of the scope of this PR

1. Optimize the performance of ngram lookup.
2. Support non-greedy decoding.
3. Add other spec decode methods.

Minor: There is a minimal example/test in tests/v1/e2e/test_basic_specdecode.py. You can check it for the current use and check correctness with pytest -s tests/v1/e2e/test_basic_specdecode.py.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LiuXiaoxuanPKU.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork