Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable preparing nested OutputSchemas for serialization #1357

Merged
merged 4 commits into from
Nov 1, 2023

Conversation

dbogunowicz
Copy link
Contributor

@dbogunowicz dbogunowicz commented Oct 26, 2023

Currently, the prep_outputs_for_serialization function assumes that pipeline_outputs cannot be arbitrarily nested (allows up to one level of nesting). This is why, for more complex BaseModels returned by the python (e.g. TextGenerationOutput), the function will not properly convert nested numpy arrays to lists for serialization.

My diff makes sure that any pipeline_output: BaseModel that contains an arbitrary number and nesting depth of any fields that are either a BaseModel, list of numpy.ndarray is supported.

As a result, we can now serialize logits of the LLM output, that previously was not properly converted from numpy array to a list.

import requests

model_path = "hf:mgoin/TinyStories-1M-deepsparse"
prompt =  ["name one former president of the USA"]
server_address = "http://0.0.0.0:5543/v2/models/gen/infer"

payload = {"prompt": prompt, "output_scores": True, "include_prompt_logits": True}
response = requests.post(server_address, json=payload)
response = response.json()

num_tokens_prompt_and_generated = len(response["generations"][0]["score"])

payload = {"prompt": prompt, "output_scores": True}
response = requests.post(server_address, json=payload)
response = response.json()

num_tokens_generated = len(response["generations"][0]["score"])
num_tokens_prompt = num_tokens_prompt_and_generated - num_tokens_generated
print(f"Number of prompt tokens: {num_tokens_prompt}")
print(f"Number of generated tokens: {num_tokens_generated}")

returns

Number of prompt tokens: 6
Number of generated tokens: 3

@dbogunowicz dbogunowicz changed the title [WiP] Properly unserialize nested OutputSchemas Enable preparing nested OutputSchemas for serialization Oct 26, 2023
@dbogunowicz dbogunowicz requested a review from bfineran October 26, 2023 09:29
bfineran
bfineran previously approved these changes Oct 30, 2023
src/deepsparse/server/helpers.py Show resolved Hide resolved
@dbogunowicz dbogunowicz merged commit 8dbb4ad into main Nov 1, 2023
13 checks passed
@dbogunowicz dbogunowicz deleted the feature/damian/nested_unserialisation branch November 1, 2023 13:36
dbogunowicz added a commit that referenced this pull request Nov 2, 2023
* initial commit to unblock derek

* ready for review

* add unit test
bfineran pushed a commit that referenced this pull request Nov 2, 2023
* initial commit to unblock derek

* ready for review

* add unit test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants