Support for llama3 8b instruct

[wasauce]

Is your feature request related to a problem? Please describe.
I am trying to use instructor with one of the llama 3 8b instruct models.

I have tried accessing togetherAI and https://openrouter.ai/ via instructor -- pointing at the llama 3 8b instruct models.

Together.ai doesn't support JSON mode on llama 38b instruct - so the examples I found found for accessing together.ai don't work. Is there a work around.

Describe the solution you'd like
To be able to use instructor with llama3 8b instruct.

[Zasha01]

Hey,

for me it worked this way:

from openai import OpenAI
from pydantic import BaseModel, Field

from typing import List

import instructor


class Character(BaseModel):

    name: str
    age: int
    fact: List[str] = Field(..., description="5 facts about the character")


# enables `response_model` in create call
client = instructor.from_openai(
    OpenAI(
        base_url="http://localhost:11434/v1",

        api_key="ollama",  # required, but unused
    ),
    #mode=instructor.Mode.JSON,
)

resp = client.chat.completions.create_with_completion(
    model="llama3",
    messages=[
        {
            "role": "user",
            "content": "Tell me about Harry Potter",
        }
    ],
    response_model=Character,
)

I got the code from here: https://python.useinstructor.com/examples/ollama/#ollama

[wasauce]

Hey,

for me it worked this way:

@Zasha01 thank you! If I am understanding correctly -- you are running this via ollama correct? Not via one of the cloud providers right? Sadly my local machine isn't powerful enough to do the processing volume I need in a reasonable amount of time. Maybe I can try though on an e.g. modal or other provider where I set it up myself.

[jxnl]

you can try anyscale or together ai or even grok!

[wasauce]

@jxnl thanks for this reply. I must be doing something wrong.

I have tried with Together.ai and things work when I use one of the 3 JSON mode supported models: https://docs.together.ai/docs/json-mode but if I try to use llama-3 - it does not work.

Explicitly:
raise self._make_status_error_from_response(err.response) from None openai.BadRequestError: Error code: 400 - {'error': {'message': 'meta-llama/Llama-3-8b-chat-hf is not supported for JSON mode/function calling', 'type': 'invalid_request_error', 'param': None, 'code': 'constraints_model'}}

Can you please point me in the right direction?

I've included sample code below (that requires a together AI key).

import os 
import openai
import instructor

from pydantic import BaseModel


# Define your desired output structure
class UserInfo(BaseModel):
    name: str
    age: int


os.environ["TOGETHER_API_KEY"] = "SET_YOUR_KEY_HERE"


together_client = openai.OpenAI(
    base_url="https://api.together.xyz/v1",
    api_key=os.environ["TOGETHER_API_KEY"],
)
instructor_client = instructor.from_openai(together_client)

user_info = instructor_client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.1", # "meta-llama/Llama-3-8b-chat-hf" - llama does not work.
    response_model=UserInfo,
    messages=[
        {"role": "user", "content": "John Doe is 30 years old."},
    ],
)

print(user_info)


user_info = instructor_client.chat.completions.create(
    model="meta-llama/Llama-3-8b-chat-hf",
    response_model=UserInfo,
    messages=[
        {"role": "user", "content": "John Doe is 30 years old."},
    ],
)

print(user_info)

[ivanleomk]

@wasauce I think the easiest way is to host it via a openai compatible server. Have you tried llama-cpp-python? I think getting it to work on modal should be relatively simple ( https://github.com/abetlen/llama-cpp-python )