Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: Server mode - missing tool call id #4420

Open
1 of 3 tasks
SauceChord opened this issue Jan 8, 2025 · 0 comments
Open
1 of 3 tasks

bug: Server mode - missing tool call id #4420

SauceChord opened this issue Jan 8, 2025 · 0 comments
Assignees
Labels
category: local API server Server page UI UX type: bug Something isn't working

Comments

@SauceChord
Copy link

SauceChord commented Jan 8, 2025

Jan version

0.5.13

Describe the Bug

Jan Server does not include a tool call id for called tools.
This causes an issue with using langchain/langgraph which requires a tool call id.
Included is a python script that tests OpenAI and Jan server tool calls (using qwen2.5-coder-7b-instruct model for Jan)

Expected: tool call id for tool calls.

Steps to Reproduce

  • Install Jan
  • Download Qwen2.5 Coder 7B Instruct Q4 from the Hub
  • Activate Qwen2.5 Coder 7B Instruct Q4
  • Start Local API Server
    • Note: It says it's an OpenAI compatible local HTTP server
  • Install python
  • pip install openai to install library dependencies
  • Set your OPENAI_API_KEY environment variable to your OpenAI Api Key
  • Add code to test.py
from openai import OpenAI

def main():
    jan = OpenAI(base_url="http://localhost:1337/v1")
    oai = OpenAI()
    TEST(oai, "gpt-4o-mini")
    TEST(jan, "qwen2.5-coder-7b-instruct")

def TEST(client, m):
    try:
        history = []
        history.append({"role": "system", "content": "You are a helpful assistant."})
        history.append({"role": "user", "content": "Add 1 and 2"})
        response = client.chat.completions.create(
            model=m,
            messages=history,
            tools=[
                schema_add(),
            ],
            tool_choice="auto"
        )
        for tool in response.choices[0].message.tool_calls:
            if tool.id is None:
                raise Exception(f"{m}: tool call id is None for {tool.function.name}")        
        print(f"Pass: {m}")
    except Exception as e:
        print(f"Fail: {e}")

# A function that we can use as a tool
def add(a: float, b: float) -> float:
    print(f"{a} + {b} = {a + b}")
    return a + b

# A schema that lets the LLM know that we can use the add function as a tool
def schema_add() -> dict:
    return {
            "type": "function",
            "function": {
                "name": "add",
                "description": "Add two numbers",
                "parameters": {
                    "type": "object", 
                    "properties": {
                        "a": {"type": "number"},
                        "b": {"type": "number"}
                    },
                    "required": ["a", "b"]
                },
            },
        }

if __name__ == "__main__":
    main()
  • python test.py

Screenshots / Logs

Pass: gpt-4o-mini
Fail: qwen2.5-coder-7b-instruct: tool call id is None for add

What is your OS?

  • MacOS
  • Windows
  • Linux
@SauceChord SauceChord added the type: bug Something isn't working label Jan 8, 2025
@github-project-automation github-project-automation bot moved this to Investigating in Jan & Cortex Jan 8, 2025
@imtuyethan imtuyethan added the category: local API server Server page UI UX label Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: local API server Server page UI UX type: bug Something isn't working
Projects
Status: Investigating
Development

No branches or pull requests

3 participants