SYSTEM_PROMPT Not Behaving as Expected with Larger Queries

[AashifAhamed]

Issue Title: SYSTEM_PROMPT Not Behaving as Expected with Larger Queries

Issue Description:

The custom SYSTEM_PROMPT designed to ensure the AI Voice Agent communicates exclusively in Japanese while maintaining a polite and concise conversational tone is not functioning as expected. Despite accurately implementing similar functionality in a simpler prompt, the behavior diverges when incorporating a more detailed and complex prompt.

const ULTRAVOX_CALL_CONFIG = {
    systemPrompt: SYSTEM_PROMPT,
    model: 'fixie-ai/ultravox',
    voice: 'Morioki-Japanese',
    temperature: 0.3,
    firstSpeaker: 'FIRST_SPEAKER_USER',
    medium: { "twilio": {} },
    "selectedTools":[
        { "toolId": "56294126-5a7d-4948-b67d-3b7e13d55ea7" }
    ]
};

Observed Behavior:

• The AI does not adhere to the specified instruction to terminate the call "only when explicitly instructed or when there are no further interactions."
• The agent's language use and conversational tone may deviate from the exclusive Japanese (敬語) context defined in the prompt.
• The desired concise responses (1–3 sentences) are not consistently observed.

Suspected Cause:

The issue may stem from the following:

1. Prompt Length: The extended prompt contains more details, which might lead to parsing or processing challenges within the Ultravox API or the underlying AI model.
2. Complexity of Instructions: Multiple directives (e.g., exclusive language use, short responses, and a specific tone) might be causing priority conflicts or diluting the primary call termination condition.
3. Tool or API Limitations: The Ultravox system may not effectively parse or execute larger and more nuanced prompts, leading to unintended behavior.

Steps to Reproduce:

1. Use the detailed SYSTEM_PROMPT provided in the integration script.
2. Initiate a call through the Ultravox API with the configured prompt.
3. Observe the agent's behavior, focusing on language use, tone, response length, and call termination criteria.

Expected Behavior:

• The AI agent communicates exclusively in Japanese (敬語) throughout the call.
• Responses are concise (1–3 sentences), maintaining a polite yet approachable tone.
• The call ends only when explicitly instructed or when interactions naturally cease.

Actual Behavior:

• The AI does not consistently meet the exclusivity or conciseness criteria.
• Call termination may occur prematurely or contrary to instructions.

Suggested Resolution:

1. Simplify the Prompt: Divide complex instructions into smaller, more focused prompts to reduce parsing complexity.
2. Test Incrementally: Validate behavior with a simplified version of the prompt and gradually add details to identify the breaking point.
3. Consult Ultravox Documentation: Verify prompt length and complexity limitations within the API guidelines.
4. Enable Debugging: Log detailed API responses and execution flow to identify parsing or configuration issues.
5. Optimize API Configuration: Adjust parameters like temperature or tool settings to better handle complex instructions.

Priority: High

This issue directly impacts the functionality and reliability of the AI Voice Agent, particularly in adhering to language and termination conditions critical for user experience.