Understand the .NET BroadcastQueue

[dsgrieve]

No description provided.

[dsgrieve]

The intent of BroadcastQueue (notes from Teams chat with Chris Rickman):

Functionally, the state of each channel needs to be "in-sync" at the point an agent based on said channel is invoked. This becomes critical when ChatCompletionAgent and OpenAIAssistantAgent are participating in the same AgentChat.
If they don't have the most up-to-date message the response won't have fidelity.

Syncing a ChatHistoryChannel is virtually instantaneous...for an assistant-agent, its based on an REST API call (to store the messages on the thread).

Syncing OpenAIAssistantChannel could therefore have non-trivial latency.

We don't want to block the chat on each turn to sync every channel.
Each ChatCompletionAgent (by default) does share the same history since they utiliize the same channel (based on the channel-key contract).

An assistant requires that all messages be saved to the thread, which is managed remotely by the API

Multiple assistants will share the same thread-id / channel (by default) [I'll come back to qualify "default" in a minute)

Ultimately, if an assistant-agent isn't taking a turn...it doesn't make sense to block on synchronization (just slow down the entire experience)

The BroadcastQueue is responsible for taking a request to broadcast the most recent messages to all channels, without blocking the caller. Blocking only occurs when a channel is being used whose queue is not empty.

I specifically didn't want to manage a background thread looping to manage the queue...so it's all driven / triggered by adding to the queue.

I also took care to minimize as much as possible how much locking was utilized (since that can also impact performance)

In terms of the "default" qualification - it is possible for different ChatCompletionAgents to be managed in separate channels based on the presence / configuration of the HistoryReducer...this is because the reducer alters the channel specific history.

Example: Two chat-completion agents, one with a reducer and one without will each use a different channel so that their channel history is managed as each expects. Any chat-completion agent using a reducer with equivalent configuration will consolidate to the same channel.

For an assistant-agent, it is possible for the caller to create one that targets one endpoint and another that targets a different endpoint. This might be an assistant that targets OpenAI and one that targets Azure...or maybe they target different azure regions or subscriptions.

In these cases, an assistant wouldn't be able to share the same thread-id. This necessarily requires a different channel based on the assistant configuration

So, in the "vanilla" case for mixing chat-completion and assistant agents, one would expect two channels...but this isn't necessarily a given for all usage cases.