Blocking callbacks

[DanielDuranSW]

Could you clarify the future support for registering callbacks on dbt's EventManager? "The current behavior blocks subsequent steps, but this may not be guaranteed in future versions". "Given that the Python interface for events is less mature than the structured logging interface", would it be better to rely on parsing JSON-formatted logs for handling real-time events, or are callbacks still the recommended approach for this use case?

[QMalcolm]

@DanielDuranSW Hi, thank you for the question! I don't envision a future where callbacks go away. Please use callbacks if they are the right use case for you. Whether they remain blocking or not is harder to say. It isn't presently on our roadmap to make callbacks non-blocking anytime soon, but that could change. Additionally, we welcome any community contributions to make callbacks non-blocking.

If your need is to handle real-time events in a non-blocking fashion, then the easiset way is likely to rely on JSON logs since we write JSON logs in real time. That said, one could create a new sub processes from their custom callback which would then allow execution of the dag to continue while the event is being handled by the sub process. If you don't have a need to respect the dbt thread pool, this could be fairly straight forward. If you want to respect the thread pool, it becomes a fair bit more complicated.

[DanielDuranSW]

Hi, thank you for your thoughtful response and for clarifying the current roadmap for callbacks. Personally, I see callbacks as a more natural fit for real-time workflows, especially because they allow for immediate handling of events without the need for additional processing steps like reading and parsing logs.

Additionally, callbacks make it easier to add events and distinguish them across different levels, while avoiding the need for a constant consumer reading logs in real-time. That said, your suggestion of leveraging subprocesses within callbacks for non-blocking execution is very insightful and aligns well with certain use cases. I'll explore it further while keeping in mind the trade-offs with the dbt thread pool. Thanks again for the guidance!