Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Child stream that runs after all records synced by parent stream #2809

Open
ReubenFrankel opened this issue Dec 11, 2024 · 0 comments
Open
Assignees
Labels
kind/Feature New feature or request valuestream/SDK

Comments

@ReubenFrankel
Copy link
Contributor

Feature scope

Taps (catalog, state, tests, etc.)

Description

I'm imagining something like

class ParentStream(Stream):

    def get_deferred_child_context(self, records, context)
        return {"record_ids": [record["id"] for record in records]}


class DeferredChildStream(Stream):

    parent_stream_type = ParentStream

    def get_records(self, context):
        for record_id in context["record_ids"]:
            ...

This would useful to support APIs that offer supplemental "detail" endpoints (e,g, GET /resource/details) that accept one or more resource identifiers in bulk (request body data), rather than one at a time (e.g. GET /resources/{id}/details) as is the case in a traditional parent-child stream relationship. I don't really know enough about context or state here to fully understand the implications of something like this, but it feels possible.

@github-project-automation github-project-automation bot moved this to To Discuss in Office Hours Dec 11, 2024
@edgarrmondragon edgarrmondragon moved this from To Discuss to Up Next in Office Hours Dec 11, 2024
@edgarrmondragon edgarrmondragon moved this from Up Next to To Discuss in Office Hours Dec 11, 2024
@edgarrmondragon edgarrmondragon moved this from To Discuss to Up Next in Office Hours Dec 14, 2024
@edgarrmondragon edgarrmondragon moved this from Up Next to To Discuss in Office Hours Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/Feature New feature or request valuestream/SDK
Projects
Status: To Discuss
Development

No branches or pull requests

2 participants