Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(ingest/tableau): add test for extract_project_hierarchy scenario #12079

Merged
merged 17 commits into from
Dec 20, 2024

Conversation

sid-acryl
Copy link
Contributor

Consider a scenario where a Tableau site contains four projects: A, B, C, and D, with the following hierarchical relationship:

    - A
      - B (Child of A)
      - C (Child of A)
    - D

    In this setup:

    - `project_pattern` is configured with `allow: ["A"]` and `deny: ["B"]`.
    - `extract_project_hierarchy` is set to `True`.

The goal is to extract assets from project A and its children while explicitly denying the child project B.

If we rely solely on the project_pattern.allowed() method, project C's assets will not be ingested.
This happens because project C is not explicitly included in the allow list, nor is it part of the deny list.
However, since extract_project_hierarchy is enabled, project C should ideally be included in the ingestion process unless explicitly denied.

To address this, the function explicitly checks the deny regex to ensure that project C’s assets are ingested if it is not specifically denied in the deny list. This approach ensures that the hierarchy is respected while adhering to the configured allow/deny rules.

@github-actions github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Dec 10, 2024
@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Dec 10, 2024
@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Dec 10, 2024
@datahub-cyborg datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Dec 11, 2024
@hsheth2 hsheth2 changed the title test(ingestion/tableau): add test for extract_project_hierarchy scenario test(ingest/tableau): add test for extract_project_hierarchy scenario Dec 11, 2024
@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Dec 11, 2024
@datahub-cyborg datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Dec 13, 2024
@hsheth2
Copy link
Collaborator

hsheth2 commented Dec 17, 2024

@sid-acryl there's conflicts on this one - please take a look

@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Dec 17, 2024
@sid-acryl
Copy link
Contributor Author

@sid-acryl there's conflicts on this one - please take a look

done

@datahub-cyborg datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Dec 18, 2024
@datahub-cyborg datahub-cyborg bot added pending-submitter-merge and removed needs-review Label for PRs that need review from a maintainer. labels Dec 18, 2024
@hsheth2
Copy link
Collaborator

hsheth2 commented Dec 20, 2024

Dagster plugin failures are unrelated.

@hsheth2 hsheth2 merged commit 3c3d032 into datahub-project:master Dec 20, 2024
72 of 74 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ingestion PR or Issue related to the ingestion of metadata pending-submitter-merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants