[Core] microbatch, running batches in parallel #6550
Comments
graciegoheen
added
content
|
once documented, link to new section in line 35. see: #6544 (comment). specifically, this line here where 'concurrently' needs to link out to the new section: This is a powerful abstraction that makes it possible for dbt to run batches separately, concurrently, and retry them independently. |
|
Hiya @mirnawong1 I've assigned this one to myself. I'll pick this up. Kind Regards |
|
Hiya @graciegoheen I'm picking this up for @mirnawong1 Kind Regards |
|
Hey @nataliefiann, you should work with @mirnawong1 and @QMalcolm on this! Grace is OOO |
|
Thanks @runleonarun Kind Regards |
|
Hiya @QMalcolm I'm going to start drafting my PR on this but whilst I do so, I have a few questions if you wouldn't mind shedding some light please:
Is this (running in parallel) only supported for models? And within models, is it only supported in project.yml file, properties.yml and sql file (concurrent batches) config? Is there a limit to how many models can be run in parallel, does it depend on factors such a batch size, event time etc? Also, with regards to the adapters that support the microbatches running in parallel feature - are those the adapters mentioned here under the microbatch section? Kind Regards |
## What are you changing in this pull request and why? I have created this PR following this Git issue: #6550 for Parallel batch execution ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." - [ ] The content in this PR requires a dbt release note, so I added one to the [release notes page](https://docs.getdbt.com/docs/dbt-versions/dbt-cloud-release-notes). <!-- PRE-RELEASE VERSION OF dbt (if so, uncomment): - [ ] Add a note to the prerelease version [Migration Guide](https://github.com/dbt-labs/docs.getdbt.com/tree/current/website/docs/docs/dbt-versions/core-upgrade) --> <!-- ADDING OR REMOVING PAGES (if so, uncomment): - [ ] Add/remove page in `website/sidebars.js` - [ ] Provide a unique filename for new pages - [ ] Add an entry for deleted pages in `website/vercel.json` - [ ] Run link testing locally with `npm run build` to update the links that point to deleted pages --> <!-- vercel-deployment-preview --> --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-nfiann-rbip-dbt-labs.vercel.app/docs/build/incremental-microbatch - https://docs-getdbt-com-git-nfiann-rbip-dbt-labs.vercel.app/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9 - https://docs-getdbt-com-git-nfiann-rbip-dbt-labs.vercel.app/reference/resource-properties/concurrent_batches <!-- end-vercel-deployment-preview --> --------- Co-authored-by: Mirna Wong <[email protected]> Co-authored-by: Quigley Malcolm <[email protected]> Co-authored-by: Leona B. Campbell <[email protected]>
|
closed by #6589 |
Link to the page(s) on docs.getdbt.com requiring updates
Tell us more about this update
concurrent_batches(default based on usage of {{ this }} and whether or not adapter supports)Core PR here - dbt-labs/dbt-core#10958
Adapter PRs to come
Reviewers/Stakeholders/SMEs
Grace, Doug, Quigley, Michelle
Related GitHub issues
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: