Skip to content

Commit

Permalink
Merge branch 'current' into patch-1
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Dec 10, 2024
2 parents 6899619 + 50dcd1d commit a1de14b
Show file tree
Hide file tree
Showing 22 changed files with 62 additions and 36 deletions.
1 change: 1 addition & 0 deletions website/dbt-versions.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ exports.versions = [
},
{
version: "1.9",
EOLDate: "2025-12-08",
},
{
version: "1.8",
Expand Down
5 changes: 4 additions & 1 deletion website/docs/docs/build/conversion-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ Note that we use the double colon (::) to indicate whether a parameter is nested
| `base_measure:name` | The base conversion event measure. | Required | String |
| `base_measure:fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | String |
| `base_measure:join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | Boolean |
| `base_measure:filter` | Optional `filter` used to apply to the base measure. | Optional | String |
| `conversion_measure` | A list of conversion measure inputs. | Required | Dict |
| `conversion_measure:name` | The base conversion event measure.| Required | String |
| `conversion_measure:fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | String |
Expand Down Expand Up @@ -61,6 +62,7 @@ metrics:
name: The name of the measure # Required
fill_nulls_with: Set the value in your metric definition instead of null (such as zero) # Optional
join_to_timespine: true/false # Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. # Optional
filter: The filter used to apply to the base measure. # Optional
conversion_measure:
name: The name of the measure # Required
fill_nulls_with: Set the value in your metric definition instead of null (such as zero) # Optional
Expand Down Expand Up @@ -105,12 +107,13 @@ Next, define a conversion metric as follows:
- name: visit_to_buy_conversion_rate_7d
description: "Conversion rate from visiting to transaction in 7 days"
type: conversion
label: Visit to Buy Conversion Rate (7-day window)
label: Visit to buy conversion rate (7-day window)
type_params:
conversion_type_params:
base_measure:
name: visits
fill_nulls_with: 0
filter: {{ Dimension('visits__referrer_id') }} = 'facebook'
conversion_measure:
name: sellers
entity: user
Expand Down
11 changes: 5 additions & 6 deletions website/docs/docs/build/incremental-microbatch.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,11 +184,10 @@ Several configurations are relevant to microbatch models, and some are required:
| Config | Description | Default | Type | Required |
|----------|---------------|---------|------|---------|
| [`event_time`](/reference/resource-configs/event-time) | The column indicating "at what time did the row occur." Required for your microbatch model and any direct parents that should be filtered. | N/A | Column | Required |
| `begin` | The "beginning of time" for the microbatch model. This is the starting point for any initial or full-refresh builds. For example, a daily-grain microbatch model run on `2024-10-01` with `begin = '2023-10-01` will process 366 batches (it's a leap year!) plus the batch for "today." | N/A | Date | Required |
| `batch_size` | The granularity of your batches. Supported values are `hour`, `day`, `month`, and `year` | N/A | String | Required |
| `lookback` | Process X batches prior to the latest bookmark to capture late-arriving records. | `1` | Integer | Optional |
| `concurrent_batches` | An override for whether batches run concurrently (at the same time) or sequentially (one after the other). | `None` | Boolean | Optional |

| [`begin`](/reference/resource-configs/begin) | The "beginning of time" for the microbatch model. This is the starting point for any initial or full-refresh builds. For example, a daily-grain microbatch model run on `2024-10-01` with `begin = '2023-10-01` will process 366 batches (it's a leap year!) plus the batch for "today." | N/A | Date | Required |
| [`batch_size`](/reference/resource-configs/batch-size) | The granularity of your batches. Supported values are `hour`, `day`, `month`, and `year` | N/A | String | Required |
| [`lookback`](/reference/resource-configs/lookback) | Process X batches prior to the latest bookmark to capture late-arriving records. | `1` | Integer | Optional |
| [`concurrent_batches`](/reference/resource-properties/concurrent_batches) | An override for whether batches run concurrently (at the same time) or sequentially (one after the other). | `None` | Boolean | Optional |

<Lightbox src="/img/docs/building-a-dbt-project/microbatch/event_time.png" title="The event_time column configures the real-world time of this record"/>

Expand Down Expand Up @@ -290,7 +289,7 @@ The microbatch strategy offers the benefit of updating a model in smaller, more

Parallel batch execution means that multiple batches are processed at the same time, instead of one after the other (sequentially) for faster processing of your microbatch models.

dbt automatically detects whether a batch can be run in parallel in most cases, which means you don’t need to configure this setting. However, the `concurrent_batches` config is available as an override (not a gate), allowing you to specify whether batches should or shouldn’t be run in parallel in specific cases.
dbt automatically detects whether a batch can be run in parallel in most cases, which means you don’t need to configure this setting. However, the [`concurrent_batches` config](/reference/resource-properties/concurrent_batches) is available as an override (not a gate), allowing you to specify whether batches should or shouldn’t be run in parallel in specific cases.

For example, if you have a microbatch model with 12 batches, you can execute those batches to run in parallel. Specifically they'll run in parallel limited by the number of [available threads](/docs/running-a-dbt-project/using-threads).

Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/build/incremental-strategy.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Click the name of the adapter in the below table for more information about supp
| [dbt-databricks](/reference/resource-configs/databricks-configs#incremental-models) ||| |||
| [dbt-snowflake](/reference/resource-configs/snowflake-configs#merge-behavior-incremental-models) |||| ||
| [dbt-trino](/reference/resource-configs/trino-configs#incremental) |||| | |
| [dbt-fabric](/reference/resource-configs/fabric-configs#incremental) || || | |
| [dbt-fabric](/reference/resource-configs/fabric-configs#incremental) || || | |
| [dbt-athena](/reference/resource-configs/athena-configs#incremental-models) ||| || |

### Configuring incremental strategy
Expand Down
8 changes: 7 additions & 1 deletion website/docs/docs/build/simple.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ Note that we use the double colon (::) to indicate whether a parameter is nested
| `type_params` | The type parameters of the metric. | Required | Dict |
| `measure` | A list of measure inputs. | Required | List |
| `measure:name` | The measure you're referencing. | Required | String |
| `measure:alias` | Optional [`alias`](/reference/resource-configs/alias) to rename the measure. | Optional | String |
| `measure:filter` | Optional `filter` applied to the measure. | Optional | String |
| `measure:fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | String |
| `measure:join_to_timespine` | Indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | Boolean |

Expand All @@ -38,6 +40,8 @@ metrics:
type_params: # Required
measure:
name: The name of your measure # Required
alias: The alias applied to the measure. # Optional
filter: The filter applied to the measure. # Optional
fill_nulls_with: Set value instead of null (such as zero) # Optional
join_to_timespine: true/false # Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. # Optional

Expand Down Expand Up @@ -65,9 +69,11 @@ If you've already defined the measure using the `create_metric: true` parameter,
name: customers # The measure you are creating a proxy of.
fill_nulls_with: 0
join_to_timespine: true
alias: customer_count
filter: {{ Dimension('customer__customer_total') }} >= 20
- name: large_orders
description: "Order with order values over 20."
type: SIMPLE
type: simple
label: Large orders
type_params:
measure:
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/build/unit-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Starting in dbt Core v1.8, we have introduced an additional type of test to dbt
- We currently _don't_ support unit testing models that use recursive SQL.
- If your model has multiple versions, by default the unit test will run on *all* versions of your model. Read [unit testing versioned models](/reference/resource-properties/unit-testing-versions) for more information.
- Unit tests must be defined in a YML file in your [`models/` directory](/reference/project-configs/model-paths).
- Table names must be [aliased](/docs/build/custom-aliases) in order to unit test `join` logic.
- Table names must be aliased in order to unit test `join` logic.
- Include all [`ref`](/reference/dbt-jinja-functions/ref) or [`source`](/reference/dbt-jinja-functions/source) model references in the unit test configuration as `input`s to avoid "node not found" errors during compilation.

#### Adapter-specific caveats
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,14 @@ To set up [personal access tokens (PATs)](https://help.tableau.com/current/serve
<Lightbox src="/img/docs/cloud-integrations/auto-exposures/cloud-integration-details.jpg" title="Enter the details for the exposure connection."/>
4. Select the collections you want to include for auto exposures.

dbt Cloud automatically imports and syncs any workbook within the selected collections. New additions to the collections will be added to the lineage in dbt Cloud during the next automatic sync (usually once per day).
<Lightbox src="/img/docs/cloud-integrations/auto-exposures/cloud-select-collections.jpg" title="Select the collections you want to include for auto exposures."/>

:::info
dbt Cloud automatically imports and syncs any workbook within the selected collections. New additions to the collections will be added to the lineage in dbt Cloud during the next sync (automatically once per day).

dbt Cloud immediately starts a sync when you update the selected collections list, capturing new workbooks and removing irrelevant ones.
:::

5. Click **Save**.

dbt Cloud imports everything in the collection(s) and you can continue to view them in Explorer. For more information on how to view and use auto-exposures, refer to [View auto-exposures from dbt Explorer](/docs/collaborate/auto-exposures) page.
Expand Down
7 changes: 7 additions & 0 deletions website/docs/docs/cloud/cloud-cli-installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,3 +319,10 @@ This alias will allow you to use the <code>dbt-cloud</code> command to invoke th
If you've ran a dbt command and receive a <code>Session occupied</code> error, you can reattach to your existing session with <code>dbt reattach</code> and then press <code>Control-C</code> and choose to cancel the invocation.
</DetailsToggle>
<DetailsToggle alt_header="Why am I receiving a `Stuck session` error when trying to run a new command?">
The Cloud CLI allows only one command that writes to the data warehouse at a time. If you attempt to run multiple write commands simultaneously (for example, `dbt run` and `dbt build`), you will encounter a `stuck session` error. To resolve this, cancel the specific invocation by passing its ID to the cancel command. For more information, refer to [parallel execution](/reference/dbt-commands#parallel-execution).
</DetailsToggle>
2 changes: 1 addition & 1 deletion website/docs/docs/cloud/secure/databricks-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ The following steps will walk you through the setup of a Databricks AWS PrivateL
1. Once dbt Cloud support has notified you that setup is complete, [register the VPC endpoint in Databricks](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html#step-3-register-privatelink-objects-and-attach-them-to-a-workspace) and attach it to the workspace:
- [Register your VPC endpoint](https://docs.databricks.com/en/security/network/classic/vpc-endpoints.html) &mdash; Register the VPC endpoint using the VPC endpoint ID provided by dbt Support.
- [Create a Private Access Settings object](https://docs.databricks.com/en/security/network/classic/private-access-settings.html) &mdash; Create a Private Access Settings (PAS) object with your desired public access settings, and setting Private Access Level to **Endpoint**. Choose the registered endpoint created in the previous step.
- [Create or update your workspace](https://docs.databricks.com/en/security/network/classic/privatelink.html#step-3d-create-or-update-the-workspace-front-end-back-end-or-both) &mdash; Create a workspace, or update your an existing workspace. Under **Advanced configurations → Private Link** choose the private access settings object created in the previous step.
- [Create or update your workspace](https://docs.databricks.com/en/security/network/classic/privatelink.html#step-3d-create-or-update-the-workspace-front-end-back-end-or-both) &mdash; Create a workspace, or update an existing workspace. Under **Advanced configurations → Private Link** choose the private access settings object created in the previous step.
:::warning
If using an existing Databricks workspace, all workloads running in the workspace need to be stopped to enable Private Link. Workloads also can't be started for another 20 minutes after making changes. From the [Databricks documentation](https://docs.databricks.com/en/security/network/classic/privatelink.html#step-3d-create-or-update-the-workspace-front-end-back-end-or-both):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ Starting in Core 1.9, you can use the new [microbatch strategy](/docs/build/incr
- Simplified query design: Write your model query for a single batch of data. dbt will use your `event_time``lookback`, and `batch_size` configurations to automatically generate the necessary filters for you, making the process more streamlined and reducing the need for you to manage these details.
- Independent batch processing: dbt automatically breaks down the data to load into smaller batches based on the specified `batch_size` and processes each batch independently, improving efficiency and reducing the risk of query timeouts. If some of your batches fail, you can use `dbt retry` to load only the failed batches.
- Targeted reprocessing: To load a *specific* batch or batches, you can use the CLI arguments `--event-time-start` and `--event-time-end`.
- [Automatic parallel batch execution](/docs/build/incremental-microbatch#parallel-batch-execution): Process multiple batches at the same time, instead of one after the other (sequentially) for faster processing of your microbatch models. dbt intelligently auto-detects if your batches can run in parallel, while also allowing you to manually override parallel execution with the `concurrent_batches` config.
- [Automatic parallel batch execution](/docs/build/incremental-microbatch#parallel-batch-execution): Process multiple batches at the same time, instead of one after the other (sequentially) for faster processing of your microbatch models. dbt intelligently auto-detects if your batches can run in parallel, while also allowing you to manually override parallel execution with the [`concurrent_batches` config](/reference/resource-properties/concurrent_batches).


Currently microbatch is supported on these adapters with more to come:
Expand Down
2 changes: 2 additions & 0 deletions website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,11 @@ Release notes are grouped by month for both multi-tenant and virtual private clo

## December 2024

- **New**: [Model notifications](/docs/deploy/model-notifications) are now generally available in dbt Cloud. These notifications alert model owners through email about any issues encountered by models and tests as soon as they occur while running a job.
- **New**: You can now use your [Azure OpenAI key](/docs/cloud/account-integrations?ai-integration=azure#ai-integrations) (available in beta) to use dbt Cloud features like [dbt Copilot](/docs/cloud/dbt-copilot) and [Ask dbt](/docs/cloud-integrations/snowflake-native-app) . Additionally, you can use your own [OpenAI API key](/docs/cloud/account-integrations?ai-integration=openai#ai-integrations) or use [dbt Labs-managed OpenAI](/docs/cloud/account-integrations?ai-integration=dbtlabs#ai-integrations) key. Refer to [AI integrations](/docs/cloud/account-integrations#ai-integrations) for more information.
- **New**: The [`hard_deletes`](/reference/resource-configs/hard-deletes) config gives you more control on how to handle deleted rows from the source. Supported options are `ignore` (default), `invalidate` (replaces the legacy `invalidate_hard_deletes=true`), and `new_record`. Note that `new_record` will create a new metadata column in the snapshot table.


## November 2024
- **Enhancement**: Trust signal icons in dbt Explorer are now available for Exposures, providing a quick view of data health while browsing resources. To view trust signal icons, go to dbt Explorer and click **Exposures** under the **Resource** tab. Refer to [Trust signal for resources](/docs/collaborate/explore-projects#trust-signals-for-resources) for more info.
- **Bug**: Identified and fixed an error with Semantic Layer queries that take longer than 10 minutes to complete.
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/deploy/deploy-environments.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ To create a new dbt Cloud deployment environment, navigate to **Deploy** -> **En

In dbt Cloud, each project can have one designated deployment environment, which serves as its production environment. This production environment is _essential_ for using features like dbt Explorer and cross-project references. It acts as the source of truth for the project's production state in dbt Cloud.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/prod-settings.jpg" width="70%" title="Set your production environment as the default environment in your Environment Settings"/>
<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/prod-settings-1.png" width="100%" title="Set your production environment as the default environment in your Environment Settings"/>

### Semantic Layer

Expand Down
Loading

0 comments on commit a1de14b

Please sign in to comment.