From 687388abae86f6a9cad0fec9326176809f585d37 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 28 Sep 2023 15:49:32 -0400 Subject: [PATCH 01/13] Adding semantic models to list --- .../migration/versions/01-upgrading-to-v1.6.md | 1 + website/docs/reference/commands/list.md | 12 +++++++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/website/docs/guides/migration/versions/01-upgrading-to-v1.6.md b/website/docs/guides/migration/versions/01-upgrading-to-v1.6.md index bdb47bbf2ea..7cdd6df7884 100644 --- a/website/docs/guides/migration/versions/01-upgrading-to-v1.6.md +++ b/website/docs/guides/migration/versions/01-upgrading-to-v1.6.md @@ -90,4 +90,5 @@ More consistency and flexibility around packages. Resources defined in a package - [`dbt debug --connection`](/reference/commands/debug) to test just the data platform connection specified in a profile - [`dbt docs generate --empty-catalog`](/reference/commands/cmd-docs) to skip catalog population while generating docs - [`--defer-state`](/reference/node-selection/defer) enables more-granular control +- [`dbt ls`](/reference/commands/list) adds the Semantic model selection method to allow for `dbt ls -s "semantic_model:*"` and the ability to execute `dbt ls --resource-type semantic_model`. diff --git a/website/docs/reference/commands/list.md b/website/docs/reference/commands/list.md index 6084b3dec70..61e60f91333 100644 --- a/website/docs/reference/commands/list.md +++ b/website/docs/reference/commands/list.md @@ -10,7 +10,7 @@ The `dbt ls` command lists resources in your dbt project. It accepts selector ar ### Usage ``` dbt ls - [--resource-type {model,source,seed,snapshot,metric,test,exposure,analysis,default,all}] + [--resource-type {model,semantic_model,source,seed,snapshot,metric,test,exposure,analysis,default,all}] [--select SELECTION_ARG [SELECTION_ARG ...]] [--models SELECTOR [SELECTOR ...]] [--exclude SELECTOR [SELECTOR ...]] @@ -93,6 +93,16 @@ $ dbt ls --select snowplow.* --output json --output-keys name resource_type desc + + +**Listing Semantic models** + +``` +$ dbt ls -s +semantic_model:number + +``` + + **Listing file paths** ``` From d4992113bfcaa8548e8305b9d4eb393fcabcde8d Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 28 Sep 2023 16:18:00 -0400 Subject: [PATCH 02/13] Adding to "methods" page --- website/docs/reference/node-selection/methods.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/website/docs/reference/node-selection/methods.md b/website/docs/reference/node-selection/methods.md index 2647f3416a3..18009679924 100644 --- a/website/docs/reference/node-selection/methods.md +++ b/website/docs/reference/node-selection/methods.md @@ -352,3 +352,16 @@ dbt list --select version:none # models that are *not* versioned ``` + + + +The `semantic_model` method selects [semantic models](/docs/build/semantic-models). + +```bash + +dbt ls --resource-type semantic_model # lists the semantic model resource type +dbt ls -s +semantic_model:number # lists the quantity of semantic models + +``` + + \ No newline at end of file From 933bbab5a629820900f5505dfb1ef47be44e10f4 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 11 Oct 2023 19:58:23 -0400 Subject: [PATCH 03/13] Built in aliasing for data_type --- .../reference/resource-configs/contract.md | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/website/docs/reference/resource-configs/contract.md b/website/docs/reference/resource-configs/contract.md index f9a5376bc05..40e2cc8b27d 100644 --- a/website/docs/reference/resource-configs/contract.md +++ b/website/docs/reference/resource-configs/contract.md @@ -23,8 +23,31 @@ When the `contract` configuration is enforced, dbt will ensure that your model's This is to ensure that the people querying your model downstream—both inside and outside dbt—have a predictable and consistent set of columns to use in their analyses. Even a subtle change in data type, such as from `boolean` (`true`/`false`) to `integer` (`0`/`1`), could cause queries to fail in surprising ways. + + The `data_type` defined in your YAML file must match a data type your data platform recognizes. dbt does not do any type aliasing itself. If your data platform recognizes both `int` and `integer` as corresponding to the same type, then they will return a match. + + + + +dbt uses built-in type aliasing for the `data_type` defined in your YAML. For example, you can specify `string` in your contract, and on Postgres/Redshift, dbt will convert it to `text`. If dbt doesn't recognize the `data_type` name among its known aliases, it will pass it through as-is. This is enabled by default, but you can opt-out by setting `alias_types` to `false`. + +Example: + +```yml + +models: + - name: my_model + config: + contract: + enforced: true + alias_types: false # true by default + +``` + + + When dbt compares data types, it will not compare granular details such as size, precision, or scale. We don't think you should sweat the difference between `varchar(256)` and `varchar(257)`, because it doesn't really affect the experience of downstream queriers. You can accomplish a more-precise assertion by [writing or using a custom test](/guides/best-practices/writing-custom-generic-tests). Note that you need to specify a varchar size or numeric scale, otherwise dbt relies on default values. For example, if a `numeric` type defaults to a precision of 38 and a scale of 0, then the numeric column stores 0 digits to the right of the decimal (it only stores whole numbers), which might cause it to fail contract enforcement. To avoid this implicit coercion, specify your `data_type` with a nonzero scale, like `numeric(38, 6)`. dbt Core 1.7 and higher provides a warning if you don't specify precision and scale when providing a numeric data type. From 9f8686a45c71409b7ab20eeae6e67c9b70f87ca4 Mon Sep 17 00:00:00 2001 From: Grace Goheen Date: Thu, 12 Oct 2023 10:44:36 -0600 Subject: [PATCH 04/13] updated 1.7 migration guide to call out a behavior change --- .../versions/00-upgrading-to-v1.7.md | 22 ++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md index ba079e6a0fb..8a5659c7f2d 100644 --- a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md +++ b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md @@ -14,11 +14,22 @@ description: New features and changes in dbt Core v1.7 dbt Labs is committed to providing backward compatibility for all versions 1.x, with the exception of any changes explicitly mentioned below. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt-core/issues/new). +### Behavior changes + +1.7 expands the amount of sources folks can configure freshness for. Previously, freshness was limited to sources that had a `loaded_at_field`; but now, freshness can be generated from warehouse metadata tables when available. + +As part of this change, the `loaded_at_field` is **no longer required** in order to generate source freshness. If a source has a `freshness:` block, dbt will attempt to calculate freshness for that source: +- if a `loaded_at_field` is provided, dbt will calculate freshness via a select query (previous behavior) +- if a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new!) + +This is technically a small behavior change comapared to previous versons. In order to _not_ calculate freshness for a source: +- don't add a `freshness:` block +- explicity set `freshness: null` + ## New and changed features and functionality - [`dbt docs generate`](/reference/commands/cmd-docs) now supports `--select` to generate documentation for a subset of your project. Currently available for Snowflake and Postgres only, but other adapters are coming soon. -- [Source freshness](/docs/deploy/source-freshness) can now be generated from warehouse metadata tables, currently snowflake only, but other adapters that have metadata tables are coming soon. If you configure source freshness without a `loaded_at_field`, dbt will try to determine freshness from warehouse metadata tables. -- The nodes dictionary in the `catalog.json` can now be "partial" if `dbt docs generate` is run with a selector. +- [Source freshness](/docs/deploy/source-freshness) can now be generated from warehouse metadata tables, currently Snowflake only, but other adapters that have metadata tables are coming soon. ### MetricFlow enhancements @@ -32,8 +43,9 @@ dbt Labs is committed to providing backward compatibility for all versions 1.x, - The [manifest](/reference/artifacts/manifest-json) schema version has been updated to v11. - The [run_results](/reference/artifacts/run-results-json) schema version has been updated to v5. -- Added [node attributes](/reference/artifacts/run-results-json) related to compilation (`compiled`, `compiled_code`, `relation_name`). - +- There were a handful of specific changes to the [catalog.json](/reference/artifacts/catalog-json): + - Added [node attributes](/reference/artifacts/run-results-json) related to compilation (`compiled`, `compiled_code`, `relation_name`) to the `catalog.json`. + - The nodes dictionary in the `catalog.json` can now be "partial" if `dbt docs generate` is run with a selector. ### Model governance @@ -49,4 +61,4 @@ dbt Core v1.5 introduced model governance which we're continuing to refine. v1. With these quick hits, you can now: - Configure a `delimiter` for a seed file. - Use packages with the same git repo and unique subdirectory. -- Moved the `date_spine` macro from dbt-utils to dbt-core. +- Access the `date_spine` macro directly from dbt-core (moved over from dbt-utils). From 6f78a15d1a51cc40eaff0333903e7f1831434e9d Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:04:51 -0400 Subject: [PATCH 05/13] Apply suggestions from code review --- .../guides/migration/versions/00-upgrading-to-v1.7.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md index 8a5659c7f2d..60218969266 100644 --- a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md +++ b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md @@ -16,11 +16,11 @@ dbt Labs is committed to providing backward compatibility for all versions 1.x, ### Behavior changes -1.7 expands the amount of sources folks can configure freshness for. Previously, freshness was limited to sources that had a `loaded_at_field`; but now, freshness can be generated from warehouse metadata tables when available. +dbt Core v1.7 expands the amount of sources you can configure freshness for. Previously, freshness was limited to sources with a `loaded_at_field`; now, freshness can be generated from warehouse metadata tables when available. -As part of this change, the `loaded_at_field` is **no longer required** in order to generate source freshness. If a source has a `freshness:` block, dbt will attempt to calculate freshness for that source: -- if a `loaded_at_field` is provided, dbt will calculate freshness via a select query (previous behavior) -- if a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new!) +As part of this change, the `loaded_at_field` is no longer required to generate source freshness. If a source has a `freshness:` block, dbt will attempt to calculate freshness for that source: +- If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (previous behavior). +- If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new behavior). This is technically a small behavior change comapared to previous versons. In order to _not_ calculate freshness for a source: - don't add a `freshness:` block From 1d57c34a7bac8c64e8af50eb10778d2a4f1da015 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:07:05 -0400 Subject: [PATCH 06/13] Apply suggestions from code review --- .../docs/guides/migration/versions/00-upgrading-to-v1.7.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md index 60218969266..6a1d828bff4 100644 --- a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md +++ b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md @@ -22,9 +22,9 @@ As part of this change, the `loaded_at_field` is no longer required to generate - If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (previous behavior). - If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new behavior). -This is technically a small behavior change comapared to previous versons. In order to _not_ calculate freshness for a source: -- don't add a `freshness:` block -- explicity set `freshness: null` +This is a relatively small behavior change compared to previous versions. To exclude a source from calculating source freshness: +- Don't add a `freshness:` block. +- Explicitly set `freshness: null` ## New and changed features and functionality From 6237a8e340dc72e56a99f5d9744decafff33c667 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:10:23 -0400 Subject: [PATCH 07/13] Update website/docs/guides/migration/versions/00-upgrading-to-v1.7.md --- website/docs/guides/migration/versions/00-upgrading-to-v1.7.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md index 6a1d828bff4..c92fdaeed4a 100644 --- a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md +++ b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md @@ -43,7 +43,7 @@ This is a relatively small behavior change compared to previous versions. To exc - The [manifest](/reference/artifacts/manifest-json) schema version has been updated to v11. - The [run_results](/reference/artifacts/run-results-json) schema version has been updated to v5. -- There were a handful of specific changes to the [catalog.json](/reference/artifacts/catalog-json): +- There are a few specific changes to the [catalog.json](/reference/artifacts/catalog-json): - Added [node attributes](/reference/artifacts/run-results-json) related to compilation (`compiled`, `compiled_code`, `relation_name`) to the `catalog.json`. - The nodes dictionary in the `catalog.json` can now be "partial" if `dbt docs generate` is run with a selector. From b771380baa23728f276b7e5fa342bf57c2d7ed70 Mon Sep 17 00:00:00 2001 From: Grace Goheen <53586774+graciegoheen@users.noreply.github.com> Date: Thu, 12 Oct 2023 12:20:56 -0600 Subject: [PATCH 08/13] Update website/docs/guides/migration/versions/00-upgrading-to-v1.7.md --- website/docs/guides/migration/versions/00-upgrading-to-v1.7.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md index c92fdaeed4a..a85dd0e69e0 100644 --- a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md +++ b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md @@ -22,7 +22,7 @@ As part of this change, the `loaded_at_field` is no longer required to generate - If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (previous behavior). - If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new behavior). -This is a relatively small behavior change compared to previous versions. To exclude a source from calculating source freshness: +This is a relatively small behavior change, but worth calling out in case you notice that dbt is calculating freshness for _more_ sources than before. To exclude a source from freshness calculations, you have two options: - Don't add a `freshness:` block. - Explicitly set `freshness: null` From 8e097d7a98b978b5440f30a93c5501d2facfb71c Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:22:05 -0400 Subject: [PATCH 09/13] Update website/docs/reference/node-selection/methods.md Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- website/docs/reference/node-selection/methods.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/website/docs/reference/node-selection/methods.md b/website/docs/reference/node-selection/methods.md index 18009679924..b5ffe4b1520 100644 --- a/website/docs/reference/node-selection/methods.md +++ b/website/docs/reference/node-selection/methods.md @@ -353,6 +353,10 @@ dbt list --select version:none # models that are *not* versioned +### The "semantic_model" method + +Supported in v1.6 or newer. + The `semantic_model` method selects [semantic models](/docs/build/semantic-models). From e89dfc8a656690623838d0bc6c7bae4561e42124 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:22:18 -0400 Subject: [PATCH 10/13] Update website/docs/reference/commands/list.md Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- website/docs/reference/commands/list.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/reference/commands/list.md b/website/docs/reference/commands/list.md index 61e60f91333..40e58178ee0 100644 --- a/website/docs/reference/commands/list.md +++ b/website/docs/reference/commands/list.md @@ -98,8 +98,8 @@ $ dbt ls --select snowplow.* --output json --output-keys name resource_type desc **Listing Semantic models** ``` -$ dbt ls -s +semantic_model:number - +# list all resources upstream of your orders semantic model +dbt ls -s +semantic_model:orders ``` From e7a472b76e27f34fc716b4456f543a93491f6957 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:22:57 -0400 Subject: [PATCH 11/13] Update website/docs/reference/node-selection/methods.md Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- website/docs/reference/node-selection/methods.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/website/docs/reference/node-selection/methods.md b/website/docs/reference/node-selection/methods.md index b5ffe4b1520..7801f29aa33 100644 --- a/website/docs/reference/node-selection/methods.md +++ b/website/docs/reference/node-selection/methods.md @@ -362,10 +362,8 @@ Supported in v1.6 or newer. The `semantic_model` method selects [semantic models](/docs/build/semantic-models). ```bash - -dbt ls --resource-type semantic_model # lists the semantic model resource type -dbt ls -s +semantic_model:number # lists the quantity of semantic models - +dbt list --select semantic_model:* # list all semantic models +dbt list --select +semantic_model:orders # list your semantic model named "orders" and all upstream resources ``` \ No newline at end of file From f37d1b62670be6f5ebeced9e97f16d421a8fa06c Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:25:05 -0400 Subject: [PATCH 12/13] Update website/docs/reference/resource-configs/contract.md Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- website/docs/reference/resource-configs/contract.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/contract.md b/website/docs/reference/resource-configs/contract.md index 40e2cc8b27d..59cc511890b 100644 --- a/website/docs/reference/resource-configs/contract.md +++ b/website/docs/reference/resource-configs/contract.md @@ -33,7 +33,7 @@ The `data_type` defined in your YAML file must match a data type your data platf dbt uses built-in type aliasing for the `data_type` defined in your YAML. For example, you can specify `string` in your contract, and on Postgres/Redshift, dbt will convert it to `text`. If dbt doesn't recognize the `data_type` name among its known aliases, it will pass it through as-is. This is enabled by default, but you can opt-out by setting `alias_types` to `false`. -Example: +Example for disabling: ```yml From f116fe9cb158a5c5add8aec3e21e9ebcec6b9caa Mon Sep 17 00:00:00 2001 From: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Date: Thu, 12 Oct 2023 13:20:44 -0600 Subject: [PATCH 13/13] Remove comment from code example --- website/docs/reference/commands/list.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/commands/list.md b/website/docs/reference/commands/list.md index 40e58178ee0..93a0b87dd93 100644 --- a/website/docs/reference/commands/list.md +++ b/website/docs/reference/commands/list.md @@ -97,8 +97,8 @@ $ dbt ls --select snowplow.* --output json --output-keys name resource_type desc **Listing Semantic models** +List all resources upstream of your orders semantic model: ``` -# list all resources upstream of your orders semantic model dbt ls -s +semantic_model:orders ```