Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cloud cli updates #4354

Merged
merged 23 commits into from
Nov 8, 2023
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
138a4ca
adding new updates
mirnawong1 Oct 27, 2023
bdc6aed
remove duplicative
mirnawong1 Oct 27, 2023
769aeb5
add sso
mirnawong1 Oct 27, 2023
622c584
Merge branch 'current' into mwong-cli-extensions
mirnawong1 Oct 27, 2023
8e4cc78
update per eng
mirnawong1 Oct 27, 2023
dceb36e
Merge branch 'mwong-cli-extensions' of https://github.com/dbt-labs/do…
mirnawong1 Oct 27, 2023
4a23df5
Update website/docs/docs/cloud/cloud-cli-installation.md
mirnawong1 Oct 27, 2023
d0b0706
Update website/docs/reference/dbt-commands.md
mirnawong1 Oct 27, 2023
e169f4d
Merge branch 'current' into mwong-cli-extensions
mirnawong1 Oct 31, 2023
83e349f
Update cloud-cli-installation.md
mirnawong1 Oct 31, 2023
a747480
Update cloud-cli-installation.md
mirnawong1 Oct 31, 2023
b613974
Update website/docs/docs/cloud/cloud-cli-installation.md
mirnawong1 Oct 31, 2023
c7a9497
Update configure-cloud-cli.md
mirnawong1 Oct 31, 2023
d752208
Merge branch 'current' into mwong-cli-extensions
mirnawong1 Nov 1, 2023
95789bc
tweak
mirnawong1 Nov 1, 2023
c4a81d0
Merge branch 'current' into mwong-cli-extensions
mirnawong1 Nov 1, 2023
2e8d29d
Merge branch 'current' into mwong-cli-extensions
mirnawong1 Nov 1, 2023
f461616
fold greg's feedback
mirnawong1 Nov 1, 2023
921208c
Merge branch 'mwong-cli-extensions' of https://github.com/dbt-labs/do…
mirnawong1 Nov 1, 2023
52207df
Update website/docs/docs/cloud/cloud-cli-installation.md
mirnawong1 Nov 2, 2023
b741024
Merge branch 'current' into mwong-cli-extensions
mirnawong1 Nov 6, 2023
76df025
Merge branch 'current' into mwong-cli-extensions
mirnawong1 Nov 8, 2023
4d194ca
fold greg's feedback
mirnawong1 Nov 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 53 additions & 32 deletions website/docs/docs/cloud/cloud-cli-installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,14 @@ dbt commands are run against dbt Cloud's infrastructure and benefit from:
The dbt Cloud CLI is available in all [deployment regions](/docs/cloud/about-cloud/regions-ip-addresses) and for both multi-tenant and single-tenant accounts (Azure single-tenant not supported at this time).

- Ensure you are using dbt version 1.5 or higher. Refer to [dbt Cloud versions](/docs/dbt-versions/upgrade-core-in-cloud) to upgrade.
- Avoid using SSH tunneling for [Postgres and Redshift](/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb) connections.
- Avoid using [PrivateLink](/docs/cloud/secure/about-privatelink).
- Note that SSH tunneling for [Postgres and Redshift](/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb) connections, [PrivateLink](/docs/cloud/secure/about-privatelink), and [Single sign-on (SSO)](/docs/cloud/manage-access/sso-overview) isn't supported yet.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PrivateLink will be supported as of tomorrow!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome! will it be compatible with the SL ? cc @aiguofer

mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

## Install dbt Cloud CLI

You can install the dbt Cloud CLI on the command line by using one of these methods.

<details>
<summary>View a video tutorial for a step-by-step guide to installing</summary>
<summary>View a video tutorial for a step-by-step guide to installation.</summary>

<LoomVideo id="dd80828306c5432a996d4580135041b6?sid=fe1895b7-1281-4e42-9968-5f7d11768000"/>

Expand All @@ -44,7 +43,7 @@ You can install the dbt Cloud CLI on the command line by using one of these meth

<TabItem value="brew" label="macOS (brew)">

Before you begin, make sure you have [Homebrew installed](http://brew.sh/) in your code editor or command line terminal. Refer to the [FAQs](#faqs) if your operating system runs into path conflicts.
Before you begin, make sure you have [Homebrew installed](http://brew.sh/) in your code editor or command line terminal. Refer to the [FAQs](#faqs) if your operating system runs into path conflicts.

1. Run the following command to verify that you don't already have dbt Core installed:

Expand Down Expand Up @@ -145,43 +144,45 @@ Advanced users can configure multiple projects to use the same Cloud CLI executa

<TabItem value="pip" label="Existing dbt Core users (pip)">

If you already have dbt Core installed, the dbt Cloud CLI may conflict. Here are some considerations:

:::info Use native packages or a virtual environment to prevent dbt Core conflicts

To prevent overwriting dbt Core, avoid installing the dbt Cloud CLI with pip. Instead, consider using the native installation method and configuring your PATH or create a new virtual environment.

If you've already installed the dbt Cloud CLI and need to switch back to dbt Core, uninstall the dbt Cloud CLI, and follow the dbt Core installation instructions.

You can also have both dbt Cloud CLI and dbt Core installed simultaneously. To avoid conflicts, alias the dbt Cloud CLI as `dbt-cloud`. For more details, check the [FAQs](#faqs) if your operating system experiences path conflicts.
:::
- **Prevent conflicts** <br /> If you want to use `pip`, create a new virtual environment.<br /><br />
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
- **Use both dbt Cloud CLI and dbt Core** <br /> If you use Homebrew, consider aliasing the dbt Cloud CLI as "dbt-cloud" to avoid conflict. For more details, check the [FAQs](#faqs) if your operating system experiences path conflicts.<br /><br />
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
- **Switch from the dbt Cloud CLI to dbt Core** <br />
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
If you've already installed the dbt Cloud CLI and need to switch back to dbt Core:<br />
- Uninstall the dbt Cloud CLI using the command: `pip uninstall dbt`
- Reinstall dbt Core using the following command, replacing "adapter_name" with the appropriate adapter name:
```shell
pip install dbt-adapter_name --force-reinstall
```
For example, if I used Snowflake as an adapter, I would run: `pip install dbt-snowflake --force-reinstall`

--------

Before installing the dbt Cloud CLI, make sure you have Python installed and your virtual environment venv or pyenv . If you already have a Python environment configured, you can skip to the [pip installation step](#install-dbt-cloud-cli-in-pip).


### Install a virtual environment

We recommend using virtual environments (venv) to namespace `cloud-cli`.

1. Create a new venv:
1. Create a new virtual environment named "dbt-cloud" with this command:
```shell
python3 -m venv dbt-cloud
```

2. Activate the virtual environment each time you create a shell window or session:
```shell
source dbt-cloud/bin/activate # activate the environment for Mac and Linux OR
dbt-env\Scripts\activate # activate the environment for Windows
```
2. Activate the virtual environment each time you create a shell window or session, depending on your operating system:

3. (Mac and Linux only) Create an alias to activate your dbt environment with every new shell window or session. You can add the following to your shell's configuration file (for example, $HOME/.bashrc, $HOME/.zshrc) while replacing `<PATH_TO_VIRTUAL_ENV_CONFIG>` with the path to your virtual environment configuration:
- For Mac and Linux, use: `source dbt-cloud/bin/activate`<br/>
- For Windows, use: `dbt-env\Scripts\activate`

3. (Mac and Linux only) Create an alias to activate your dbt environment with every new shell window or session. You can add the following to your shell's configuration file (for example, `$HOME/.bashrc, $HOME/.zshrc`) while replacing `<PATH_TO_VIRTUAL_ENV_CONFIG>` with the path to your virtual environment configuration:
```shell
alias env_dbt='source <PATH_TO_VIRTUAL_ENV_CONFIG>/bin/activate'
```

### Install dbt Cloud CLI in pip

1. (Optional) If you already have dbt Core installed, this installation will override that package. Note your dbt Core version in case you need to reinstall it later:
1. (Optional) If you already have dbt Core installed, this installation will override that package. Check your dbt Core version in case you need to reinstall it later by running the following command :

```bash
dbt --version
Expand All @@ -193,21 +194,20 @@ We recommend using virtual environments (venv) to namespace `cloud-cli`.
pip3 install dbt
```

3. (Optional) To revert back to dbt Core, first uninstall both the dbt Cloud CLI and dbt Core
4. Reinstall dbt Core using the version from Step 2.
1. (Optional) To revert back to dbt Core, first uninstall both the dbt Cloud CLI and dbt Core. Then reinstall dbt Core.

```bash
pip3 uninstall dbt-core dbt
pip3 install dbt-core==VERSION
pip install dbt-adapter_name --force-reinstall
```

4. After you've verified the installation, [configure](/docs/cloud/configure-cloud-cli) the dbt Cloud CLI for your dbt Cloud project and use it to run [dbt commands](/reference/dbt-commands) similar to dbt Core. For example, execute `dbt compile` to compile a project using dbt Cloud and validate your models and tests.
4. After you've verified the installation, [configure](/docs/cloud/configure-cloud-cli) the dbt Cloud CLI for your dbt Cloud project. You can then use it to run [dbt commands](/reference/dbt-commands) similar to dbt Core. For example, execute `dbt compile` to compile a project using dbt Cloud and validate your models and tests.
* If you're using the dbt Cloud CLI, you can connect to your data platform directly in the dbt Cloud interface and don't need a [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml) file locally on your machine.

</TabItem>

</Tabs>

</Tabs>

## Update dbt Cloud CLI

Expand Down Expand Up @@ -246,6 +246,13 @@ To update:

</Tabs>

## Using VS Code extensions
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

Visual Studio (VS) Code extensions enhance command line tools by adding extra functionalities. The dbt Cloud CLI is fully compatible with dbt Core, however it doesn't support some dbt Core APIs required by certain tools, for example VS Code extensions.

To use these extensions, such as dbt-power-user, with the dbt Cloud CLI, you can install it using Homebrew (along with dbt Core) and create an alias to run the dbt Cloud CLI as `dbt-cloud`. This allows dbt-power-user to continue to work alongside the dbt Cloud CLI.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved


## FAQs

<details>
Expand All @@ -257,19 +264,33 @@ The dbt Cloud CLI and <a href="https://github.com/dbt-labs/dbt-core">dbt Core</a

<details>
<summary>How do I run both the dbt Cloud CLI and dbt Core?</summary>
For compatibility, both the dbt Cloud CLI and dbt Core are invoked by running <code>dbt</code>. This can create path conflicts if your operating system selects one over the other based on your $PATH environment variable (settings).
For compatibility, both the dbt Cloud CLI and dbt Core are invoked by running <code>dbt</code>. This can create path conflicts if your operating system selects one over the other based on your $PATH environment variable (settings).<br />

If you have dbt Core installed locally, either:

1. Install using [pip](/docs/cloud/cloud-cli-installation?install=pip#install-dbt-cloud-cli).

2. Install natively, but ensure that you deactivate your Python environment or uninstall it using `pip uninstall dbt` before proceeding.

1. Install using the <code>pip3 install dbt</code> [pip](/docs/cloud/cloud-cli-installation?install=pip#install-dbt-cloud-cli) command, or
2. Install natively, ensuring you either deactivate the virtual environment containing dbt Core or create an alias for the dbt Cloud CLI.
3. (Advanced users) Install natively, but modify the $PATH environment variable to correctly point to the dbt Cloud CLI binary to use both dbt Cloud CLI and dbt Core together.

You can always uninstall the dbt Cloud CLI to return to using dbt Core.
</details>

<details>
<summary>How to create an alias?</summary>
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
To create an alias for the dbt Cloud CLI: <br />

1. Open your shell's profile configuration file. Deepending on your shell and system, this could be <code>~/.bashrc</code>, <code>~/.bash_profile</code>, <code>~/.zshrc</code>, or another file.<br />
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

2. Add an alias that points to the dbt Cloud CLI binary. For example:<code>alias dbt-cloud="path_to_dbt_cloud_cli_binary</code>

Replace <code>path_to_dbt_cloud_cli_binary</code> with the actual path to the dbt Cloud CLI binary. With this alias, you can use the command <code>dbt-cloud</code> to invoke the dbt Cloud CLI.<br />
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

3. Save the file and then either restart your shell or run <code>source</code> on the profile file to apply the changes.
As an example, in bash you would run: <code>source ~/.bashrc</code><br />

This alias will allow you to use the <code>dbt-cloud</code> command to invoke the dbt Cloud CLI while having dbt Core installed natively.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
</details>

<details>
<summary>Why am I receiving a <code>Session occupied</code> error?</summary>
If you've ran a dbt command and receive a <code>Session occupied</code> error, you can reattach to your existing session with <code>dbt reattach</code> and then press <code>Control-C</code> and choose to cancel the invocation.
Expand Down
6 changes: 3 additions & 3 deletions website/docs/docs/cloud/configure-cloud-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,10 +87,10 @@ To set environment variables in the dbt Cloud CLI for your dbt project:

## Use the dbt Cloud CLI

- The dbt Cloud CLI shares the same set of [dbt commands](/reference/dbt-commands) as dbt Core and processes the commands you invoke.
- It allows you to use automatic deferral of build artifacts to your Cloud project's production environment.
- The dbt Cloud CLI uses the same set of [dbt commands](/reference/dbt-commands) and [MetricFlow commands](/docs/build/metricflow-commands) as dbt Core to execute the commands you provide.
- It allows you to automatically defer build artifacts to your Cloud project's production environment.
- It also supports [project dependencies](/docs/collaborate/govern/project-dependencies), which allows you to depend on another project using the metadata service in dbt Cloud.
- Project dependencies instantly connect to and reference (or `ref`) public models defined in other projects. This means you don't need to execute or analyze these upstream models yourself. Instead, you treat them as an API that returns a dataset.
- Project dependencies instantly connect to and reference (or `ref`) public models defined in other projects. You don't need to execute or analyze these upstream models yourself. Instead, you treat them as an API that returns a dataset.

:::tip Use the <code>--help</code> flag
As a tip, most command-line tools have a `--help` flag to show available commands and arguments. Use the `--help` flag with dbt in two ways:
Expand Down
17 changes: 17 additions & 0 deletions website/docs/reference/commands/clone.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ The `clone` command is useful for:
- handling incremental models in dbt Cloud CI jobs (on data warehouses that support zero-copy cloning tables)
- testing code changes on downstream dependencies in your BI tool


```bash
# clone all of my models from specified state to my target schema(s)
dbt clone --state path/to/artifacts
Expand All @@ -37,3 +38,19 @@ Unlike deferral, `dbt clone` requires some compute and creation of additional ob
For example, by creating actual data warehouse objects, `dbt clone` allows you to test out your code changes on downstream dependencies _outside of dbt_ (such as a BI tool).

As another example, you could `clone` your modified incremental models as the first step of your dbt Cloud CI job to prevent costly `full-refresh` builds for warehouses that support zero-copy cloning.

## Cloning in dbt Cloud

You can clone nodes between states in dbt Cloud using the `dbt clone` command. This is available in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) and the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) and relies on the [`--defer`](/reference/node-selection/defer) feature. For more details on defer in dbt Cloud, read [Using defer in dbt Cloud](/docs/cloud/about-cloud-develop-defer).

- **Using dbt Cloud CLI** &mdash; The `dbt clone` command in the dbt Cloud CLI automatically includes the `--defer` flag. This means you can use the `dbt clone` command without any additional setup.

- **Using dbt Cloud IDE** &mdash; To use the `dbt clone` command in the dbt Cloud IDE, follow these steps before running the `dbt clone` command:

- Set up your **Production environment** and have a successful job run.
- Enable **Defer to production** by toggling the switch in the lower-right corner of the command bar.
<Lightbox src="/img/docs/dbt-cloud/defer-toggle.jpg" width="80%" title="Select the 'Defer to production' toggle on the bottom right of the command bar to enable defer in the dbt Cloud IDE."/>
- Run the `dbt clone` command from the command bar.



8 changes: 4 additions & 4 deletions website/docs/reference/dbt-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The following sections outline the commands supported by dbt and their relevant

### Available commands

<VersionBlock firstVersion="1.5">
<VersionBlock firstVersion="1.6">

All commands in the table are compatible with either the dbt Cloud IDE, dbt Cloud CLI, or dbt Core.

Expand All @@ -22,9 +22,9 @@ You can run dbt commands in your specific tool by prefixing them with `dbt`. Fo
| [build](/reference/commands/build) | Build and test all selected resources (models, seeds, snapshots, tests) | All | All [supported versions](/docs/dbt-versions/core) |
| cancel | Cancels the most recent invocation.| dbt Cloud CLI | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) |
| [clean](/reference/commands/clean) | Deletes artifacts present in the dbt project | All | All [supported versions](/docs/dbt-versions/core) |
| [clone](/reference/commands/clone) | Clone selected models from the specified state | dbt Cloud CLI <br /> dbt Core | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) |
| [clone](/reference/commands/clone) | Clone selected models from the specified state | All | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) |
| [compile](/reference/commands/compile) | Compiles (but does not run) the models in a project | All | All [supported versions](/docs/dbt-versions/core) |
| [debug](/reference/commands/debug) | Debugs dbt connections and projects | dbt Core | All [supported versions](/docs/dbt-versions/core) |
| [debug](/reference/commands/debug) | Debugs dbt connections and projects | dbt Cloud IDE <br /> dbt Core | All [supported versions](/docs/dbt-versions/core) |
| [deps](/reference/commands/deps) | Downloads dependencies for a project | All | All [supported versions](/docs/dbt-versions/core) |
| [docs](/reference/commands/cmd-docs) | Generates documentation for a project | All | All [supported versions](/docs/dbt-versions/core) |
| help | Displays help information for any command | dbt Core <br /> dbt Cloud CLI | All [supported versions](/docs/dbt-versions/core) |
Expand All @@ -44,7 +44,7 @@ You can run dbt commands in your specific tool by prefixing them with `dbt`. Fo

</VersionBlock>

<VersionBlock lastVersion="1.4">
<VersionBlock lastVersion="1.5">

Select the tabs that are relevant to your development workflow. For example, if you develop in the dbt Cloud IDE, select **dbt Cloud**.

Expand Down
Loading