Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to CI jobs page #4706

Merged
merged 73 commits into from
Jan 22, 2024
Merged
Show file tree
Hide file tree
Changes from 59 commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
7cf6633
Update CI jobs
nghi-ly Jan 4, 2024
d8a0e3e
Merge branch 'current' into ly-docs-ci-jobs
nghi-ly Jan 4, 2024
f6e645f
This branch was auto-updated!
github-actions[bot] Jan 4, 2024
84ae6e7
This branch was auto-updated!
github-actions[bot] Jan 4, 2024
2d83e75
This branch was auto-updated!
github-actions[bot] Jan 4, 2024
2fef3fc
Update website/docs/docs/deploy/ci-jobs.md
nghi-ly Jan 5, 2024
d122f0b
This branch was auto-updated!
github-actions[bot] Jan 5, 2024
15adb4f
This branch was auto-updated!
github-actions[bot] Jan 5, 2024
de780bc
This branch was auto-updated!
github-actions[bot] Jan 5, 2024
68419f6
This branch was auto-updated!
github-actions[bot] Jan 8, 2024
ed53de7
This branch was auto-updated!
github-actions[bot] Jan 8, 2024
063ded0
This branch was auto-updated!
github-actions[bot] Jan 8, 2024
ec1460c
This branch was auto-updated!
github-actions[bot] Jan 8, 2024
f1bd0b6
This branch was auto-updated!
github-actions[bot] Jan 8, 2024
33ba942
This branch was auto-updated!
github-actions[bot] Jan 8, 2024
1186555
This branch was auto-updated!
github-actions[bot] Jan 8, 2024
a30ad63
This branch was auto-updated!
github-actions[bot] Jan 8, 2024
7cd5199
This branch was auto-updated!
github-actions[bot] Jan 8, 2024
e5a191e
Feedback
nghi-ly Jan 8, 2024
c438cd1
Merge branch 'ly-docs-ci-jobs' of github.com:dbt-labs/docs.getdbt.com…
nghi-ly Jan 8, 2024
ecf9427
Update website/docs/docs/deploy/ci-jobs.md
nghi-ly Jan 8, 2024
553e347
This branch was auto-updated!
github-actions[bot] Jan 8, 2024
81f11d7
This branch was auto-updated!
github-actions[bot] Jan 9, 2024
1186ea1
This branch was auto-updated!
github-actions[bot] Jan 9, 2024
a0f341b
This branch was auto-updated!
github-actions[bot] Jan 9, 2024
7d1b006
This branch was auto-updated!
github-actions[bot] Jan 9, 2024
9ff5f50
This branch was auto-updated!
github-actions[bot] Jan 9, 2024
64a1b56
This branch was auto-updated!
github-actions[bot] Jan 9, 2024
39cf192
This branch was auto-updated!
github-actions[bot] Jan 10, 2024
8701209
This branch was auto-updated!
github-actions[bot] Jan 10, 2024
cf83200
This branch was auto-updated!
github-actions[bot] Jan 10, 2024
1ae5e0c
This branch was auto-updated!
github-actions[bot] Jan 10, 2024
50b6451
This branch was auto-updated!
github-actions[bot] Jan 10, 2024
de41200
This branch was auto-updated!
github-actions[bot] Jan 10, 2024
36a3145
This branch was auto-updated!
github-actions[bot] Jan 11, 2024
4bf3409
This branch was auto-updated!
github-actions[bot] Jan 11, 2024
42fd534
This branch was auto-updated!
github-actions[bot] Jan 12, 2024
e139148
This branch was auto-updated!
github-actions[bot] Jan 12, 2024
42b4df5
This branch was auto-updated!
github-actions[bot] Jan 12, 2024
c33e4df
This branch was auto-updated!
github-actions[bot] Jan 12, 2024
6715cb0
This branch was auto-updated!
github-actions[bot] Jan 12, 2024
fa7678f
This branch was auto-updated!
github-actions[bot] Jan 12, 2024
1e6e4bd
This branch was auto-updated!
github-actions[bot] Jan 15, 2024
bd29c64
This branch was auto-updated!
github-actions[bot] Jan 15, 2024
a40fd15
This branch was auto-updated!
github-actions[bot] Jan 16, 2024
21bdf78
This branch was auto-updated!
github-actions[bot] Jan 16, 2024
8df4c9e
This branch was auto-updated!
github-actions[bot] Jan 16, 2024
b8026f5
This branch was auto-updated!
github-actions[bot] Jan 16, 2024
6a623b3
This branch was auto-updated!
github-actions[bot] Jan 16, 2024
78c3d94
Feedback 2
nghi-ly Jan 17, 2024
dd0b400
This branch was auto-updated!
github-actions[bot] Jan 17, 2024
3e80e48
This branch was auto-updated!
github-actions[bot] Jan 17, 2024
2f60ca7
This branch was auto-updated!
github-actions[bot] Jan 17, 2024
bab688e
Differentiate btn free and developer plans
nghi-ly Jan 17, 2024
92f34a7
Merge branch 'ly-docs-ci-jobs' of github.com:dbt-labs/docs.getdbt.com…
nghi-ly Jan 17, 2024
4670729
This branch was auto-updated!
github-actions[bot] Jan 17, 2024
cb67daa
This branch was auto-updated!
github-actions[bot] Jan 17, 2024
e16c845
This branch was auto-updated!
github-actions[bot] Jan 17, 2024
2466c0d
This branch was auto-updated!
github-actions[bot] Jan 18, 2024
3790e35
This branch was auto-updated!
github-actions[bot] Jan 18, 2024
2afccb9
This branch was auto-updated!
github-actions[bot] Jan 18, 2024
f8bdd5a
This branch was auto-updated!
github-actions[bot] Jan 18, 2024
5eb237c
This branch was auto-updated!
github-actions[bot] Jan 19, 2024
f47fbf3
This branch was auto-updated!
github-actions[bot] Jan 19, 2024
e534328
This branch was auto-updated!
github-actions[bot] Jan 19, 2024
c7c173a
This branch was auto-updated!
github-actions[bot] Jan 19, 2024
bc098b6
This branch was auto-updated!
github-actions[bot] Jan 19, 2024
4025d83
This branch was auto-updated!
github-actions[bot] Jan 22, 2024
cedaff6
This branch was auto-updated!
github-actions[bot] Jan 22, 2024
ebb488f
This branch was auto-updated!
github-actions[bot] Jan 22, 2024
68e7d20
This branch was auto-updated!
github-actions[bot] Jan 22, 2024
c18a172
Update website/docs/docs/deploy/ci-jobs.md
nghi-ly Jan 22, 2024
4596048
Feedback
nghi-ly Jan 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 8 additions & 23 deletions website/docs/docs/deploy/ci-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,12 @@ You can set up [continuous integration](/docs/deploy/continuous-integration) (CI

dbt Labs recommends that you create your CI job in a dedicated dbt Cloud [deployment environment](/docs/deploy/deploy-environments#create-a-deployment-environment) that's connected to a staging database. Having a separate environment dedicated for CI will provide better isolation between your temporary CI schema builds and your production data builds. Additionally, sometimes teams need their CI jobs to be triggered when a PR is made to a branch other than main. If your team maintains a staging branch as part of your release process, having a separate environment will allow you to set a [custom branch](/faqs/environments/custom-branch-settings) and, accordingly, the CI job in that dedicated environment will be triggered only when PRs are made to the specified custom branch. To learn more, refer to [Get started with CI tests](/guides/set-up-ci).


### Prerequisites
- You have a dbt Cloud account.
- You have a dbt Cloud account.
- For the [Concurrent CI checks](/docs/deploy/continuous-integration#concurrent-ci-checks) and [Smart cancellation of stale builds](/docs/deploy/continuous-integration#smart-cancellation) features, your dbt Cloud account must be on the [Team or Enterprise plan](https://www.getdbt.com/pricing/).
- You must be connected using dbt Cloud’s native Git integration with [GitHub](/docs/cloud/git/connect-github), [GitLab](/docs/cloud/git/connect-gitlab), or [Azure DevOps](/docs/cloud/git/connect-azure-devops).
nghi-ly marked this conversation as resolved.
Show resolved Hide resolved
- With GitLab, you need a paid or self-hosted account which includes support for GitLab webhooks and [project access tokens](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html). With GitLab Free, merge requests will invoke CI jobs but CI status updates (success or failure of the job) will not be reported back to GitLab.
- If you previously configured your dbt project by providing a generic git URL that clones using SSH, you must reconfigure the project to connect through dbt Cloud's native integration.
- Set up a [connection with your Git provider](/docs/cloud/git/git-configuration-in-dbt-cloud). This integration lets dbt Cloud run jobs on your behalf for job triggering.
- If you're using a native [GitLab](/docs/cloud/git/connect-gitlab) integration, you need a paid or self-hosted account which includes support for GitLab webhooks and [project access tokens](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html). With GitLab Free, merge requests will invoke CI jobs but CI status updates (success or failure of the job) will not be reported back to GitLab.
nghi-ly marked this conversation as resolved.
Show resolved Hide resolved


To make CI job creation easier, many options on the **CI job** page are set to default values that dbt Labs recommends that you use. If you don't want to use the defaults, you can change them.
Expand Down Expand Up @@ -63,12 +63,13 @@ If you're not using dbt Cloud’s native Git integration with [GitHub](/docs/cl


1. Set up a CI job with the [Create Job](/dbt-cloud/api-v2#/operations/Create%20Job) API endpoint using `"job_type": ci` or from the [dbt Cloud UI](#set-up-ci-jobs).
nghi-ly marked this conversation as resolved.
Show resolved Hide resolved
1. Call the [Trigger Job Run](/dbt-cloud/api-v2#/operations/Trigger%20Job%20Run) API endpoint to trigger the CI job. You must include these fields to the payload:
- Provide the pull request (PR) ID with one of these fields, even if you're using a different Git provider (like Bitbucket). This can make your code less human-readable but it will _not_ affect dbt functionality.
1. Call the [Trigger Job Run](/dbt-cloud/api-v2#/operations/Trigger%20Job%20Run) API endpoint to trigger the CI job. You must include both of these fields to the payload:
- Provide the pull request (PR) ID using one of these fields:

- `github_pull_request_id`
- `gitlab_merge_request_id`
- `azure_devops_pull_request_id` 
- `azure_devops_pull_request_id`
- `non_native_pull_request_id` (for example, BitBucket)
nghi-ly marked this conversation as resolved.
Show resolved Hide resolved
- Provide the `git_sha` or `git_branch` to target the correct commit or branch to run the job against.

## Example pull requests
Expand Down Expand Up @@ -110,22 +111,6 @@ If you're experiencing any issues, review some of the common questions and answe
</div>
</div>
</details>
<details>
<summary>Reconnecting your dbt project to use dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps</summary>
<div>
<div>If your dbt project relies the generic git clone method that clones using SSH and deploy keys to connect to your dbt repo, you need to disconnect your repo and reconnect it using the native GitHub, GitLab, or Azure DevOps integration in order to enable dbt Cloud CI.<br></br><br></br>
First, make sure you have the <a href="https://docs.getdbt.com/docs/cloud/git/connect-github">native GitHub authentication</a>, <a href="https://docs.getdbt.com/docs/cloud/git/connect-gitlab">native GitLab authentication</a>, or <a href="https://docs.getdbt.com/docs/cloud/git/connect-azure-devops">native Azure DevOps authentication</a> set up depending on which git provider you use. After you have gone through those steps, go to <strong>Account Settings</strong>, select <strong>Projects</strong> and click on the project you'd like to reconnect through native GitHub, GitLab, or Azure DevOps auth. Then click on the repository link.<br></br><br></br>

Once you're in the repository page, select <strong>Edit</strong> and then <strong>Disconnect Repository</strong> at the bottom.<br></br>
<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/Enabling-CI/Disconnect-Repository.png" alt="Disconnect repo"/>
<br></br>
Confirm that you'd like to disconnect your repository. You should then see a new Configure a repository link in your old repository's place. Click through to the configuration page:<br></br>
<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/Enabling-CI/repo-config.png" alt="Configure repo"/>
<br></br>

Select the <b>GitHub</b>, <b>GitLab</b>, or <b>AzureDevOps</b> tab and reselect your repository. That should complete the setup of the project and enable you to set up a dbt Cloud CI job.</div>
</div>
</details>
<details>
<summary>Error messages that refer to schemas from previous PRs</summary>
<div>
Expand Down
3 changes: 3 additions & 0 deletions website/docs/docs/deploy/continuous-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,6 @@ When you push a new commit to a PR, dbt Cloud enqueues a new CI run for the late

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-smart-cancel-job.png" width="70%" title="Example of an automatically canceled run"/>

### Run slot treatment
nghi-ly marked this conversation as resolved.
Show resolved Hide resolved

CI runs for accounts on the [Enterprise, Team](https://www.getdbt.com/pricing), and [Free (trial)](https://www.getdbt.com/signup) plans don't consume run slots so a CI check will never block a production run.
Copy link
Contributor

@mirnawong1 mirnawong1 Jan 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sugg making it more explicit that's it the 14 day trial (nothing to do with a free plan even though they also apply) and switching the emphasis from 'free' to '14-day'. also sugg removing 'accounts on' as it's a little redundant since 'plans' is also in the sentence.

also i don't believe we often clarify what features are accessible to 14-day free trials in the docs, however, always good to clarify this. (is this a plan pill use case too?)

Suggested change
CI runs for accounts on the [Enterprise, Team](https://www.getdbt.com/pricing), and [Free (trial)](https://www.getdbt.com/signup) plans don't consume run slots so a CI check will never block a production run.
CI runs won't consume run slots for [Enterprise or Team ](https://www.getdbt.com/pricing) plans, or [14-day free trials](https://www.getdbt.com/signup). This ensures a CI check will never block a production run. However, CI runs will count toward run slots for Developer plans.

Copy link
Contributor

@mirnawong1 mirnawong1 Jan 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or CI runs won't consume run slots for [Enterprise or Team](https://www.getdbt.com/pricing) plans, and the [14-day free trial](https://www.getdbt.com/signup). This ensures a CI check will never block a production run.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how are the 14-day free trials categorized in the back end? in this case, we do mean the free plan (which should have all the same features as enterprise plans)

Copy link
Contributor

@mirnawong1 mirnawong1 Jan 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revised suggestion:
CI runs won't consume run slots for Enterprise or Team plans, or 14-day free trials. This ensures a CI check will never block a production run. However, CI runs will count toward run slots for Developer plans.

Copy link
Contributor

@mirnawong1 mirnawong1 Jan 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@knartovich 14-day free trials are labeled as 'trial' in the backend:
Screenshot 2024-01-18 at 15 09 39

oh ok so Ly's original below was meant for our partner 'free' plan? i thought it was the 14 day trial bc it had trial in the parenthesis and linked to the trial sign up page 🙈:

CI runs for accounts on the Enterprise, Team, and Free (trial) plans don't consume run slots so a CI check will never block a production run.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh ok so Ly's original below was meant for our partner 'free' plan? i thought it was the 14 day trial bc it had trial in the parenthesis and linked to the trial sign up page 🙈

i think we were just confused about what exactly the free plan was 😅

this does apply to trial accounts, but i'm not sure if that's worth calling out here. these are the backend values that get this feature:
enterprise
free
team
team_2022
trial
trial_2022

does that clear it up?

Copy link
Contributor

@mirnawong1 mirnawong1 Jan 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah gotcha, makes sense! totally agree about the trial call out and said similar here @knartovich but maybe doesn't hurt to callout! we don't usually state whether features are avail for 14-day trials (or our free partner plans for that matter too).

i do sugg we should callout that dev plans don't get this feature, IMHO. But may be ott

Copy link
Contributor Author

@nghi-ly nghi-ly Jan 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you so much, this clears up a lot, @mirnawong1 and @knartovich !

i agree we don't have prior art for calling out our trials or the free partner plans. will update so it says team and enterprise.

Loading