Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc: add clarification for cross-region PL #31018

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 15 additions & 9 deletions doc/user/content/ingest-data/kafka/amazon-msk.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,24 +32,30 @@ Before you begin, you must have:

## Creating a connection

There are various ways to configure your Kafka network to allow Materialize to connect:
There are various ways to configure your Kafka network to allow Materialize to
connect:

- **Use AWS PrivateLink**: If your Kafka cluster is running on AWS, you can use AWS PrivateLink to connect Materialize to the cluster.
- **Allow Materialize IPs:** If your Kafka cluster is publicly accessible, you
can configure your firewall to allow connections from a set of static
Materialize IP addresses.

- **Use an SSH tunnel**: If your Kafka cluster is running in a private network, you can use an SSH tunnel to connect Materialize to the cluster.
- **Use AWS PrivateLink**: If your Kafka cluster is running in a private network, you
can use [AWS PrivateLink](/ingest-data/network-security/privatelink/) to
connect Materialize to the cluster. For details, see [AWS PrivateLink](/ingest-data/network-security/privatelink/).

- **Allow Materialize IPs**: If your Kafka cluster is publicly accessible, you can configure your firewall to allow connections from a set of static Materialize IP addresses.

Select the option that works best for you.
- **Use an SSH tunnel:** If your Kafka cluster is running in a private network, you
can use an SSH tunnel to connect Materialize to the cluster.

{{< tabs tabID="1" >}}

{{< tab "Privatelink">}}

{{< note >}}
Materialize provides Terraform modules for both [Amazon MSK clusters](https://github.com/MaterializeInc/terraform-aws-msk-privatelink) and [self-managed Kafka clusters](https://github.com/MaterializeInc/terraform-aws-kafka-privatelink) which can be used to create the target groups for each Kafka broker (step 1), the network load balancer (step 2),
the TCP listeners (step 3) and the VPC endpoint service (step 5).
{{< /note >}}
Materialize provides a Terraform module that automates the creation and
configuration of AWS resources for a PrivateLink connection. For more details,
see the Terraform module repositories for [Amazon MSK](https://github.com/MaterializeInc/terraform-aws-msk-privatelink)
and [self-managed Kafka clusters](https://github.com/MaterializeInc/terraform-aws-kafka-privatelink).
{{</ note >}}

{{% network-security/privatelink-kafka %}}

Expand Down
17 changes: 9 additions & 8 deletions doc/user/content/ingest-data/mysql/amazon-aurora.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,15 +64,12 @@ to connect:
configure your database's security group to allow connections from a set of
static Materialize IP addresses.

- **Use an SSH tunnel:** If your database is running in a private network, you
can use an SSH tunnel to connect Materialize to the database.

- **Use AWS PrivateLink**: If your database is running in a private network, you
can use [AWS PrivateLink](/ingest-data/network-security/privatelink/) to
connect Materialize to the database. For details, see [AWS
PrivateLink](/ingest-data/network-security/privatelink/).
connect Materialize to the database. For details, see [AWS PrivateLink](/ingest-data/network-security/privatelink/).

Select the option that works best for you.
- **Use an SSH tunnel:** If your database is running in a private network, you
can use an SSH tunnel to connect Materialize to the database.

{{< tabs >}}

Expand Down Expand Up @@ -131,7 +128,7 @@ see the [Terraform module repository](https://github.com/MaterializeInc/terrafor

- Choose the **IP addresses** type.

- Set the protocol and port to **TCP** and **5432**.
- Set the protocol and port to **TCP** and **3306**.

- Choose the same VPC as your RDS instance.

Expand All @@ -154,7 +151,7 @@ see the [Terraform module repository](https://github.com/MaterializeInc/terrafor
in.

- For **Listeners and routing**, set the protocol and port to **TCP**
and **5432** and select the target group you created in the previous
and **3306** and select the target group you created in the previous
step.

1. In the security group of your Aurora instance, [allow traffic from the the
Expand Down Expand Up @@ -276,6 +273,10 @@ networking configuration, so start by selecting the relevant option.
{{% mysql-direct/ingesting-data/allow-materialize-ips %}}
{{< /tab >}}

{{< tab "Use AWS PrivateLink">}}
{{% mysql-direct/ingesting-data/use-aws-privatelink %}}
{{< /tab >}}

{{< tab "Use an SSH tunnel">}}
{{% mysql-direct/ingesting-data/use-ssh-tunnel %}}
{{< /tab >}}
Expand Down
17 changes: 10 additions & 7 deletions doc/user/content/ingest-data/mysql/amazon-rds.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,13 +118,12 @@ to connect:
configure your database's security group to allow connections from a set of
static Materialize IP addresses.

- **Use an SSH tunnel:** If your database is running in a private network, you
can use an SSH tunnel to connect Materialize to the database.

- **Use AWS PrivateLink**: If your database is running in a private network, you
can use [AWS PrivateLink](/ingest-data/network-security/privatelink/) to
connect Materialize to the database. For details, see [AWS
PrivateLink](/ingest-data/network-security/privatelink/).
connect Materialize to the database. For details, see [AWS PrivateLink](/ingest-data/network-security/privatelink/).

- **Use an SSH tunnel:** If your database is running in a private network, you
can use an SSH tunnel to connect Materialize to the database.

{{< tabs >}}

Expand Down Expand Up @@ -184,7 +183,7 @@ see the [Terraform module repository](https://github.com/MaterializeInc/terrafor
- Choose the **IP addresses** type.
- Set the protocol and port to **TCP** and **5432**.
- Set the protocol and port to **TCP** and **3306**.
- Choose the same VPC as your RDS instance.
Expand All @@ -207,7 +206,7 @@ see the [Terraform module repository](https://github.com/MaterializeInc/terrafor
in.
- For **Listeners and routing**, set the protocol and port to **TCP**
and **5432** and select the target group you created in the previous
and **3306** and select the target group you created in the previous
step.
1. In the security group of your RDS instance, [allow traffic from the network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-register-targets.html).
Expand Down Expand Up @@ -327,6 +326,10 @@ start by selecting the relevant option.
{{% mysql-direct/ingesting-data/allow-materialize-ips %}}
{{< /tab >}}
{{< tab "Use AWS PrivateLink">}}
{{% mysql-direct/ingesting-data/use-aws-privatelink %}}
{{< /tab >}}
{{< tab "Use an SSH tunnel">}}
{{% mysql-direct/ingesting-data/use-ssh-tunnel %}}
{{< /tab >}}
Expand Down
34 changes: 26 additions & 8 deletions doc/user/content/ingest-data/network-security/privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,35 +10,53 @@ menu:
name: "AWS PrivateLink connections"
---

[//]: # "TODO(morsapaes) Add shortcode with instructions for AWS RDS MySQL"

Materialize can connect to a Kafka broker, a Confluent Schema Registry server, a
PostgreSQL database, or a MySQL database through an [AWS PrivateLink](https://aws.amazon.com/privatelink/)
service.

In this guide, we'll cover how to create `AWS PRIVATELINK` connections
and retrieve the AWS principal needed to configure the AWS PrivateLink service.
In this guide, we'll cover how to create `AWS PRIVATELINK` connections and
retrieve the AWS principal needed to configure the AWS PrivateLink service.

## Create an AWS PrivateLink connection

{{< tabs tabID="1" >}}
{{< tab "Kafka on AWS">}}

{{< note >}}
Materialize provides Terraform modules for both [MSK cluster](https://github.com/MaterializeInc/terraform-aws-msk-privatelink) and [self-managed Kafka clusters](https://github.com/MaterializeInc/terraform-aws-kafka-privatelink) which can be used to create the target groups for each Kafka broker (step 1), the network load balancer (step 2),
the TCP listeners (step 3) and the VPC endpoint service (step 5).
{{< /note >}}
Materialize provides a Terraform module that automates the creation and
configuration of AWS resources for a PrivateLink connection. For more details,
see the Terraform module repositories for [Amazon MSK](https://github.com/MaterializeInc/terraform-aws-msk-privatelink)
and [self-managed Kafka clusters](https://github.com/MaterializeInc/terraform-aws-kafka-privatelink).
{{</ note >}}

{{% network-security/privatelink-kafka %}}

{{< /tab >}}

{{< tab "AWS RDS">}}
{{< tab "AWS RDS (PostgreSQL)">}}

{{< note >}}
Materialize provides a Terraform module that automates the creation and
configuration of AWS resources for a PrivateLink connection. For more details,
see the [Terraform module repository](https://github.com/MaterializeInc/terraform-aws-rds-privatelink).
{{</ note >}}

{{% network-security/privatelink-postgres %}}

{{< /tab >}}

{{< tab "AWS RDS (MySQL)">}}

{{< note >}}
Materialize provides a Terraform module that automates the creation and
configuration of AWS resources for a PrivateLink connection. For more details,
see the [Terraform module repository](https://github.com/MaterializeInc/terraform-aws-rds-privatelink).
{{</ note >}}

{{% network-security/privatelink-mysql %}}

{{< /tab >}}

{{< /tabs >}}

## Related pages
Expand Down
19 changes: 10 additions & 9 deletions doc/user/content/ingest-data/postgres/amazon-aurora.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ menu:
---

This page shows you how to stream data from [Amazon Aurora for PostgreSQL](https://aws.amazon.com/rds/aurora/)
to Materialize using the[PostgreSQL source](/sql/create-source/postgres/).
to Materialize using the [PostgreSQL source](/sql/create-source/postgres/).

{{< tip >}}
{{< guided-tour-blurb-for-ingest-data >}}
Expand Down Expand Up @@ -56,17 +56,18 @@ to connect:
configure your database's security group to allow connections from a set of
static Materialize IP addresses.

- **Use AWS PrivateLink** or **Use an SSH tunnel:** If your database is running
in a private network, you can use either [AWS PrivateLink](https://aws.amazon.com/privatelink/)
or an SSH tunnel to connect Materialize to the database.
- **Use AWS PrivateLink**: If your database is running in a private network, you
can use [AWS PrivateLink](/ingest-data/network-security/privatelink/) to
connect Materialize to the database. For details, see [AWS PrivateLink](/ingest-data/network-security/privatelink/).

Select the option that works best for you.
- **Use an SSH tunnel:** If your database is running in a private network, you
can use an SSH tunnel to connect Materialize to the database.

{{< tabs >}}

{{< tab "Allow Materialize IPs">}}

1. In the [Materialize console's SQL Shell](https://console.materialize.com/) or your preferred SQL
1. In the [SQL Shell](https://console.materialize.com/) or your preferred SQL
client connected to Materialize, find the static egress IP addresses for the
Materialize region you are running in:

Expand Down Expand Up @@ -323,15 +324,15 @@ start by selecting the relevant option.

{{< tab "Use AWS PrivateLink">}}

1. In the [Materialize console's SQL Shell](https://console.materialize.com/),
1. In the [SQL Shell](https://console.materialize.com/),
or your preferred SQL client connected to Materialize, use the [`CREATE
CONNECTION`](/sql/create-connection/#aws-privatelink) command to create an
AWS PrivateLink connection:

```mzsql
CREATE CONNECTION privatelink_svc TO AWS PRIVATELINK (
SERVICE NAME 'com.amazonaws.vpce.us-east-1.vpce-svc-0356210a8a432d9e9',
AVAILABILITY ZONES ('use1-az1', 'use1-az2', 'use1-az3')
AVAILABILITY ZONES ('use1-az1', 'use1-az2', 'use1-az4')
);
```

Expand Down Expand Up @@ -428,7 +429,7 @@ details for Materialize to use:

{{< tab "Use an SSH tunnel">}}

1. In the [Materialize console's SQL Shell](https://console.materialize.com/),
1. In the [SQL Shell](https://console.materialize.com/),
or your preferred SQL client connected to Materialize, use the [`CREATE
CONNECTION`](/sql/create-connection/#ssh-tunnel) command to create an SSH
tunnel connection:
Expand Down
26 changes: 13 additions & 13 deletions doc/user/content/ingest-data/postgres/amazon-rds.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,19 +113,20 @@ to connect:
configure your database's security group to allow connections from a set of
static Materialize IP addresses.

- **Use AWS PrivateLink** or **Use an SSH tunnel:** If your database is running
in a private network, you can use either [AWS PrivateLink](https://aws.amazon.com/privatelink/)
or an SSH tunnel to connect Materialize to the database.
- **Use AWS PrivateLink**: If your database is running in a private network, you
can use [AWS PrivateLink](/ingest-data/network-security/privatelink/) to
connect Materialize to the database. For details, see [AWS PrivateLink](/ingest-data/network-security/privatelink/).

Select the option that works best for you.
- **Use an SSH tunnel:** If your database is running in a private network, you
can use an SSH tunnel to connect Materialize to the database.

{{< tabs >}}

{{< tab "Allow Materialize IPs">}}

1. In the [Materialize console's SQL Shell](https://console.materialize.com/),
or your preferred SQL client connected to Materialize, find the static egress
IP addresses for the Materialize region you are running in:
1. In the [SQL Shell](https://console.materialize.com/), or your preferred SQL
client connected to Materialize, find the static egress IP addresses for the
Materialize region you are running in:

```mzsql
SELECT * FROM mz_egress_ips;
Expand Down Expand Up @@ -316,7 +317,7 @@ start by selecting the relevant option.

{{< tab "Allow Materialize IPs">}}

1. In [Materialize console's SQL Shell](https://console.materialize.com/), or
1. In the [SQL Shell](https://console.materialize.com/), or
your preferred SQL client connected to Materialize, use the [`CREATE
SECRET`](/sql/create-secret/) command to securely store the password for the
`materialize` PostgreSQL user you created
Expand Down Expand Up @@ -373,15 +374,14 @@ start by selecting the relevant option.

{{< tab "Use AWS PrivateLink">}}

1. In the [Materialize console's SQL Shell](https://console.materialize.com/),
or your preferred SQL client connected to Materialize, use the [`CREATE
CONNECTION`](/sql/create-connection/#aws-privatelink) command to create an
AWS PrivateLink connection:
1. In the [SQL Shell](https://console.materialize.com/), or your preferred SQL
client connected to Materialize, use the [`CREATE CONNECTION`](/sql/create-connection/#aws-privatelink)
command to create an AWS PrivateLink connection:

```mzsql
CREATE CONNECTION privatelink_svc TO AWS PRIVATELINK (
SERVICE NAME 'com.amazonaws.vpce.us-east-1.vpce-svc-0356210a8a432d9e9',
AVAILABILITY ZONES ('use1-az1', 'use1-az2', 'use1-az3')
AVAILABILITY ZONES ('use1-az1', 'use1-az2', 'use1-az4')
);
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
- Replace `<host>` with your MySQL endpoint.

1. Use the [`CREATE SOURCE`](/sql/create-source/) command to connect Materialize
to your Azure instance and start ingesting data:
to your MySQL instance and start ingesting data:

```mzsql
CREATE SOURCE mz_source
Expand Down
Loading
Loading