Add mTLS authN to direct connections #125
Comments
This was referenced Oct 26, 2024
rhauch
added a commit
that referenced
this issue
Oct 26, 2024
## Summary of Changes Resolves #96 This adds the _initial but limited_ support for a new type of connection resources, called `DIRECT` connections, that represent a local or remote Kafka cluster and/or Schema Registry. The extension might create a direct connection to talk to a locally running CP installation, or a remote Apache Kafka cluster with a remote Schema Registry, or even a CCloud cluster with Schema Registry. The user may have to provide credentials to successfully authenticate and connect to each remote service/cluster. The direction connections are not necessarily the constructs we want to expose. It is likely that near-term versions of the VS Code extension will allow users to connect to local or remote Kafka and SR clusters, but it may also expose to users different kinds of connections that are all represented in the sidecar as direct connections. For example, maybe the VS Code extension allows users to connect to “Confluent Platform”, “Apache Kafka”, and “WarpStream” (in addition to CCloud and local AK+SR), and these might all be customized facades on top of direct connections. This is the first step in full support for direct connections, and as such only limited functionality is available in this PR: * CRUDL connection resources with `type=DIRECT`, with validation of fields for each type of connection. * A direct connection contains zero or one `kafka_cluster` object and zero or one `schema_registry` objects, allowing it to connect to a (remote or local) Kafka cluster and/or a (remote or local) Schema Registry. * GraphQL support for the Kafka cluster and SR referenced by direct connections. Once this PR is merged, the following will handled as followups to complete the support for direct connections: - #123 - #124 - #125 - #126 - #127 After that, we will have more improvements to better support Confluent Platform: - #128 Other improvements may include: - #129 - #130 - Allow direct connections to connect to Flink cluster and Kafka Connect cluster. Also, we avoided using `javax` and instead used `jakarta` imports due to the transfer of Java EE to the Eclipse Foundation, where it became Jakarta EE. The older `javax` packages within Java EE have been deprecated. Jakarta EE uses the `jakarta` namespace, which is incompatible with the legacy javax namespace. Mixing the two can cause issues. Several projects and frameworks… have already adopted Jakarta. Additionally, upgrading to Jakarta allows you to keep up with the latest changes and modernize your applications. https://github.com/jakartaee/platform/blob/main/namespace/mappings.adoc#mapping-javax-to-jakarta ## Pull request checklist Please check if your PR fulfills the following (if applicable): - Tests: - [x] Added new - [x] Updated existing - [x] Deleted existing - [x] Have you validated this change locally against a running instance of the Quarkus dev server? ```shell make quarkus-dev ``` - [x] Have you validated this change against a locally running native executable? ```shell make mvn-package-native && ./target/ide-sidecar-*-runner ```
5 tasks
airlock-confluentinc bot
pushed a commit
that referenced
this issue
Nov 8, 2024
…nnections Resolves #124 Adds basic and API key+secret credentials to direct connections, including validating the credentials in the Connections API and using credentials when connecting to the Kafka cluster and SR defined in the direct connection spec. This also adds a new `Password` class and `ApiSecret` class that extend a new `Redactable` abstract class representing any literal String value that must be redacted in all API responses and never logged in messages (or output by the sidecar). These are essentially write-only values that prevent reads. It overrides the includes a custom serializer that always writes a _masked_ representation consisting of exactly eight asterisk (`*`) characters _regardless of the actual literal value_. The `toString()` method also outputs the same _masked_ representation, primarily to help prevent sensitive literal values from being included in logs or exception messages. (Note that these behaviors are defined in `Redactable` and marked as final to ensure subclasses do not alter the behavior.) The `Credentials` interface and `BasicCredentials` and `ApiKeyAndSecret` record types have methods that build the auth-related configuration properties for Kafka clients and SR clients. Each concrete `Credentials` type customizes the logic, though parameters are used to supply information not in the `Credentials` objects. The `ConnectionState` has helper methods to obtain the `Credentials` for a Kafka cluster with a given ID or a Schema Registry cluster with a given ID. The `DirectConnectionState` subclass always returns the credentials for the one Kafka cluster or one SR cluster. In the future, other `ConnectionState` subclasses (e.g., for CP MDS) might need to maintain a map of credentials by cluster ID for any clusters do not have the same MDS credentials (e.g., the Kafka or SR cluster does not delegate authN functionality to MDS). And modified the way Kafka admin, consumer and producer clients and the Schema Registry clients are configured, by moving the logic into a new `ClientConfigurator` bean. This encapsulates the logic of building the configuration properties for these clients, by relying on the `Credentials` methods to get only the auth-related config properties and to supply the information needed by those methods from the `KafkaCluster` or `SchemaRegistry` cluster. The `ClientConfigurator` bean’s methods have a boolean parameter as to whether any redactable values should be redacted rather than use the actual literal secrets. This is so that we can reuse the logic and expose the connection properties to the user, say to allow them to copy the connection properties and use them in their application, or if we use the generated (but redacted) connection configs in the template service. This builds in support to easily add other types of credentials (see #125, #126, #127) by defining new record types that implement the `Credentials` interface and implementing all methods.
5 tasks
rhauch
added a commit
that referenced
this issue
Nov 13, 2024
…er configs in direct connections (#152) Resolves #124 Adds basic and API key+secret credentials to direct connections, including validating the credentials in the Connections API and using credentials when connecting to the Kafka cluster and SR defined in the direct connection spec. ### New Credentials types The `Credentials` interface and `BasicCredentials` and `ApiKeyAndSecret` record types have methods that build the auth-related configuration properties for Kafka clients and SR clients. Each concrete `Credentials` type customizes the logic, though parameters are used to supply information not in the `Credentials` objects. The `Credentials` interface defines three methods that will likely be overridden by each concrete subtype: * `kafkaClientProperties(...)` -- Construct the auth-related Kafka client configuration properties. The method parameter defines connectivity options that might affect these properties. * `schemaRegistryClientProperties(...)` -- Construct the auth-related SR client configuration properties. The method parameter defines connectivity options that might affect these properties. * `httpClientHeaders(...)` -- Construct the auth-related HTTP headers. ### New Redactable types for write-only objects The `BasicCredentials` has a `password` field, and the `ApiKeyAndSecret` record type has a `api_secret` field. Because these fields will contain secrets, they must ensure that these fields are always masked (e.g., `********`) when written to the log or in API responses. To do this, this PR defines a new `Password` class and `ApiSecret` class that extend a new `Redactable` abstract class representing any literal String value that must be redacted in all API responses and never logged in messages (or output by the sidecar). These are essentially write-only values that prevent external reads. The `Redactable` class includes a custom serializer that always writes a _masked_ representation consisting of exactly eight asterisk (`*`) characters _regardless of the actual literal value_. The `toString()` method also outputs the same _masked_ representation, primarily to help prevent sensitive literal values from being included in logs or exception messages. There are also a few methods that can be used in validating, such as checking whether the value is empty or longer than some size. The `hashCode()` and `equals()` methods never use the value. All of these methods are marked as final to ensure subclasses do not alter the behavior.) ### Building Kafka and SR client configurations The logic to build the complete configurations for the Kafka admin, consumer and producer clients and the Schema Registry clients are moved into a new `ClientConfigurator` bean that is `@ApplicationScoped`. These methods rely upon the `Credentials` methods for the auth-related config properties and the `KafkaCluster` or `SchemaRegistry` cluster for the remaining configuration properties. The `ClientConfigurator` bean’s methods have a boolean parameter as to whether the resulting configuration should redact secrets, so that the configuration can be expose the connection properties to the user, say to allow them to copy the connection properties and use them in their application, or if we use the generated (but redacted) connection configs in the template service. But the `AdminClients`, `KafkaProducerClients`, `KafkaConsumerFactory` and `SchemaRegistryClients` beans use the configurator and do not redact the configuration. New methods have been added to the `ConnectionState` class to make it easy to get the `Credentials` for a Kafka cluster with a given ID or a Schema Registry cluster with a given ID. The `DirectConnectionState` subclass always returns the credentials for the one Kafka cluster or one SR cluster. In the future, other `ConnectionState` subclasses (e.g., for CP MDS) might need to maintain a map of credentials by cluster ID for any clusters do not have the same MDS credentials (e.g., the Kafka or SR cluster does not delegate authN functionality to MDS). ### Adding other types of credentials in the future In the future, the only thing we need to do to support other types of authN credentials, such as [OAuth 2.0](#125), [mTLS](#126), [Kerberos (SASL/GSSAPI)](#127), etc., is to define new `Credentials` subtypes and implement the methods to construct the auth-related client properties using the subtype-specific credential information. ### Limitations There are a few shortcuts taken for direct connections that will be addressed in subsequent PR as part of #123: * the `status` for direct connections is not accurate or useful, and will have to use an AdminClient and SR client to verify the credentials and update the status. * the `kafka_cluster.id` and `schema_registry.id` are currently optional in the OpenAPI spec but are required until we can obtain the cluster ID of the remote system and verify it matches. The `RealDirectFetcher` will need to perform a describe-cluster using the admin client, and set the cluster ID. (We might consider remove the `id` fields from the connection spec, if we always get a good ID from the describe-cluster.) ### Testing I've done some manual testing with `quarkus:dev` and native executable by using the REST API to create a direct connection that uses a CCloud cluster with API key and secret, and have verified the admin client and consumer clients are built correctly and will successfully work with the remote cluster.
airlock-confluentinc bot
pushed a commit
that referenced
this issue
Dec 7, 2024
Resolves #125 #126 Adds preliminary support to allow Kafka and SR clients to use mTLS and OAuth 2.0 to authenticate for direct connections. These changes have been unit tested to verify the expected client configurations are generated. However, those expected configurations have NOT yet been tested against CCloud or CP via integration tests.
5 tasks
airlock-confluentinc bot
pushed a commit
that referenced
this issue
Dec 10, 2024
Resolves #125 #126 Adds preliminary support to allow Kafka and SR clients to use mTLS and OAuth 2.0 to authenticate for direct connections. These changes have been unit tested to verify the expected client configurations are generated. However, those expected configurations have NOT yet been tested against CCloud or CP via integration tests.
|
Already created #237. Re-closing this. :homer-disappear: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Continuation of #96, #122, #123 and #124, and followup to #116 that added initial and limited direct connections (with GraphQL support) but did not support credentials and only worked with Kafka clusters and SR clusters that did not require authN.
We want to add mTLS support to direct connections (similar to basic creds) via the
kafka_clusterandschema_registrydefinitions, and use this to connect to downstream resources (e.g., Kafka REST API, SR API endpoints, and Message Consumer API). All secrets in credentials will be write-only: the Connection REST API endpoints must always mask secrets.We'll track adding support for other credential types in separate issues:
The text was updated successfully, but these errors were encountered: