“unable to create provider: oidc: issuer did not match the issuer returned by provider” with keycloak idp

[ptrovatelli]

Describe the bug
When configuring a jwt_auth_backend in vault with a provider that vault reaches through a different URL than the issuer URL, vault fails with unable to create provider: oidc: issuer did not match the issuer returned by provider

To Reproduce
When configuring vault with keycloak IPD so that keycloak authenticates vault users through oidc, we have a problem with a verification that vault makes on the issuer field.
We have a specific setup with keycloak:

• Vault reaches keycloak through a private network (backend url)
• end users reach keycloak from a public network (frontend url)

With this setup, keycloak considers the issuer as a frontend element: in the .well-known/openid-configuration discovery URL, the issuer field is https://<frontend-keycloak-hostname>/auth/realms/myrealm

Upon initializing the oidc configuration (when building the auth backend), vault makes a verification about the issuer that makes the process fail : it seems that vault is trying to guess what the issuer should be by concatenating

• the hostname vault is using to reach keycloak : https://<backend-keycloak-hostname>
• with the path configured in the oidc_discovery_url: /auth/realms/myrealm

We can see in vault logs:

2024-01-16T11:57:12.474+0100 [ERROR] auth.oidc.auth_oidc_xxxx: error checking oidc discovery URL: error="error creating provider with given values: NewProvider: 
	unable to create provider: oidc: issuer did not match the issuer returned by provider, 
	expected \"https://<backend-keycloak-hostname>/auth/realms/myrealm\"
	     got \"https://<frontend-keycloak-hostname>/auth/realms/myrealm\""

The assumption that the issuer must have the same hostname as the oidc_discovery_url address is incorrect when the idp can be reached from different URLS.

The error appears even when bound_issuer is configured in the auth backend. Vault could simply compare the value of bound_issuer with the issuer given by the discovery URL in that case rather than assuming what the issuer hostname should be.

Setup:
vault (terraform)

resource "vault_jwt_auth_backend" "keycloak" {
  description        = "Keycloak"
  path               = "oidc"
  type               = "oidc"
  oidc_discovery_url = "https://<backend-keycloak-hostname>/auth/realms/myrealm"
  bound_issuer       = "https://<frontend-keycloak-hostname>/auth/realms/myrealm"
  oidc_client_id     = var.keycloak_client_id
  oidc_client_secret = var.keycloak_client_secret
  default_role       = "default-keycloak"
  tune {
    default_lease_ttl            = "2h"
    max_lease_ttl                = "8h"
    allowed_response_headers     = []
    audit_non_hmac_request_keys  = []
    audit_non_hmac_response_keys = []
    listing_visibility           = "unauth"
    passthrough_request_headers  = []
    token_type                   = "default-service"
  }
}

resource "vault_jwt_auth_backend_role" "default_keycloak" {
  backend        = vault_jwt_auth_backend.keycloak.path
  role_name      = "default-keycloak"
  token_policies = ["default"]

  user_claim            = "xxx"
  groups_claim          = "groups"
  role_type             = "oidc"
  allowed_redirect_uris = var.oidc_authorized_redirects
  verbose_oidc_logging  = true
}

keycloak:

• KC_HOSTNAME_STRICT=false : backend urls are built based on the request
• KC_HOSTNAME_URL=https://<frontend-keycloak-hostname>/auth : The front-end url is forced in keycloak with so that end users are not redirected to the backend URL

With this setup, accessing to https://<backend-keycloak-hostname>/auth/realm/myrealm/.well-known/openid-configuration displays the issuer as https://<frontend-keycloak-hostname>/auth/realm/myrealm

Issue could be circumvented by making vault access keycloak through the public URL https://<frontend-keycloak-hostname> but it’s not what we want to do from a network point of view.

Another solution would be to not use the discovery URL but jwks_url instead but that doesn't seem to be working at the moment: see

• "'oidc_discovery_url' must be set for OIDC" when trying to use jwks_url vault-plugin-auth-jwt#272
• OIDC should support Authorization,Token and Userinfo endpoints if well-known configuration endpoint is not present on provider vault-plugin-auth-jwt#169 (comment)

FYI related discussions on keycloak side:

• New hostname config UX · Issue #24252 · keycloak/keycloak · GitHub 1
• https://groups.google.com/g/keycloak-user/c/xot6XrGCM8k

Expected behavior
Vault should be able to log users authenticated through keycloak using oidc

Environment:

• hashicorp vault 1.15.4+ent
• keycloak 23.0.4

Additional context

[phmcder]

I've just hit the same problem as well when deploying to docker desktop. I can't connect to keycloak using the public host name as that keeps getting resolved to 127.0.0.1. As above, I set the oidc_discovery_url to the internal host name but I get the miss match issuer error message

[ptrovatelli]

quick update on this issue. i'm in contact with hashicorp support. What we've established so far:

• this verification is done by the Go-oidc library that vault uses and is described in the oidc specification https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderConfigurationValidation
  The issuer value returned MUST be identical to the Issuer URL that was used as the prefix to /.well-known/openid-configuration to retrieve the configuration information
• However this part of the specification is known to be problematic and the Go-oidc library has a bypass ready to use for this case. This bypass is used by Azure.
• we're about to open a feature request so that vault implements this bypass too
• this project replicates the issue: https://github.com/ptrovatelli/keycloak-vault/tree/issuer-did-not-match-the-issuer-returned-by-provider

[Aladex]

I encountered an issue where Vault needed to resolve the Keycloak domain (keycloak.example.com) to an internal IP, but the domain points to an external IP. Here's how I solved it:

1. Using hostAliases:
   I added a hostAliases entry on one of the nodes. Since my NGINX ingress is running as a DaemonSet with hostNetwork: true, this allowed Vault to resolve keycloak.example.com to the internal IP.

2. Alternative solution with a Service:
   Another approach is to create a Service that either points directly to Keycloak or to the NGINX ingress. By assigning a static IP to this Service, you can reference it from Vault and avoid exposing NGINX externally, while still ensuring that traffic is routed to Keycloak internally.

[heatherezell]

quick update on this issue. i'm in contact with hashicorp support. What we've established so far:

• this verification is done by the Go-oidc library that vault uses and is described in the oidc specification openid.net/specs/openid-connect-discovery-1_0.html#ProviderConfigurationValidation
  The issuer value returned MUST be identical to the Issuer URL that was used as the prefix to /.well-known/openid-configuration to retrieve the configuration information
• However this part of the specification is known to be problematic and the Go-oidc library has a bypass ready to use for this case. This bypass is used by Azure.
• we're about to open a feature request so that vault implements this bypass too
• this project replicates the issue: ptrovatelli/keycloak-vault@issuer-did-not-match-the-issuer-returned-by-provider

Were you able to get support on this? Would you be willing to share the ticket ID with me? You can email it to me, my full name at hashicorp dot com. Thanks!

[ptrovatelli]

I encountered an issue where Vault needed to resolve the Keycloak domain (keycloak.example.com) to an internal IP, but the domain points to an external IP. Here's how I solved it:

1. Using hostAliases:
   I added a hostAliases entry on one of the nodes. Since my NGINX ingress is running as a DaemonSet with hostNetwork: true, this allowed Vault to resolve keycloak.example.com to the internal IP.
2. Alternative solution with a Service:
   Another approach is to create a Service that either points directly to Keycloak or to the NGINX ingress. By assigning a static IP to this Service, you can reference it from Vault and avoid exposing NGINX externally, while still ensuring that traffic is routed to Keycloak internally.

It is a solution indeed thanks for your input. We are deployed on VM so we could simply override the DNS resolution through /etc/hosts. However, if the keycloak IP changes then we need to think of changing this configuration too. It is not the ideal solution

[ptrovatelli]

Were you able to get support on this? Would you be willing to share the ticket ID with me? You can email it to me, my full name at hashicorp dot com. Thanks!

Still waiting for support on this... Sent the ticket id to you by email, thanks for your interest in this issue.