Kubernetes no longer allows RW mounting of config maps

[philosifer]

Newer versions of K8S don't allow secrets or config maps to be mounted RW. I ran into a problem with the example dex yamls not starting because the web-templates were mounted RO not RW as expected. "kubectl describe pod" clearly showed the RO error in this case.

A workaround for this is to add an init container to copy the config map to an emptyDir store.

So in dex.yaml I added this volume
volumes:
- name: dex-web
emptyDir: {}

Then the init container

  initContainers:
    - name: copy-dex-web-templates
      image: busybox
      command: ['sh', '-c', 'cp /configmap/* /web/templates']
      volumeMounts:
        - name: dex-web-templates
          mountPath: /configmap
        - name: dex-web
          mountPath: /web/templates

and then for the dex container you would replace the 7 web-template mounts with dex-web instead of dex-web-templates. For mine I just replaced them with this one

    - mountPath: /web/templates/
      name: dex-web

That let dex start up and work correctly as it did previously.

[jsravn]

Hi @philosifer. It should work even with the k8s change. I don't think dex requires it to be rw. However I believe subpath mounts were temporarily broken in some k8s versions (1.8.8 iirc) which would cause the dex yaml to fail to work.

[philosifer]

I put the subpaths back in but still with the initcontainer intermediate step and it still works so i think its more to do with the config map than using subpaths. I'm on 1.10.2 btw.

[totahuanocotl]

What error do you get with the Config map @philosifer?
I submitted a PR for Dex itself a while ago because it has an issue resolving the templates when mounting them as volumes: dexidp/dex#1206.
It was rejected though, and I did not submit a second one. See if it the description of the issue matches what you are seeing.

[philosifer]

This is the error I get from kubectl describe pod if i don't do the initcontainer trick

Error: failed to start container "dex": Error response from daemon: error setting label on mount source '/var/lib/kubelet/pods/b7f8be8b-65a8-11e8-a1a1-005056a6113c/volume-subpaths/dex-web-templates/dex/5': read-only file system

That's repeated for a few of the other subpath mounts before it gives up and fails

I've done some more investigating. I have two clusters, development which uses default settings for security and production on which i've set it to use certificates everywhere, https for etcd etc. I only see the read-only issue on the production cluster, it works fine as you wrote it on the dev cluster but i've no idea what exactly is causing that difference. Both are built with virtually identical kubeadm config files apart from the certificate lines added to the production setup.

[jsravn]

@totahuanocotl ping

[totahuanocotl]

@philosifer, sorry for the late response.
Do you still have this issue?
If so, I'll try to replicate it and try to find the issue.
If you could provide the configuration you used that would be helpful.

[philosifer]

Yes I still have the issue but not much time to troubleshoot it further currently. It is also now affecting my attempt to redeploy harbor with helm (their issue #4496) and i might end up having to use the ReadOnlyAPIDataVolumes feature gate for now as i have too many places to put in workarounds for everything.