Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatic EST certificate retrieval does not work on Ubuntu 22.04 copy to reopen issue #7152

Closed
omedirk opened this issue Nov 15, 2023 · 17 comments
Closed

Automatic EST certificate retrieval does not work on Ubuntu 22.04 copy to reopen issue #7152

omedirk opened this issue Nov 15, 2023 · 17 comments
Assignees
@vadim-kovalyov
Labels
customer-reported iotedge

Comments

@omedirk
Copy link

omedirk commented Nov 15, 2023 •
edited
Loading

Please reopen the original issue.

The root cause has been solved in openssl, but the solution is not propagated in ubuntu 22.04 libraries.

I have so far compiled 3.0.12 and 3.1.14 but they still don't solve this.

Please supply a walkthrough or any other workaround to get EST working with iotedge on ubuntu 22.04

original issue

Thanks for your help
@vadim-kovalyov
Copy link
Contributor

Hey @omedirk, if an issue was resolved in openssl and you compiled the version with the fix, double check that the process is actually using (dynamically loading) the openssl libraries you compiled and not the distribution libraries.

@vadim-kovalyov vadim-kovalyov self-assigned this Nov 16, 2023
@omedirk
Copy link
Author

omedirk commented Nov 16, 2023 via email

@omedirk
Copy link
Author

omedirk commented Nov 17, 2023 •
edited
Loading

we have tried many versions of the same installation/compile technique.
example compile tutorial

they all compile, they all link to the new openSSL but the provisioning will still not pull in the needed certificates. Will try to post a screenshot of it later.

Questions:

  • are there specific compile options needed?
  • are there links needed that are not in the manuals but solely needed for IOTedge?
  • How does microsoft test this? apparently it works for them..

optional solvers:
@gordonwang0 , @ggjjj do you have some pointers on how to get the fixes in a openssl build for ubuntu22.04? you both were in the original discussion. Thanks for any input here.

error still the same as in original post
Nov 17 09:58:56 ubuntuiot aziot-certd[24743]: 2023-11-17T09:58:56Z [ERR!] - !!! internal error Nov 17 09:58:56 ubuntuiot aziot-certd[24743]: 2023-11-17T09:58:56Z [ERR!] - !!! caused by: could not create cert Nov 17 09:58:56 ubuntuiot aziot-certd[24743]: 2023-11-17T09:58:56Z [ERR!] - !!! caused by: cert "device-id" is configured to be issued by EST, but neither EST identity nor EST bootstrap identity could be obtained: error trying to connect: error:0A080006:SSL routines:ssl_generate_param_group:EVP lib:../ssl/s3_lib.c:4735:, error:0A00013A:SSL routines:tls_process_ske_ecdhe:unable to find ecdh parameters:../ssl/statem/statem_clnt.c:2140:

@vadim-kovalyov
Copy link
Contributor

vadim-kovalyov commented Nov 20, 2023 •
edited
Loading

Hey @omedirk, sorry to hear you still having issues. Let me answer some of your questions.

We don't require any specific compile options for openssl, because, we don't required to compile any dependencies. I know it is possible that our dependencies can have problems, and we have no control over it and you would need to follow any guidance on that dependency to resolve the issue.

About openssl specifically, Iotedge is dynamically linked to the platform-specific openssl shared libs. The aziot-edge package you get from Ubuntu 22.04 repository is complied for openssl3 and expects libssl.so.3, and aziot-edge package from Ubuntu 20.04 repository is complied for openssl 1.1 and expects libssl.so.1.1. You can use ldd to check where the current shared lib is loaded from (ldd /usr/libexec/aziot-identity-service/aziotd) and then you can use different ways to force the aziotd and aziot-edged to load the version you compiled - https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html

I hope this will help you. If you still have issues, please provide detailed steps on what you have done, how and what version of iotedge you installed, what commands did you run for compiling openssl, what have you tried to make it load compiled version of shared libs, etc. Thanks!

UPD: this link could be useful as well - https://unix.stackexchange.com/questions/22926/where-do-executables-look-for-shared-objects-at-runtime

@omedirk
Copy link
Author

omedirk commented Nov 22, 2023 •
edited
Loading

Hi Vadim,

we have now most likely pointed to the azure identity service: it simply does not support ubuntu 22 and openssl 3 out of the box.
Looking in the source it's supports openssl 1.1.1 with standard options. To compile it for v3 it needs specific setting in the makefile.

this matches the behaviour that it works when co-installing openssl111.

we will now investigate this further.

@vadim-kovalyov
Copy link
Contributor

Hey @omedirk, are you trying to compile iotedge yourself? Aziot-edge supports Ubuntu 22.04 and openssl3 since version 1.4.9 - https://github.com/Azure/azure-iotedge/releases/tag/1.4.9. If you download the package from microsoft repo for Ubuntu 22.04 https://packages.microsoft.com/ubuntu/22.04/prod/pool/main/ it is already compiled for openssl3.

@omedirk
Copy link
Author

omedirk commented Nov 23, 2023 via email

@omedirk
Copy link
Author

omedirk commented Nov 23, 2023

Hey @omedirk, sorry to hear you still having issues. Let me answer some of your questions.

We don't require any specific compile options for openssl, because, we don't required to compile any dependencies. I know it is possible that our dependencies can have problems, and we have no control over it and you would need to follow any guidance on that dependency to resolve the issue.

About openssl specifically, Iotedge is dynamically linked to the platform-specific openssl shared libs. The aziot-edge package you get from Ubuntu 22.04 repository is complied for openssl3 and expects libssl.so.3, and aziot-edge package from Ubuntu 20.04 repository is complied for openssl 1.1 and expects libssl.so.1.1. You can use ldd to check where the current shared lib is loaded from (ldd /usr/libexec/aziot-identity-service/aziotd) and then you can use different ways to force the aziotd and aziot-edged to load the version you compiled - https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html

I hope this will help you. If you still have issues, please provide detailed steps on what you have done, how and what version of iotedge you installed, what commands did you run for compiling openssl, what have you tried to make it load compiled version of shared libs, etc. Thanks!

UPD: this link could be useful as well - https://unix.stackexchange.com/questions/22926/where-do-executables-look-for-shared-objects-at-runtime

Coming back to the question on how we installed openssl3.0.9

  • removed all reference to openssl (apt remove openssl libssl3 && apt autoremove)
  • downloaded openssl3.0.9
  • untar
  • configured with:
  • /Configure --prefix=/usr/local/ssl --openssldir=/usr/local/ssl '-Wl,--enable-new-dtags,-rpath,$(LIBRPATH)'
  • make, make install
  • ldconfig
  • then this trick to update links:
  • sudo tee /etc/profile.d/openssl.sh<<EOF export PATH=/usr/local/ssl/bin:\$PATH export LD_LIBRARY_PATH=/usr/local/ssl/lib:\$LD_LIBRARY_PATH EOF
  • and run it with source /etc/profile.d/openssl.sh

when running openssl version -a
OpenSSL 3.0.9 30 May 2023 (Library: OpenSSL 3.0.9 30 May 2023) built on: Thu Nov 23 13:00:02 2023 UTC platform: linux-x86_64 options: bn(64,64) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG OPENSSLDIR: "/usr/local/ssl" ENGINESDIR: "/usr/local/ssl/lib64/engines-3" MODULESDIR: "/usr/local/ssl/lib64/ossl-modules" Seeding source: os-specific CPUINFO: OPENSSL_ia32cap=0x80182201078bffff:0x1

$ ldd /usr/libexec/aziot-identity-service/aziotd

        linux-vdso.so.1 (0x00007ffcdbfe9000)
        libtss2-rc.so.0 => /lib/x86_64-linux-gnu/libtss2-rc.so.0 (0x00007fd3e293d000)
        libtss2-mu.so.0 => /lib/x86_64-linux-gnu/libtss2-mu.so.0 (0x00007fd3e28ee000)
        libtss2-esys.so.0 => /lib/x86_64-linux-gnu/libtss2-esys.so.0 (0x00007fd3e285a000)
        libtss2-tctildr.so.0 => /lib/x86_64-linux-gnu/libtss2-tctildr.so.0 (0x00007fd3e284f000)
        libaziot_keys.so => /lib/libaziot_keys.so (0x00007fd3e26d4000)
        libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007fd3e262e000)
        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007fd3e21eb000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd3e21cb000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd3e20e4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd3e1ebc000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd3e3401000)
        libtss2-sys.so.1 => /lib/x86_64-linux-gnu/libtss2-sys.so.1 (0x00007fd3e1e99000)

this is where i end my work .. apparently this is not enough to get azure identity working and i still have the same error:

Nov 23 19:19:03 ubuntuiot2204 aziot-keyd[850]: 2023-11-23T19:19:03Z [ERR!] - error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:../crypto/encode_decode/decoder_lib.c:101:No supported data to decode. Input type: PEM Nov 23 19:19:03 ubuntuiot2204 aziot-keyd[850]: 2023-11-23T19:19:03Z [ERR!] - !!! internal error Nov 23 19:19:03 ubuntuiot2204 aziot-keyd[850]: 2023-11-23T19:19:03Z [ERR!] - !!! caused by: could not load key pair Nov 23 19:19:03 ubuntuiot2204 aziot-keyd[850]: 2023-11-23T19:19:03Z [ERR!] - !!! caused by: could not load key pair: AZIOT_KEYS_RC_ERR_EXTERNAL Nov 23 19:19:03 ubuntuiot2204 aziot-keyd[850]: 2023-11-23T19:19:03Z [INFO] - --> 500 {"content-type": "application/json"} Nov 23 19:19:03 ubuntuiot2204 aziot-keyd[850]: 2023-11-23T19:19:03Z [INFO] - <-- POST /keypair?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "56"} Nov 23 19:19:03 ubuntuiot2204 aziot-keyd[850]: 2023-11-23T19:19:03Z [ERR!] - error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:../crypto/encode_decode/decoder_lib.c:101:No supported data to decode. Input type: PEM Nov 23 19:19:03 ubuntuiot2204 aziot-keyd[850]: 2023-11-23T19:19:03Z [ERR!] - !!! internal error Nov 23 19:19:03 ubuntuiot2204 aziot-keyd[850]: 2023-11-23T19:19:03Z [ERR!] - !!! caused by: could not create key pair Nov 23 19:19:03 ubuntuiot2204 aziot-keyd[850]: 2023-11-23T19:19:03Z [ERR!] - !!! caused by: could not create key pair: AZIOT_KEYS_RC_ERR_EXTERNAL

@vadim-kovalyov
Copy link
Contributor

It seems to me that you've set LD_LIBRARY_PATH for your local user but not the aziot-* services. Try setting it for all aziot-* services with systemd config.

@omedirk
Copy link
Author

omedirk commented Nov 28, 2023 via email •
edited by vadim-kovalyov
Loading

@vadim-kovalyov
Copy link
Contributor

Hey @omedirk, I think you can easily find that online - https://serverfault.com/questions/413397/how-to-set-environment-variable-in-systemd-service

But specifically, for aziot-* services we already create override file for AZIOT_LOG env var, so you can edit /etc/systemd/system/aziot-*.service.d/log-level.conf to add a new env var.

@jlian
Copy link
Member

jlian commented Jan 4, 2024

@omedirk how's it going? Were you able to make progress?

@jlian
Copy link
Member

jlian commented Jan 11, 2024

@omedirk checking again - how's it going?

@omedirk
Copy link
Author

omedirk commented Jan 13, 2024 via email

@bishal41
Copy link
Contributor

bishal41 commented Mar 26, 2024 •
edited
Loading

Hi @omedirk - we have asked our Canonical contacts to backport this OpenSSL fix and include it in Ubuntu 22.04LTS. You can now track the status of this issue via this bug.

If you don't have any questions, I will proceed and close this issue and you can always re-open if needed. Thanks!

@omedirk
Copy link
Author

omedirk commented Mar 27, 2024 via email

@bishal41
Copy link
Contributor

@thanks for the update @omedirk, let me know if you have any additional questions/concerns that we need to look at.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vadim-kovalyov vadim-kovalyov

Labels
customer-reported iotedge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@vadim-kovalyov @jlian @omedirk @bishal41