Unable to get Proxmox list widget to work

[ImChet]

Discussed in #1315

^{Originally posted by ImChet September 14, 2023}
Error:

An error occurred, see the logs for more info.

Unable to fetch data

Here is the code block for my proxmox-list:

sections:
  - name: Server Management
    icon: ''
    items:
      - title: xxxxxxxxxx
        description: ''
        icon: hl-proxmox
        url: xxxxxxxxxx
        target: newtab
        id: xxxxxxxx
      - title: xxxxxxxxx
        icon: hl-proxmox
        url: xxxxxxxxxxx
        target: newtab
        id: xxxxxxxx
      - title: xxxxxxxx
        icon: hl-truenas
        url: xxxxxxxxxx
        target: newtab
        id: xxxxxxx
    displayData:
      sortBy: default
      rows: 1
      cols: 1
      collapsed: false
      hideForGuests: true
    widgets:
    - type: proxmox-lists
      useProxy: true
      options:
        cluster_url: https://x.x.x.x:xxxx
        user_name: root@pam
        token_name: dashy
        token_uuid: xxxxxxxxxxxxxxxxxxxxxxxx
```</div>

[liss-bot]

If you're enjoying Dashy, consider dropping us a ⭐
_{🤖 I'm a bot, and this message was automated}

[ilium007]

Searched and found this issue before creating the same. No VM data is listed. If I put an intentionally wrong value in for node the widget throws an error so I can only assume that with the correct value and no error on screen it is gettign data back from the API but not parsing it correctly.

[ilium007]

This is the JSON from the constructed API call in the browser:

{"data":[{"mem":7728077790,"diskread":0,"maxmem":8589934592,"cpu":0.0041496411882829,"status":"running","diskwrite":0,"pid":2374150,"disk":0,"name":"omv01","netin":0,"vmid":103,"cpus":2,"netout":0,"uptime":22400,"maxdisk":34359738368},{"netin":0,"disk":0,"name":"debian01","diskwrite":0,"status":"running","pid":791159,"cpu":0.0117877760698244,"maxdisk":34359738368,"netout":0,"cpus":4,"uptime":364263,"vmid":101,"mem":5502137380,"maxmem":8589934592,"diskread":0},{"mem":0,"maxmem":4294967296,"diskread":0,"name":"windows11","disk":0,"netin":0,"cpu":0,"status":"stopped","diskwrite":0,"uptime":0,"cpus":2,"netout":0,"maxdisk":34359738368,"vmid":102}]}

[nortuzar]

Same problem here, fresh install of dashy and couldnt make this widget work.
Anyway im enjoying dashy very much

[wnaude]

Hi, I also struggled at first, but then I changed the permissions of the API token in the datacenter web gui. Unticked the privilege separation., then the "show permissions" section got populated where it was empty before. I am a noob, so I must still go rtfm on the detail of what it means and what the risks are.
Just dropping the note incase it might help someone.

[lumper5]

No luck here getting this widget to connect. I had hopes that the "privilege separation" would work (it was the solution to a similar problem in homepage and Heimdall). I wonder what the trick is to get this to authenticate?

[Lissy93]

I don't think I'm going to be much help here, as I didn't build the Proxmox widget, and I don't have a Proxmox instance to test with. But what does the error message in the browser logs say? (here's how to check)

[CrazyWolf13]

@alayham Could you take over here?

[alayham]

Gladly. I have a video that shows how to install a development environment for Dashy and install the widget to fix a bug in it. I also added documentation to handle the common errors I faced with the widget.
please look at my video here:
https://www.youtube.com/watch?v=6H9j2OFr69U&list=PLZLlJrtYGHany2Mvz-Q59SPeWp88iESU1&index=8

The documentation for the errors is in the repo, but not visible on the documentation site.
https://github.com/Lissy93/dashy/blob/master/docs/widgets.md#proxmox-lists

Suggestions for users:

1. Make sure you use Dashy in production mode.
2. Make sure PVE certificate is trusted by nodejs on the host that runs Dashy.

[alayham]

@ilium007 @ImChet @nortuzar @wnaude
Were you able to resolve the issue with the information above?

[CrazyWolf13]

Hi @alayham Sadly I could not resolve the issue with the info provided above and in the docs.

Sadly the part how to extract the CA is compledtely missing, did you forget that?!

By manually clicking "view" on the root ca I could see the CA key, I copy pasted it to a file and did the export command with my path, sadly even after a reboot and rebuild of dashy I'm still getting the error code: axios request failed with status code 500

Is it really neccessary for everyone to extract that certificate?

[alayham]

Yes, It is necessary unless the Proxmox VE uses a trusted certificate. The reason is that Nodejs has a list of trusted certificates authority compiled inside the binary, and will only accept them for security. We have the ability to add to them using the following process:
1- Visit the certificates page of your PVE, you should see at lease two certificates, one ends with -ca.pem, click on it.
2- click on View Certificate
3- click on Raw certificate

4- Copy everything in the text area below View Certificate and paste it into a file called pve-root-ca.pem
5- Move the file to your Dashy host.
Either:
6- start dashy with NODE_EXTRA_CA_CERTS=<path of the copied file> yarn start
or
6- A a line to the file .bash_profile in the home directory of the user that run dashy containing: export NODE_EXTRA_CA_CERTS=<path of the copied file> and restart the shell session for the export to be active, then start Dashy using yarn start
You can watch my video about setting up Dashy using Ansible, or read this article about the subject.

[CrazyWolf13]

@alayham Thanks for the guide!
I set everything up according to your guide above, but I cannot seem to get it working.

Does Dashy have to have a certificate and run with https /ssl ?
Because currently mine runs in http mode.

Also is it okay to use the IP instead of a domain name for the cluster url?

Here a little screenshot.

My config:

  - type: proxmox-lists
    useProxy: true
    options:
      cluster_url: https://10.50.20.10:8006
      user_name: root@pam
      token_name: dashy
      token_uuid: {token}

Thanks.-

[alayham]

Timeout is usually a firewall issue. Can you access the cluster from the Dashy host using curl? try
curl https://10.50.20.10:8006 from the dashy host. Install curl if missing.

[CrazyWolf13]

Timeout is usually a firewall issue. Can you access the cluster from the Dashy host using curl? try
curl https://10.50.20.10:8006 from the dashy host. Install curl if missing.

Seems to work just fine when pinging and curl works if I add -k (think that was to ignore self signed cert) curl -k https://10.50.20.10:8006

[alayham]

Troubleshooting integrations is tricky, and to help you I need to identify the root cause of the issue. Let's try first to identify whether the cause is in the Dashy host, the PVE host, or the network.

Since you need to use -k, this means that the PVE CA certificate is not trusted on the Dashy host. Can you try to trust it? the steps to trust a ca certificate depend on the operating system and the browser. I can advise you if needed but I need to know the OS on the dashy host.

After trusting the PVE root certificate, we can do the following on the dashy host, not on your machine. It is important to run the command from the Dashy host using the Dashy user, not root.

replace YOURAPITOKEN with your token in the following command after dashy= and before the single quote, then run run the command on the dashy host and put the response here.

curl  -H 'Authorization: PVEAPIToken=root@pam!dashy=YOURAPITOKEN' https://10.50.20.10:8006/api2/json/nodes/

[CrazyWolf13]

replace YOURAPITOKEN with your token in the following command after dashy= and before the single quote, then run run the command on the dashy host and put the response here.

Hi, Thank you!

Yes as stated above, dashy is not running in SSL mode and Proxmox currently only has it's self signed cert.

Host of Dashy is a raspberrypi running raspbian.

Here the result which seems that you guess with the cert not being trusted is true;

pi@raspberrypi:~ $ curl  -H 'Authorization: PVEAPIToken=root@pam!dashy=MYAPITOKEN' https://10.50.20.10:8006/api2/json/nodes/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Again when using-k switch it seems to output the data, is there any way to get such a -k option in the widget as well?
Maybe it would also be worth to include all those steps you describe in the docs, as I'm pretty sure others will run into this as well :)

[alayham]

Aha, This is very good news.
Now we know the error. This means the CA certificate works, but you are calling the server with a domain name that does not exist in the Subject Alternative Names field of your server certificate. I do not have this issue because I use my own certificate authority and I run my own DNS service in my devlab.
Here is an example of what I have in my server:

You can see this if you inspect the certificate, and you can see it also in the PVE web admin when you click on the node, and select certificates:

To solve your issue, you need to:
1- Determine the Subject Alternative names of your server's certificate. You can do that by either inspecting the server certificate, or checking the certificates page in your PVE.
2. Make sure those DNS names resolve to the server IP in your network: You can try passing those names to the nslookup command, and the command should return the server IP.
3. Make sure those DNS names resolve to the server IP inside the Docker container. Here you might need to replace the IP with a name that resolves in the previous curl command.
4. Update your widget in the Dashy conf to use any name in the SAN field that resolves to the server IP, rebuild Dashy, and let me know if you get an error.

[CrazyWolf13]

Okay so a little update on this I dug a lot deeper into this and found out, due to me once changing from a simple ip to 10x network the cert had the old ip included, so I regenerated them to official proxmox commands. So I thought I finally finally solved it, however I still get the same exact error.

Here the alternative names:

So I cannot use the direct ip 10.50.20.10 ?

[alayham]

1- Try to use lenovo1 or lenovo1.home instead of the IP in the conf file
2- Try to edit /etc/hosts inside the container, and add the something like:

10.50.20.10     lenovo1
10.50.20.10     lenovo1.home

then rebuild Dashy and try again

[CrazyWolf13]

Okay, edited the hosts file, ping works.
Traceroute resolves perfectly fine.

When trying to execute the same curl command or with lenovo1 or lenovo1.home i just get straight up no output to cli at all.

Even disabled the proxmox firewall compledtely, still no output to curl at all. Visiting the url in browser on raspberrypi works just fine.

[alayham]

There is a huge difference between no output (got an empty response) and no output (timeout, or no response at all)

Which one do you get?

Did you test the widget?

[CrazyWolf13]

There is a huge difference between no output (got an empty response) and no output (timeout, or no response at all)

Which one do you get?

Did you test the widget?
Not sure.

But curl with debug seems to show promising results.
Also the widget does obv. not yet work, as the import did not work.

curl  --cacert /app/public/proxmox_ca.pem  -v -H 'Authorization: PVEAPIToken=root@pam!dashy={api_key}' https://10.50.20.10:8006/api2/json/nodes/
*   Trying 10.50.20.10:8006...
* Connected to 10.50.20.10 (10.50.20.10) port 8006
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /app/public/proxmox_ca.pem
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=lenovo1.home
*  start date: Mar 27 20:22:17 2024 GMT
*  expire date: Mar 27 20:22:17 2026 GMT
*  subjectAltName: host "10.50.20.10" matched cert's IP address!
*  issuer: CN=Proxmox Virtual Environment; OU={some_key}; O=PVE Cluster Manager CA
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
> GET /api2/json/nodes/ HTTP/1.1
> Host: 10.50.20.10:8006
> User-Agent: curl/8.5.0
> Accept: */*
> Authorization: PVEAPIToken=root@pam!dashy={api_key}
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 401 invalid token value!
< Cache-Control: max-age=0
< Connection: close
< Date: Fri, 29 Mar 2024 17:59:43 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Expires: Fri, 29 Mar 2024 17:59:43 GMT
<
* Closing connection
* TLSv1.3 (OUT), TLS alert, close notify (256):

[CrazyWolf13]

Edit: After regenerating the api_key on proxmox it now outputs the api data. Probably due to ssl cert change.

Surprisingly curl even outputs the data when not specifiying the cert file, so the import seems also successful.

However dashy is not working, e.g. still the same error.

[alayham]

< HTTP/1.1 401 invalid token value!

Where did you run the command? Did you open a shell session inside the Dashy container?

Surprisingly curl even outputs the data when not specifiying the cert file, so the import seems also successful.

Because the CA cert is trusted on the host that you ran the command on. Is it the Dashy container or the PI?

However dashy is not working, e.g. still the same error.

Can you paste the widget code in your conf, and the output of this command set | grep NODE_EXTRA_CA_CERTS inside the Dashy container.

[CrazyWolf13]

Where did you run the command? Did you open a shell session inside the Dashy container?

This is probably only because the api-key was outdated as described in the edit.
Here the output again:

*   Trying 10.50.20.10:8006...
* Connected to 10.50.20.10 (10.50.20.10) port 8006
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=lenovo1.home
*  start date: Mar 27 20:22:17 2024 GMT
*  expire date: Mar 27 20:22:17 2026 GMT
*  subjectAltName: host "10.50.20.10" matched cert's IP address!
*  issuer: CN=Proxmox Virtual Environment; OU={some_key}; O=PVE Cluster Manager CA
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
> GET /api2/json/nodes/ HTTP/1.1
> Host: 10.50.20.10:8006
> User-Agent: curl/8.5.0
> Accept: */*
> Authorization: PVEAPIToken=root@pam!dashy=`{api_key}
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 200 OK
< Cache-Control: max-age=0
< Connection: Keep-Alive
< Date: Fri, 29 Mar 2024 20:19:40 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Content-Length: 604
< Content-Type: application/json;charset=UTF-8
< Expires: Fri, 29 Mar 2024 20:19:40 GMT
<
* Connection #0 to host 10.50.20.10 left intact
{"data":[{"id":"node/

Because the CA cert is trusted on the host that you ran the command on. Is it the Dashy container or the PI?

I run it on the docker container so dashy host.

Can you paste the widget code in your conf, and the output of this command set | grep NODE_EXTRA_CA_CERTS inside the Dashy container.

Output:

/app # set | grep NODE_EXTRA_CA_CERTS
NODE_EXTRA_CA_CERTS='/app/public/proxmox_ca.pem'

Here the widget code:

      - type: proxmox-lists
        useProxy: true
        options:
          cluster_url: https://10.50.20.10:8006
          user_name: root@pam
          token_name: dashy
          token_uuid: {api_key}
        id: 1_1782_proxmoxlists

[alayham]

You are not following my advice!!

Here is what you I suggest you do again:
1- Determine the Subject Alternative names of your server's certificate. You can do that by either inspecting the server certificate, or checking the certificates page in your PVE.
2. Make sure those DNS names resolve to the server IP in your network: You can try passing those names to the nslookup command, and the command should return the server IP.
3. Make sure those DNS names resolve to the server IP inside the Docker container. Here you might need to replace the IP with a name that resolves in the previous curl command.
4. Update your widget in the Dashy conf to use any name in the SAN field that resolves to the server IP, rebuild Dashy, and let me know if you get an error.

You followed item 1 only which lead you to the certificate issue, which is what I initially suspected. Now that was fixed, and I assume the certificate you mounted in /app/public/proxmox_ca.pem is the new certificate. Now you need to make DNS resolution work for you using the /etc/hosts file, and to use the DNS name instead of the IP in the widget.

[CrazyWolf13]

You are not following my advice!!

Here is what you I suggest you do again: 1- Determine the Subject Alternative names of your server's certificate. You can do that by either inspecting the server certificate, or checking the certificates page in your PVE. 2. Make sure those DNS names resolve to the server IP in your network: You can try passing those names to the nslookup command, and the command should return the server IP. 3. Make sure those DNS names resolve to the server IP inside the Docker container. Here you might need to replace the IP with a name that resolves in the previous curl command. 4. Update your widget in the Dashy conf to use any name in the SAN field that resolves to the server IP, rebuild Dashy, and let me know if you get an error.

You followed item 1 only which lead you to the certificate issue, which is what I initially suspected. Now that was fixed, and I assume the certificate you mounted in /app/public/proxmox_ca.pem is the new certificate. Now you need to make DNS resolution work for you using the /etc/hosts file, and to use the DNS name instead of the IP in the widget.

Apologies for skipping this, I tought as the ip was listed as alternative name I could use that as well.

Here the requested steps:

Made sure that dns name resolve on the host:

Made sure dns resolves inside the container:

Updated the dashy conf:

Still not working, when disabling the cors proxy:
The following:

If I enable the cors proxy:

Thank you for your time so far!

[alayham]

It seems that Dashy is still unable to resolve the name although you are able to resolve it on the shell.

Anyway, I think this is a network issue and not related to Dashy or the widget. I am not familiar with your setup therefore my ability to help you is very limited.

The lesson from this issue is that the proxy should not crash if it gets an error. It should return the error message instead of the 500 crash, and that is a potential ticket for Dashy if someone is interested.

[CrazyWolf13]

@ilium007 @ImChet @nortuzar @wnaude
Were any of you able to get the widget to work?

[CrazyWolf13]

@alayham
Thanks for this awesome widget, after the recent changes, that headers in widgets do work again (#1549) , my widget is working and looks awesome!

[CrazyWolf13]

As I got no Feedback from the others I'll close this ticket.

[CrazyWolf13]

@alayham

Sadly I have to reopen this.

The Widget seems to work for me -> It shows the noes all fine.
However when adding the following:

      - type: proxmox-lists
        useProxy: true
        options:
          cluster_url: https://lenovo1.{domain}.dev
          user_name: root@pam
          token_name: {token_name}
          token_uuid: {token}
        id: 1_688_proxmoxlists

      - type: proxmox-lists
        useProxy: true
        options:
          cluster_url: https://lenovo1.{domain}.dev
          user_name: root@pam
          token_name: {token_name}
          token_uuid: {token}
          node: lenovo1
          node_data: lxc
        id: 1_688_proxmoxlists

I get a empty widget with no errors in the browser console nor any error in networks tab, the CORS-Proxy request is green and the auth headers were present.

The Network tab shows the following response:
{"data":[]}

Have you got any idea here?
Did I miss out anything?

[alayham]

The response you got means that the connection to the API was successful, Authentication was successful, but the result of your query is empty.
This can be either an authorization problem, for example, the token user is not authorized to get the data, or if the authorization is successful, the data for your query is empty.
In your case, I think the data is empty, most probably because the node name is wrong.

[CrazyWolf13]

Thank you for the response!

Actually disabling Priviledge Seperation did the trick for me.
Maybe I'd be a good idea to include this into the docs, as it defaults to on.

Btw. Amazing widget you made, really like it!