journald log driver doesn't work as advertised

[alitvak69]

Description

No log outputs directly to journald and fields like CONTAINER_NAME are populated. The only way to get logs in journal is via SYSLOG_IDENTIFIER

Steps to reproduce the issue

nerdctl run -d --privileged --log-driver=journald nginx
ac67ef179dd3 docker.io/library/nginx:alpine "/docker-entrypoint.…" 5 minutes ago Up nginx-ac67e
journalctl -a CONTAINER_NAME=nginx-ac67e
-- No entries --
docker logs 61a2f083b42c
^C
root@sbc21n1-mke:~# docker inspect nginx-ac67e
[
{
"Id": "ac67ef179dd34ce3146047728986c5247117d1faab3c7b2efcfbca3e39b6303d",
"Created": "2024-10-02T17:48:04.18807001Z",
"Path": "/docker-entrypoint.sh",
"Args": [
"nginx",
"-g",
"daemon off;"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"Pid": 1704668,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-10-02T17:48:04.505471873Z",
"FinishedAt": ""
},
"Image": "docker.io/library/nginx:alpine",
"ResolvConfPath": "/var/lib/nerdctl/1935db59/containers/default/ac67ef179dd34ce3146047728986c5247117d1faab3c7b2efcfbca3e39b6303d/resolv.conf",
"HostnamePath": "/var/lib/nerdctl/1935db59/containers/default/ac67ef179dd34ce3146047728986c5247117d1faab3c7b2efcfbca3e39b6303d/hostname",
"LogPath": "",
"Name": "nginx-ac67e",
"RestartCount": 0,
"Driver": "overlayfs",
"Platform": "linux",
"AppArmorProfile": "",
"Mounts": [
{
"Type": "bind",
"Source": "/var/lib/nerdctl/1935db59/containers/default/ac67ef179dd34ce3146047728986c5247117d1faab3c7b2efcfbca3e39b6303d/resolv.conf",
"Destination": "/etc/resolv.conf",
"Mode": "bind,rprivate",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/lib/nerdctl/1935db59/etchosts/default/ac67ef179dd34ce3146047728986c5247117d1faab3c7b2efcfbca3e39b6303d/hosts",
"Destination": "/etc/hosts",
"Mode": "bind,rprivate",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/lib/nerdctl/1935db59/containers/default/ac67ef179dd34ce3146047728986c5247117d1faab3c7b2efcfbca3e39b6303d/hostname",
"Destination": "/etc/hostname",
"Mode": "bind,rprivate",
"RW": true,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "ac67ef179dd3",
"AttachStdin": false,
"Labels": {
"io.containerd.image.config.stop-signal": "SIGQUIT",
"maintainer": "NGINX Docker Maintainers \[email protected]\u003e",
"nerdctl/extraHosts": "null",
"nerdctl/hostname": "ac67ef179dd3",
"nerdctl/ipc": "{"mode":"private"}",
"nerdctl/log-uri": "binary:///usr/local/bin/nerdctl?_NERDCTL_INTERNAL_LOGGING=%2Fvar%2Flib%2Fnerdctl%2F1935db59",
"nerdctl/name": "nginx-ac67e",
"nerdctl/namespace": "default",
"nerdctl/networks": "["bridge"]",
"nerdctl/platform": "linux/amd64",
"nerdctl/state-dir": "/var/lib/nerdctl/1935db59/containers/default/ac67ef179dd34ce3146047728986c5247117d1faab3c7b2efcfbca3e39b6303d"
}
},
"NetworkSettings": {
"Ports": {},
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "10.4.0.5",
"IPPrefixLen": 24,
"MacAddress": "26:1f:36:d0:89:1f",
"Networks": {
"unknown-eth0": {
"IPAddress": "10.4.0.5",
"IPPrefixLen": 24,
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "26:1f:36:d0:89:1f"
}
}
}
}
]

Describe the results you received and expected

journalctl -a CONTAINER_NAME=middle
...

Sep 15 17:41:22 sbc21n1-mke middle[38321]: Executing middle on middle
Sep 15 17:41:22 sbc21n1-mke middle[38321]: + exec /usr/local/registrator/bin/middle -c /usr/local/registrator/cfg/middle.cfg
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + set -e
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + trap stop_middle SIGTERM
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + export LD_LIBRARY_PATH=/usr/local/registrator/lib:/usr/local/lib:
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + LD_LIBRARY_PATH=/usr/local/registrator/lib:/usr/local/lib:
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + umask 2
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + chown xcast:xcast /usr/local/registrator/cfg/middle.cfg
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + chmod 644 /usr/local/registrator/cfg/middle.cfg
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + echo '199.182.132.70 middle'
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + cd /usr/local/registrator/run/middle
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + [[ -f /usr/local/registrator/bin/middle ]]
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + [[ -f /usr/local/registrator/cfg/middle.cfg ]]
Sep 17 13:43:51 sbc21n1-mke middle[149996]: Executing middle on middle
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + echo Executing middle on middle
Sep 17 13:43:51 sbc21n1-mke middle[149996]: + exec /usr/local/registrator/bin/middle -c /usr/local/registrator/cfg/middle.cfg
lines 964-1022/1022 (END)

What version of nerdctl are you using?

Client:
Version: v2.0.0-rc.2
OS/Arch: linux/amd64
Git commit: cfa7e53
buildctl:
Version: v0.15.2
GitCommit: 9e14164a1099d3e41b58fc879cbdd6f2b2edb04e

Server:
containerd:
Version: v2.0.0-rc.4
GitCommit: fa5bf66fdff44846dc475c022bf6b47197febffd
runc:
Version: 1.1.14
GitCommit: v1.1.14-0-g2c9f5602

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

Client:
Namespace: default
Debug Mode: false

Server:
Server Version: v2.0.0-rc.4
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: : systemd
Cgroup Version: : 2
Plugins:
Log: fluentd journald json-file syslog
Storage: native overlayfs
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-45-generic
Operating System: Ubuntu 24.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 48
Total Memory: 503.3GiB
Name: sbc21n1-mke.siptalk.com
ID: 5844d823-3362-49bf-b02e-225929245b00

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[haytok]

Hi, @alitvak69

In the following implementation at present, containers running by nerdctl dose not export entries containing fileds such as CONTAINER_NAME field to the journalog like containers runnging by docker cli.

• https://github.com/containerd/nerdctl/blob/main/pkg/logging/journald_logger.go#L99C1-L101C3

Therefore, to check logs output to journalog by containers running by nerdctl from journalctl, the short container id must be specified as follows.

sudo journalctl SYSLOG_IDENTIFIER=<short container id> -a

There are also a few references in command reference.

• https://github.com/containerd/nerdctl/blob/main/docs/command-reference.md#whale-blue_square-nerdctl-run

🐳 --log-driver=journald: Writes log messages to journald. The journald daemon must be running on the host machine.
🐳 --log-opt=tag=<TEMPLATE>: Specify template to set SYSLOG_IDENTIFIER value in journald logs.

My Investigation

In docker cli, as described in the following documentation, journalog can be read by running sudo journalctl CONTAINER_NAME=<container name>.

• Journald logging driver | Docker Docs

There is no CONTAINER_NAME in the journal log entries output by containers started with nerdctl, so this command will not output anything when executed.

The differences between the displayed items can be seen below.

Details

In case of Docker

Run a container exporting journal log

> d run -d --name nginx-docker --log-driver=journald nginx
a09d69e2cbb9465deb4e015f63df12cd5c6bbbe28adaa604717c2710c742ea80

> d ps
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS     NAMES
a09d69e2cbb9   nginx     "/docker-entrypoint.…"   5 seconds ago   Up 4 seconds   80/tcp    nginx-docker

Checking an entry from journal log

> sudo journalctl CONTAINER_NAME=nginx-docker -a -n 1 -o json-pretty
{
        "__CURSOR" : ""???",",
        "__REALTIME_TIMESTAMP" : "1730899615994892",
        "__MONOTONIC_TIMESTAMP" : "10815613147618",
        "_BOOT_ID" : "???",
        "_UID" : "0",
        "_GID" : "0",
        "_CAP_EFFECTIVE" : "1ffffffffff",
        "_MACHINE_ID" : "???",
        "_HOSTNAME" : "???.us-west-2.amazon.com",
        "_TRANSPORT" : "journal",
        "_SYSTEMD_SLICE" : "system.slice",
        "PRIORITY" : "3",
        "_PID" : "11031",
        "_COMM" : "dockerd",
        "_EXE" : "/usr/bin/dockerd",
        "_CMDLINE" : "/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --default-ulimit nofile=32768:65536",
        "_SYSTEMD_CGROUP" : "/system.slice/docker.service",
        "_SYSTEMD_UNIT" : "docker.service",
        "CONTAINER_LOG_EPOCH" : "716594cf0122890ba56c4640ed952acd3059b796f6085326952d7a72a68396aa",
        "CONTAINER_ID_FULL" : "a09d69e2cbb9465deb4e015f63df12cd5c6bbbe28adaa604717c2710c742ea80",
        "CONTAINER_TAG" : "a09d69e2cbb9",
        "CONTAINER_ID" : "a09d69e2cbb9",
        "CONTAINER_NAME" : "nginx-docker",
        "IMAGE_NAME" : "nginx",
        "SYSLOG_IDENTIFIER" : "a09d69e2cbb9",
        "MESSAGE" : "2024/11/06 13:26:55 [notice] 1#1: start worker process 44",
        "SYSLOG_TIMESTAMP" : "2024-11-06T13:26:55.993340486Z",
        "CONTAINER_LOG_ORDINAL" : "31",
        "_SOURCE_REALTIME_TIMESTAMP" : "1730899615993357"
}

In case of nerdctl

Run a container exporting journal log

> n run -d --name nginx-nerdctl --log-driver=journald nginx
bb7df47d27fd73426cec286ed88c5abf1443e74df637e2440d2dbca7229a84dc

> n ps
CONTAINER ID    IMAGE                             COMMAND                   CREATED          STATUS    PORTS    NAMES
bb7df47d27fd    docker.io/library/nginx:latest    "/docker-entrypoint.…"    3 seconds ago    Up                 nginx-nerdctl

Checking an entry from journal log

> sudo journalctl SYSLOG_IDENTIFIER=bb7df47d27fd -a -n 1 -o json-pretty
{
        "__CURSOR" : ""???",",
        "__REALTIME_TIMESTAMP" : "1730899940827182",
        "__MONOTONIC_TIMESTAMP" : "10815937979908",
        "_BOOT_ID" : "???",
        "_UID" : "0",
        "_GID" : "0",
        "_CAP_EFFECTIVE" : "1ffffffffff",
        "_MACHINE_ID" : "???",
        "_HOSTNAME" : "???.us-west-2.amazon.com",
        "_TRANSPORT" : "journal",
        "_SYSTEMD_SLICE" : "system.slice",
        "PRIORITY" : "3",
        "_SYSTEMD_CGROUP" : "/system.slice/containerd.service",
        "_SYSTEMD_UNIT" : "containerd.service",
        "_COMM" : "nerdctl",
        "_EXE" : "/usr/local/bin/nerdctl",
        "_CMDLINE" : "/usr/local/bin/nerdctl _NERDCTL_INTERNAL_LOGGING /var/lib/nerdctl/1935db59",
        "SYSLOG_IDENTIFIER" : "bb7df47d27fd",
        "_PID" : "8118",
        "MESSAGE" : "2024/11/06 13:32:20 [notice] 1#1: start worker process 44",
        "_SOURCE_REALTIME_TIMESTAMP" : "1730899940825905"
}

[haytok]

Hi, @AkihiroSuda

Can I create a PR that also outputs CONTAINER_NAME in the journal log in nerdctl, like docker cli, if necessary?

[AkihiroSuda]

Yes, thanks, it should just correspond to Docker's behavior

[apostasie]

Was this fixed by @haytok with #3667?

[haytok]

@apostasie , Yes, this issue was resolved #3667 .