-
Notifications
You must be signed in to change notification settings - Fork 632
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add fields such as CONTAINER_NAME to journald log entries sent to by containers #3667
Conversation
The following PR will resolve CI faillures of |
The following error in CI is related to this modification, so I'll investigate to resolve.
|
After my investigation, I checked the docker cli doc and found no description of the details output in the
Details
In case of Docker
In case of nerdctl in this modification.
Therefore, the |
CI passed except for the lint issue which will be fixed in PR 3666. |
Could you rebase with the current main branch? |
This pull request will add the fields such as `CONTAINER_NAME` and `IMAGE_NAME` to the journald log entries sent by containers when `nerdctl run` is run with `--log-driver=journald`. However, the following `import cycle not allowed` error occurs when trying to import `containerutil package` in `logging package` in the implementation to be changed in this pull request. ``` > make CGO_ENABLED=0 GOOS=linux go -C /local/home/haytok/workspace/nerdctl build -ldflags "-s -w -X github.com/containerd/nerdctl/v2/pkg/version.Version=0d7dc8ec.m -X github.com/containerd/nerdctl/v2/pkg/version.Revision=0d7dc8ec4cda815acfca165b0281e801c4c5ef6e.m" -o /local/home/haytok/workspace/nerdctl/_output/nerdctl ./cmd/nerdctl package github.com/containerd/nerdctl/v2/cmd/nerdctl ... imports github.com/containerd/nerdctl/v2/pkg/infoutil: import cycle not allowed make: *** [nerdctl] Error 1 ``` Therefore, this commit refactors `infoutil package` to avoid `import cycle not allowd` error in the next commit. Signed-off-by: Hayato Kiwata <[email protected]>
Thanks, I forgot to rebase ... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
BTW, I'm not sure whether we should let people config the log field they want
a9df3e5
to
67f0a10
Compare
Hi @fahedouch |
pkg/logging/journald_logger.go
Outdated
|
||
func GetJournaldImageNameField(image string) string { | ||
imageName := image | ||
if repo, tag := imgutil.ParseRepoTag(imageName); tag == "latest" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is it disturbing to to have a latest
tag ?
I would rather check for empty tag using this func
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fahedouch, Thanks!!!
I have not considered dangling image ...
Considering the behaviour in Docker, I think the name that should be specified in IMAGE_NAME
can be categorized into the following 3 types.
- dangling Image
- tag of Image is
latest
- tag of Image isn't
latest
(ex.3.13-org
)
The following is the result of checking each case with Docker.
Details
In case of a dangling image
> CONTAINER_NAME="none-image"
> NONE_IMAGE_ID="3b25b682ea82"
> d images | grep $NONE_IMAGE_ID
<none> <none> 3b25b682ea82 7 weeks ago 192MB
> d run --rm --name $CONTAINER_NAME --log-driver=journald $NONE_IMAGE_ID sh -euxc "echo foo; echo bar"
+ echo foo
+ echo bar
foo
bar
> sudo journalctl -a CONTAINER_NAME=$CONTAINER_NAME -n 1 -o json-pretty | grep $NONE_IMAGE_ID
"IMAGE_NAME" : "3b25b682ea82",
In case of an image which has latest
tag.
> CONTAINER_NAME="latest-image"
> LATEST_IMAGE="debian"
> d images | grep $LATEST_IMAGE
debian latest 617f2e89852e 5 weeks ago 117MB
> d run --rm --name $CONTAINER_NAME --log-driver=journald $LATEST_IMAGE sh -euxc "echo foo; echo bar"
+ echo foo
+ echo bar
foo
bar
> sudo journalctl -a CONTAINER_NAME=$CONTAINER_NAME -n 1 -o json-pretty | grep $LATEST_IMAGE
"IMAGE_NAME" : "debian",
In case of an image which dosen't have latest tag.
> CONTAINER_NAME="tag-image"
> TAG_IMAGE="ghcr.io/stargz-containers/alpine:3.13-org"
> d images | grep ghcr.io/stargz-containers/alpine
ghcr.io/stargz-containers/alpine 3.13-org 49f356fa4513 3 years ago 5.61MB
> d run --rm --name $CONTAINER_NAME --log-driver=journald $TAG_IMAGE sh -euxc "echo foo; echo bar"
+ echo foo
+ echo bar
foo
bar
> sudo journalctl -a CONTAINER_NAME=$CONTAINER_NAME -n 1 -o json-pretty | grep $TAG_IMAGE
"IMAGE_NAME" : "ghcr.io/stargz-containers/alpine:3.13-org",
Therefore, I think it is necessary to distinguish between the cases where the dangling image
/ tag is latest
/ tag is other than latest
, but what do you think?
The current fix does not take into account the case of dangling image, so I'll fix that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, I'll add a test for the case where tag is latest.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for your feedback @haytok, LGTM!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the comments!!! @fahedouch
When implementing tests, I noticed that Docker displays the image name specified
when running a container in the IMAGE_NAME field ...
For example, we can check this behavior when running a container using the following image (ghcr.io/containerd/busybox:latest
).
Details
> CONTAINER_NAME="latest-image"
> LATEST_IMAGE="ghcr.io/containerd/busybox:latest"
> d images | grep ghcr.io/containerd/busybox
ghcr.io/containerd/busybox latest 16ea53ea7c65 3 years ago 1.24MB
> d run --rm --name $CONTAINER_NAME --log-driver=journald $LATEST_IMAGE sh -euxc "echo foo; echo bar"
+ echo foo
+ echo bar
foo
bar
> sudo journalctl -a CONTAINER_NAME=$CONTAINER_NAME -n 1 -o json-pretty | grep $LATEST_IMAGE
"IMAGE_NAME" : "ghcr.io/containerd/busybox:latest",
This behavior makes sense in the three cases above.
Therefore, I think it's not necessary to split the process according to the tag of image, so I'll fix the commit to delete GetJournaldImageNameField()
.
pkg/logging/journald_logger.go
Outdated
"CONTAINER_ID": shortID, | ||
"CONTAINER_ID_FULL": containerID, | ||
"CONTAINER_NAME": containerutil.GetContainerName(containerLabels), | ||
"IMAGE_NAME": journaldLogger.RowImageRef, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it possible to fetch image ref from labels too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fahedouch Thanks !!!
I think it is not possible to fetch row image ref (journaldLogger.RawImageRef
) from the following Labels()
, so I added journaldLogger.RawImageRef
.
- https://github.com/containerd/containerd/blob/main/client/image.go#L48
- https://github.com/containerd/containerd/blob/main/client/container.go#L79
If I misunderstood your question, could you please tell me what image ref
and labels
are 🙇🙇
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it possible to fetch the image ref
from container.info(ctx)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you, the image ref can be fetched from container.Info(ctx)
, so I'll fix it.
The following test has failed. I'm checking ... Details
2024-11-25T06:17:46.8588148Z === RUN TestRemove/Remove_image_with_paused_container_-_with_-f
2024-11-25T06:17:46.8589708Z command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 info --format {{ json . }}
2024-11-25T06:17:46.8591338Z image_remove_test.go:306: ======================== Pre-test cleanup ========================
2024-11-25T06:17:46.8593845Z command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 rm -f testremove-remove-image-with-paused-container-with-f-41f5f718
2024-11-25T06:17:46.8595890Z command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 rmi -f
2024-11-25T06:17:46.8597114Z image_remove_test.go:306: ======================== Test setup ========================
2024-11-25T06:17:46.8600240Z command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 run --pull always -d --name testremove-remove-image-with-paused-container-with-f-41f5f718 ghcr.io/stargz-containers/alpine:3.13-org sleep 3600
2024-11-25T06:17:46.8603386Z command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 pause testremove-remove-image-with-paused-container-with-f-41f5f718
2024-11-25T06:17:46.8609064Z command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 image inspect ghcr.io/stargz-containers/alpine:3.13-org
2024-11-25T06:17:46.8611227Z image_remove_test.go:306: ======================== Test Run ========================
2024-11-25T06:17:46.8613568Z command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 rmi -f ghcr.io/stargz-containers/alpine:3.13-org
2024-11-25T06:17:46.8615453Z command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 images
2024-11-25T06:17:46.8617060Z command.go:112: assertion failed: expression is false: strings.Contains(stdout, compare): Output does not contain: "<none>"
2024-11-25T06:17:46.8618764Z Command: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 images
2024-11-25T06:17:46.8619696Z ExitCode: 0
2024-11-25T06:17:46.8620456Z Stdout: REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
2024-11-25T06:17:46.8621312Z
2024-11-25T06:17:46.8622809Z Stderr: time="2024-11-25T06:17:46Z" level=warning msg="content digest sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a: not found"
2024-11-25T06:17:46.8624200Z
2024-11-25T06:17:46.8624514Z Env:
2024-11-25T06:17:46.8624877Z HOSTNAME=ed64e3612023
2024-11-25T06:17:46.8625347Z container=docker
2024-11-25T06:17:46.8625777Z HOME=/root
2024-11-25T06:17:46.8626185Z LANG=C.UTF-8
2024-11-25T06:17:46.8626669Z INVOCATION_ID=7ddec279ffe441829a9ae2a4ba86fa56
2024-11-25T06:17:46.8627282Z TERM=xterm
2024-11-25T06:17:46.8627666Z SHLVL=3
2024-11-25T06:17:46.8628021Z CGO_ENABLED=0
2024-11-25T06:17:46.8628442Z _=/usr/local/bin/gotestsum
2024-11-25T06:17:46.8629455Z PATH=/usr/local/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-11-25T06:17:46.8653235Z ***
2024-11-25T06:17:46.8654229Z DOCKER_CONFIG=/tmp/TestRemoveRemove_image_with_paused_container_-_with_-f3820180234/001
2024-11-25T06:17:46.8655246Z NERDCTL_TOML=/tmp/TestRemoveRemove_image_with_paused_container_-_with_-f3820180234/001/nerdctl.toml
2024-11-25T06:17:46.8656401Z image_remove_test.go:306: ======================== Processing subtests ========================
2024-11-25T06:17:46.8657119Z case.go:164: ======================== Post-test cleanup ========================
2024-11-25T06:17:46.8658348Z command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 rm -f testremove-remove-image-with-paused-container-with-f-41f5f718
2024-11-25T06:17:46.8659666Z command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 rmi -f ec14c799
2024-11-25T06:17:46.8660449Z --- FAIL: TestRemove/Remove_image_with_paused_container_-_with_-f (0.94s) |
…o by containers In the current implementation, containers running by `nerdctl` dose not export entries containing fields such as `CONTAINER_NAME`, `IMAGE_NAME` , and etc to the journald log like containers running by `docker cli`. At this time, the journald log entry describes below when sending to the journald log using nerdctl. ``` > nerdctl run -d --name nginx-nerdctl --log-driver=journald nginx bb7df47d27fd73426cec286ed88c5abf1443e74df637e2440d2dbca7229a84dc > nerdctl ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES bb7df47d27fd docker.io/library/nginx:latest "/docker-entrypoint.…" 3 seconds ago Up nginx-nerdctl > sudo journalctl SYSLOG_IDENTIFIER=bb7df47d27fd -a -n 1 -o json-pretty { "__CURSOR" : "???", "__REALTIME_TIMESTAMP" : "1730899940827182", "__MONOTONIC_TIMESTAMP" : "10815937979908", "_BOOT_ID" : "???", "_UID" : "0", "_GID" : "0", "_CAP_EFFECTIVE" : "1ffffffffff", "_MACHINE_ID" : "???", "_HOSTNAME" : "???.us-west-2.amazon.com", "_TRANSPORT" : "journal", "_SYSTEMD_SLICE" : "system.slice", "PRIORITY" : "3", "_SYSTEMD_CGROUP" : "/system.slice/containerd.service", "_SYSTEMD_UNIT" : "containerd.service", "_COMM" : "nerdctl", "_EXE" : "/usr/local/bin/nerdctl", "_CMDLINE" : "/usr/local/bin/nerdctl _NERDCTL_INTERNAL_LOGGING /var/lib/nerdctl/1935db59", "SYSLOG_IDENTIFIER" : "bb7df47d27fd", "_PID" : "8118", "MESSAGE" : "2024/11/06 13:32:20 [notice] 1#1: start worker process 44", "_SOURCE_REALTIME_TIMESTAMP" : "1730899940825905" } ``` On the other hand, the output fields are listed below when we use the journald logging driver with docker cli. - https://docs.docker.com/engine/logging/drivers/journald/ As you can see, some entries are not output by nerdctl and are incompatible with the docker cli. This feature request is reported in the following: - containerd#3486 Therefore, in this pull request, we will add the fields to be output in the journald log. After applying this fix, the journald log will output the following fields. ``` { "__CURSOR": "???", "__REALTIME_TIMESTAMP": "1731385591671422", "__MONOTONIC_TIMESTAMP": "11301588824148", "_BOOT_ID": "???", "_MACHINE_ID": "???", "_HOSTNAME": "???.us-west-2.amazon.com", "PRIORITY": "3", "_TRANSPORT": "journal", "_UID": "0", "_GID": "0", "_COMM": "nerdctl", "_EXE": "/usr/local/bin/nerdctl", "_CMDLINE": "/usr/local/bin/nerdctl _NERDCTL_INTERNAL_LOGGING /var/lib/nerdctl/1935db59", "_CAP_EFFECTIVE": "1ffffffffff", "_SYSTEMD_CGROUP": "/system.slice/containerd.service", "_SYSTEMD_UNIT": "containerd.service", "_SYSTEMD_SLICE": "system.slice", "CONTAINER_NAME": "nginx-nerdctl", "IMAGE_NAME": "nginx", "CONTAINER_ID_FULL": "fe22eccbd704ba799785999079ac465ed067d5914e9e3f1020e769921d5a83c5", "SYSLOG_IDENTIFIER": "fe22eccbd704", "CONTAINER_TAG": "fe22eccbd704", "CONTAINER_ID": "fe22eccbd704", "_PID": "31643", "MESSAGE": "2024/11/12 04:26:31 [notice] 1#1: start worker process 44", "_SOURCE_REALTIME_TIMESTAMP": "1731385591669765" } ``` Signed-off-by: Hayato Kiwata <[email protected]>
At this time, the following test on Windows failed ...
Details
2024-11-26T05:15:07.1142738Z === �[31mFAIL�[0m: cmd/nerdctl/image TestRemove/Remove_image_with_running_container_-_with_-f (6.94s)
2024-11-26T05:15:07.1143588Z image_remove_test.go:306: ======================== Pre-test cleanup ========================
2024-11-26T05:15:07.1144904Z command.go:112: C:\Users\runneradmin\go\bin\nerdctl.exe --namespace=testremove-private-a0b107b8 rm -f testremove-remove-image-with-running-container-with-f-e9c3b26f
2024-11-26T05:15:07.1146269Z command.go:112: C:\Users\runneradmin\go\bin\nerdctl.exe --namespace=testremove-private-a0b107b8 rmi -f
2024-11-26T05:15:07.1147090Z image_remove_test.go:306: ======================== Test setup ========================
2024-11-26T05:15:07.1148872Z command.go:112: C:\Users\runneradmin\go\bin\nerdctl.exe --namespace=testremove-private-a0b107b8 run --pull always -d --name testremove-remove-image-with-running-container-with-f-e9c3b26f gcr.io/k8s-staging-e2e-test-images/busybox:1.36.1-1 sleep 3600
2024-11-26T05:15:07.1150940Z command.go:112: C:\Users\runneradmin\go\bin\nerdctl.exe --namespace=testremove-private-a0b107b8 image inspect gcr.io/k8s-staging-e2e-test-images/busybox:1.36.1-1
2024-11-26T05:15:07.1151994Z image_remove_test.go:306: ======================== Test Run ========================
2024-11-26T05:15:07.1153186Z command.go:112: C:\Users\runneradmin\go\bin\nerdctl.exe --namespace=testremove-private-a0b107b8 rmi -f gcr.io/k8s-staging-e2e-test-images/busybox:1.36.1-1
2024-11-26T05:15:07.1154447Z command.go:112: C:\Users\runneradmin\go\bin\nerdctl.exe --namespace=testremove-private-a0b107b8 images
2024-11-26T05:15:07.1155529Z command.go:112: assertion failed: expression is false: strings.Contains(stdout, compare): Output does not contain: "<none>"
2024-11-26T05:15:07.1156673Z Command: C:\Users\runneradmin\go\bin\nerdctl.exe --namespace=testremove-private-a0b107b8 images
2024-11-26T05:15:07.1157260Z ExitCode: 0
2024-11-26T05:15:07.1157797Z Stdout: REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
2024-11-26T05:15:07.1158268Z
2024-11-26T05:15:07.1159289Z Stderr: time="2024-11-26T05:13:51Z" level=warning msg="content digest sha256:a9155b13325b2abef48e71de77bb8ac015412a566829f621d06bfae5c699b1b9: not found"
2024-11-26T05:15:07.1160210Z When we try to I'm checking ..., but this PR has nothing to do with image deletion, so it is unclear why |
Hi, @fahedouch All tests have passed, so could you check when you have time? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks
In the current implementation, containers running by
nerdctl
don'texport entries containing fileds such as
CONTAINER_NAME
,IMAGE_NAME
, and etc to the journald log like containers runnging by
docker cli
.At this time, the journald log entry describes below when sending to the journald log using
nerdctl
.Details
On the other hand, the output fields are listed below when we use the journald logging driver with
docker cli
.As you can see, some entries are not output by
nerdctl
and are incompatible withdocker cli
.This feature request is reported in the following:
Therefore, in this pull request, we will add the fields to be output in
the journald log.
After applying this fix, the journald log will output the following
fields.
Details
Note that details are described in each commits.