Would bottlerocket block gdb to work properly?

[sfrique]

Hello,

We are trying to debug an app stuck in high cpu usage, but when trying to run gdb it seems it does not have permissions to see the memory.

gdb) t a a bt

Thread 22 (LWP 363 "FFI Callback Di"):
#0  __cp_end () at src/thread/x86_64/syscall_cp.s:29
Backtrace stopped: Cannot access memory at address 0x7ff9d83c65a8

Thread 21 (LWP 288 "worker-2"):
#0  __cp_end () at src/thread/x86_64/syscall_cp.s:29
Backtrace stopped: Cannot access memory at address 0x7ff9d969e5f8
...
...
...

Would this be something in bottlerocket?

We are using EKS, after connecting with session manager

enter-admin-container
sudo sheltie
nsenter --mount --net --pid --target XXXX bash

Any ideas?

[bcressey]

You are likely running into the default Yama restriction on ptrace scope. Try this from the sheltie session:

sysctl -w kernel.yama.ptrace_scope=0

[sfrique]

Still have the same problem, thanks anyway!

Here the relevant logs:

[root@admin]# sudo sheltie
bash-5.1# sysctl kernel.yama.ptrace_scope
kernel.yama.ptrace_scope = 1
bash-5.1# sysctl -w kernel.yama.ptrace_scope=0
kernel.yama.ptrace_scope = 0
bash-5.1# sysctl kernel.yama.ptrace_scope
kernel.yama.ptrace_scope = 0
bash-5.1# nsenter --mount --net --pid --target 37435 bash
bash-5.1#
bash-5.1# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  1.2  2.9 568576 479132 ?       Ssl  13:29   1:19 puma 5.6.5 (tcp://0.0.0.0:8080) [app]
root          24  1.9  5.8 1059520 942024 ?      Sl   13:29   2:05 puma: cluster worker 0: 1 [app]
root          37  1.6  3.7 719396 598388 ?       Sl   13:29   1:46 puma: cluster worker 1: 1 [app]
root          60  1.6  3.4 670292 546944 ?       Sl   13:29   1:45 puma: cluster worker 2: 1 [app]
root          78  2.0  6.2 1109520 994280 ?      Sl   13:29   2:11 puma: cluster worker 3: 1 [app]
root          96  1.9  5.9 1074172 949876 ?      Sl   13:29   2:08 puma: cluster worker 4: 1 [app]
root         114  1.9  4.0 761540 646204 ?       Sl   13:29   2:08 puma: cluster worker 5: 1 [app]
root         127  2.2  3.9 759192 635880 ?       Sl   13:29   2:23 puma: cluster worker 6: 1 [app]
root         150  2.2  3.5 692548 570392 ?       Sl   13:29   2:23 puma: cluster worker 7: 1 [app]
root         511  0.0  0.0      0     0 ?        Z    15:15   0:00 [gdb] <defunct>
root         515  0.0  0.0   2564  2288 ?        S    15:16   0:00 bash
root         516  0.0  0.0   1704   864 ?        R+   15:16   0:00 ps aux
bash-5.1# gdb -p 60
.....
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 60
[New LWP 61]
[New LWP 62]
[New LWP 63]
[New LWP 64]
[New LWP 65]
[New LWP 66]
[New LWP 67]
[New LWP 68]
[New LWP 69]
[New LWP 70]
[New LWP 71]
[New LWP 72]
[New LWP 73]
[New LWP 74]
[New LWP 75]
[New LWP 76]
[New LWP 77]
[New LWP 169]
[New LWP 171]
[New LWP 384]
Failed to read a valid object file image from memory.
0x00007fc8fc6674a3 in ?? () from /lib/ld-musl-x86_64.so.1
(gdb) t a a bt

Thread 21 (LWP 384 "worker-3"):
#0  0x00007fc8fc6674a3 in ?? () from /lib/ld-musl-x86_64.so.1
Backtrace stopped: Cannot access memory at address 0x7fc8d38775f8

Thread 20 (LWP 171 "Timeout stdlib "):
#0  0x00007fc8fc6674a3 in ?? () from /lib/ld-musl-x86_64.so.1
Backtrace stopped: Cannot access memory at address 0x7fc8edef50d8
.....

[bcressey]

You could check dmesg for SELinux denials. That seems like a long shot but I'm not aware of anything else that Bottlerocket might do which would block this.

[sfrique]

 9249.268817] audit: type=1400 audit(1684336615.539:3727): avc:  denied  { write } for  pid=97667 comm="gdb" name="mem" dev="proc" ino=692082 scontext=system_u:system_r:super_t:s0-s0:c0.c1023 tcontext=system_u:system_r:container_t:s0:c221,c830 tclass=file permissive=0
[ 9249.268819] audit: type=1400 audit(1684336615.539:3728): avc:  denied  { write } for  pid=97667 comm="gdb" name="mem" dev="proc" ino=692083 scontext=system_u:system_r:super_t:s0-s0:c0.c1023 tcontext=system_u:system_r:container_t:s0:c221,c830 tclass=file permissive=0

It's been denied, i look at selinux, before as well.
i will take a deeper look later on how to disable this temporary

thanks

[sfrique]

it seems we can't disable SELinux on bottlerocket =[