Errors when configuring the vector buffer

[hightwoltt]

My Vector is running in a Docker container, and when I try to configure buffering, I get these errors:
there is only 1 vector container on the host, I've tried different options with drop_newest and block combined with different types of buffering. Vector works fine, but if I write docker exec -it vector bash and write the vector command in the container, I get the following errors.

When configuring the buffer on disk:
Conf

sinks:

  intermediary_vector:
    type: vector
    inputs:
      - add_host
    address: "192.168.1.103:9009"

    buffer:
      - type: disk
	max_size: 1073741824
        when_full: drop_newest

Error:
ERROR vector::topology::builder: Configuration error. error=Sink "intermediary_vector": error occurred when building buffer: failed to build individual stage 0: failed to load/create ledger: failed to lock buffer.lock; is another Vector process running and using this buffer?

The buffer.lock file also appears in the container in the /var/lib/vector/buffer/v2/intermediary_vector directory. Deleting it doesn't help.

When configuring the buffer in memory:
Conf

sinks:

  intermediary_vector:
    type: vector
    inputs:
      - add_host
    address: "192.168.1.103:9009"

    buffer:
      - type: memory
        max_events: 5000
        when_full: drop_newest

Error:

ERROR vector::internal_events::socket: Error binding socket. error=error creating server listener: Address already in use (os error 98) error_code="socket_bind" error_type="io_failed" stage="receiving" mode=tcp internal_log_rate_limit=true
ERROR vector::app: An error occurred that Vector couldn't handle: error creating server listener: Address already in use (os error 98).

I will be very grateful for your help in solving this problem!

[jszwedko]

Hi @hightwoltt !

It sounds like you may have another Vector process running with the same configuration. That would explain both the error about something already being bound to the is bound to the address as well as the buffer locking.

[hightwoltt]

@jszwedko
The addresses of Docker containers do not overlap

[pront]

Another sanity check is to run:

lsof | grep buffer.lock

[hightwoltt]

This is how the output of the command looks like:

# lsof | grep buffer.lock
vector    1               root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1   8 vector-wo root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1   9 vector-wo root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1  10 vector-wo root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1  11 vector-wo root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1  13 vector-wo root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1  14 vector-wo root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1  15 vector-wo root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1  16 vector-wo root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1  17 vector-wo root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1  18 vector-wo root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1  36 vector-wo root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1  49 vector-wo root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock
vector    1  73 blocking- root    9uW     REG              0,718         0    548033 /var/lib/vector/buffer/v2/intermediary_vector/buffer.lock

[pront]

The above can be interpreted as "multiple threads from a single vector process are using the file. So the are no multiple processes 'racing' for the file. What happens when you restart Vector?

Also worth looking at these issues:

• Orphaned buffer lock #18831
• Investigate fslock usage #19631

[hightwoltt]

@pront I've already tried reloading the container with Vector, unfortunately it doesn't help.... I also tried reinstalling Vector (docker stop vector && docker rm vector, + docker system/volume/image prune -a).
None of this helped me, the errors do not disappear... If I completely remove the buffer setting from the configuration, I get an error:
2025-01-10T07:40:43.098259Z ERROR vector::internal_events::socket: Error binding socket. error=error creating server listener: Address already in use (os error 98) error_code="socket_bind" error_type="io_failed" stage="receiving" mode=tcp internal_log_rate_limit=true 2025-01-10T07:40:43.098375Z ERROR vector::app: An error occurred that Vector couldn't handle: error creating server listener: Address already in use (os error 98).

The launch of the Docker container with Vector was performed by the command:

docker run -d \
    --name vector \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /etc/vector/vector.yaml:/etc/vector/vector.yaml:ro \
    timberio/vector:0.43.1-debian \
    --config /etc/vector/vector.yaml