[Detector Support]: M.2 A+E Key Single TPU and 5k Major Page Faults per Second.

[reefland]

Describe the problem you are having

I did not have this issue using the USB detector. There is no obvious issue, Frigate is working fine. I would not know there was an issue if Prometheus hadn't raised the alert. The Prometheus metric is node_vmstat_pgmajfault, which identifies the computer with the issue. I then used Top, adding field nMaj and sorting by it. The value increases by 5k a second on each of the frigate.capture processes.

• I have two MS-01 PC's (each with 96GB of RAM) with M.2 A+E key TPU and can reproduce this on both of them. The PC has over 64Gi of RAM free:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            94Gi        16Gi        67Gi       120Mi        11Gi        77Gi
Swap:             0B          0B          0B

Prometheus alert: Memory major pages are occurring at very high rate at 192.168.10.218:9100, 500 major page faults per second for the last 15 minutes, is currently at 5071.13. The alerts triggers at 500 per second, I'm over 5000.

I have given Frigate memory: 4608Mi resource limit and it is using less than 1/2 of that:

Version

0.14.1

Frigate config file

        logger:
          default: info
          # logs:
          #   frigate.record: debug

        auth:
          enabled: false

        mqtt:
          host: mosquitto-mqtt.mosquitto.svc.cluster.local
          topic_prefix: frigate
          user: "{FRIGATE_MQTT_USERNAME}"
          password: "{FRIGATE_MQTT_PASSWORD}"

        database:
          path: /config/frigate.db

        detectors:
          coral:
            type: edgetpu
            device: pci

        # https://docs.frigate.video/configuration/ffmpeg_presets
        ffmpeg:
          global_args: ["-hide_banner", "-loglevel", "warning"]
          hwaccel_args: preset-vaapi
          output_args:
            record: preset-record-generic

        snapshots:
          enabled: true
          timestamp: false
          bounding_box: true
          retain:
            default: 7

        record:
          enabled: true
          retain:
            days: 7
            mode: active_objects
          events:
            pre_capture: 3
            post_capture: 3
            retain:
              default: 14
              mode: active_objects
              # objects:
              #   dog: 3
              #   bird: 5

        # NOTE: Can be overridden at the camera level
        objects:
          track:
            - person
            - dog
            # - bird
          filters:
            person:
              min_area: 5000
              max_area: 100000
              min_score: 0.6
              threshold: 0.79
            car:
              min_score: 0.60
              threshold: 0.75
            dog:
              min_score: 0.60
              threshold: 0.75
            bicycle:
              min_score: 0.60
              threshold: 0.75
            motorcycle:
              min_score: 0.60
              threshold: 0.75
            package:
              min_score: 0.60
              threshold: 0.75
            ups:
              min_score: 0.60
              threshold: 0.75
            fedex:
              min_score: 0.60
              threshold: 0.75
            amazon:
              min_score: 0.60
              threshold: 0.75

        motion:
          mask:
            - 498,34,0,36,0,0,494,0

        go2rtc:
          streams:
            frigate_kitchen:
              - rtsp://wyze-cam-outside.rich-durso.us:8554/unicast
            frigate_livingroom:
              - rtsp://wyze-cam-livingroom.rich-durso.us:8554/unicast
            frigate_office:
              - rtsp://wyze-cam-office.rich-durso.us:8554/unicast

        live:
          quality: 5

        cameras:
          kitchen:
            ffmpeg:
              inputs:
                - path: rtsp://localhost:8554/frigate_kitchen
                  roles:
                    - detect
                    - record
            detect:
              width: 1280
              height: 720
              fps: 5
            mqtt:
              enabled: true
            snapshots:
              enabled: true
            record:
              enabled: true
            motion:
              threshold: 25
              contour_area: 30
              mask:
                - 117,26,453,42,399,413,87,415
                - 720,700,1280,700,1280,0,595,0,636,316,903,374

          livingroom:
            ffmpeg:
              inputs:
                - path: rtsp://localhost:8554/frigate_livingroom
                  roles:
                    - detect
                    - record
            detect:
              width: 1280
              height: 720
              fps: 5

            mqtt:
              enabled: true
            snapshots:
              enabled: true
            record:
              enabled: true

          office:
            ffmpeg:
              inputs:
                - path: rtsp://localhost:8554/frigate_office
                  roles:
                    - detect
                    - record
            detect:
              width: 1280
              height: 720
              fps: 5
            mqtt:
              enabled: true
            snapshots:
              enabled: true
            record:
              enabled: true

docker-compose file or Docker CLI command

Running under K3s.

Relevant Frigate log output

s6-rc: info: service s6rc-fdholder: starting
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service s6rc-fdholder successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service log-prepare: starting
s6-rc: info: service log-prepare successfully started
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service certsync-log: starting
s6-rc: info: service nginx-log successfully started
s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service frigate-log successfully started
s6-rc: info: service certsync-log successfully started
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service frigate: starting
s6-rc: info: service frigate successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service go2rtc-healthcheck successfully started
2025-01-02 13:25:36.033964675 [INFO] Preparing Frigate...
2025-01-02 13:25:36.034732474 [INFO] Starting NGINX...
[WARN] /sys/fs/cgroup/cpu.max not found. Falling back to /proc/cpuinfo.
2025-01-02 13:25:36.038148450 [INFO] Removing stale config from last run...
2025-01-02 13:25:36.040389169 [INFO] Preparing new go2rtc config...
2025-01-02 13:25:36.044923083 [INFO] No TLS certificate found. Generating a self signed certificate...
2025-01-02 13:25:36.048930506 [INFO] Starting Frigate...
s6-rc: info: service nginx successfully started
s6-rc: info: service certsync: starting
s6-rc: info: service certsync successfully started
s6-rc: info: service legacy-services: starting
2025-01-02 13:25:36.242180720 [INFO] Starting certsync...
s6-rc: info: service legacy-services successfully started
2025-01-02 13:25:36.298602225 127.0.0.1 - - [02/Jan/2025:08:25:36 -0500] "" 400 0 "-" "-" "-"
2025-01-02 13:25:36.301102632 [INFO] Starting go2rtc...
2025-01-02 13:25:36.396860733 08:25:36.396 INF go2rtc platform=linux/amd64 revision=b2399f3 version=1.9.2
2025-01-02 13:25:36.396865659 08:25:36.396 INF config path=/dev/shm/go2rtc.yaml
2025-01-02 13:25:36.397040040 08:25:36.397 INF [rtsp] listen addr=:8554
2025-01-02 13:25:36.397104399 08:25:36.397 INF [webrtc] listen addr=:8555/tcp
2025-01-02 13:25:36.397203027 08:25:36.397 INF [api] listen addr=:1984
2025-01-02 13:25:36.955667554 [2025-01-02 08:25:36] frigate.app INFO : Starting Frigate (0.14.1-f4f3cfa)
2025-01-02 13:25:36.955703431 [2025-01-02 08:25:36] frigate.util.config INFO : Checking if frigate config needs migration...
2025-01-02 13:25:36.955808237 [2025-01-02 08:25:36] frigate.util.config ERROR : Config file is read-only, unable to migrate config file.
2025-01-02 13:25:36.988662768 [2025-01-02 08:25:36] peewee_migrate.logs INFO : Starting migrations
2025-01-02 13:25:37.069601707 [2025-01-02 08:25:36] peewee_migrate.logs INFO : There is nothing to migrate
2025-01-02 13:25:37.089790173 [2025-01-02 08:25:37] frigate.app INFO : Recording process started: 391
2025-01-02 13:25:37.090477191 [2025-01-02 08:25:37] frigate.app INFO : Recording process started: 392
2025-01-02 13:25:37.093061279 [2025-01-02 08:25:37] frigate.app INFO : go2rtc process pid: 98
2025-01-02 13:25:37.111706492 [2025-01-02 08:25:37] detector.coral INFO : Starting detection process: 421
2025-01-02 13:25:37.118512909 [2025-01-02 08:25:37] frigate.app INFO : Output process started: 423
2025-01-02 13:25:37.118559391 [2025-01-02 08:25:37] frigate.detectors.plugins.edgetpu_tfl INFO : Attempting to load TPU as pci
2025-01-02 13:25:37.118586805 [2025-01-02 08:25:37] frigate.detectors.plugins.edgetpu_tfl INFO : TPU found
2025-01-02 13:25:37.127546241 [2025-01-02 08:25:37] frigate.app INFO : Camera processor started for kitchen: 450
2025-01-02 13:25:37.127631579 [2025-01-02 08:25:37] frigate.app INFO : Camera processor started for livingroom: 451
2025-01-02 13:25:37.132173964 [2025-01-02 08:25:37] frigate.app INFO : Camera processor started for office: 457
2025-01-02 13:25:37.136268659 [2025-01-02 08:25:37] frigate.app INFO : Capture process started for kitchen: 475
2025-01-02 13:25:37.140293911 [2025-01-02 08:25:37] frigate.app INFO : Capture process started for livingroom: 484
2025-01-02 13:25:37.143866708 [2025-01-02 08:25:37] frigate.app INFO : Capture process started for office: 489
2025-01-02 13:25:46.033683827 [INFO] Starting go2rtc healthcheck service...

Install method

HassOS Addon

Object Detector

Coral

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

Installed using Helm Chart. Have been using Frigate for years with same config with USB TPU without issues. I removed the USB and switched to M.2 A+E single TPU detector and updated the detector section of the config.

[hawkeye217]

It's unclear that this is actually related to something Frigate can address. A a "major fault" refers to a page fault that requires the OS to retrieve data from the disk to satisfy a memory access request. There could be many reasons for this, but all that's being indicated is that there's a lot of transfer between RAM and swap space.

[reefland]

The PC has 96GB of RAM, 77Gi free and there is NO swap file.

[hawkeye217]

You'll want to dig into exactly what your Prometheus metric is measuring.

If Frigate is working fine, I'm not sure there's anything we can help with directly. Perhaps the other maintainers or the community have thoughts?

[reefland]

Yeah, all I can do is document it and see if someone else can build observations on it in the future. For now I'll silence the alert.