[Bug]: Memory leak (frigate.capture & ffmpeg) #19925
Replies: 3 comments 11 replies
-
|
You've identified a significant memory leak issue affecting both FFmpeg and the frigate.capture process. Based on the available information, these are known issues that can cause out-of-memory conditions and system instability. Memory Leak IssuesThe memory leaks you're experiencing have been documented in several cases where users report the frigate.capture process being killed by the OOM-killer (2) (3). In one documented case, the process consumed over 18GB of RAM before being terminated (2). Another user reported similar issues with WiFi cameras where the capture process would grow to 21GB of memory usage (3). FFmpeg Memory IssuesThe FFmpeg memory leak you've identified appears to be related to corrupted or unstable RTSP streams (2) (3). When FFmpeg encounters problematic streams, it can continuously allocate memory without proper cleanup, leading to the gradual memory growth you've observed. Video Pipeline ContextFrigate's video pipeline processes camera feeds through several stages: feed acquisition, video decoding, motion detection, object detection, and recording (1). The capture process is responsible for acquiring and decoding the video stream, which explains why memory issues in this component can be so severe (1). Hardware Acceleration ImpactHardware acceleration for video decoding is highly recommended to reduce CPU load (4). The Coral TPU and other detectors handle object detection, but video decoding still requires significant CPU or GPU resources (5). Memory leaks in the decoding process can compound these resource requirements (4). Shared Memory ConfigurationFrigate uses shared memory ( Mitigation StrategiesBased on the documentation, consider:
The code analysis you've provided regarding the potential memory leak in the 📚 Sources:
Have another question? Just tag @inkeep. |
Beta Was this translation helpful? Give feedback.
-
|
#17188 #16710 #14571 |
Beta Was this translation helpful? Give feedback.
-
|
What version of Frigate are you running? This issue should be corrected in the current version, 0.16.1. |
Beta Was this translation helpful? Give feedback.
-
|
@hawkeye217 Its 0.16.0b2 |
Beta Was this translation helpful? Give feedback.
-
|
We suggest always updating to the latest official release before trying to track down bugs. :) |
Beta Was this translation helpful? Give feedback.
-
|
Hi, |
Beta Was this translation helpful? Give feedback.
-
|
So to add on to this, your suggestion of where the memory leak is is incorrect. According to your docker stats, the ffmpeg process itself is using the memory, not the python process. To get more information about memory leaks and be able to truly understand where they are, it is recommended to use something like https://github.com/bloomberg/memray to create a memory dump once memory is high so we can see what is using the memory, if it is allocated or just not released, etc. The top here though suggests that it is only ffmpeg using memory, which is not something we have seen after the 0.16 fixes (and that was only under specific circumstances) |
Beta Was this translation helpful? Give feedback.
-
|
Any data is pretty anecdotal on a 4-month old version of Frigate. It will only be helpful to see these things on 0.16.1 since we've made some improvements to this very area since then. |
Beta Was this translation helpful? Give feedback.
-
|
@hawkeye217 Got it! This will take me some time, so I’ll get back to you in a couple of days. In the meantime, have a great weekend! 😄 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, you as well. |
Beta Was this translation helpful? Give feedback.
-
Just a heads-up that the 0.16.1 (stable) build appears to still exhibit a memory leak. I’ve attached an image that illustrates the increasing memory usage over time for reference. |
Beta Was this translation helpful? Give feedback.
-
|
What would be most helpful would be what @NickM-27 described above: #19925 (comment) |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Checklist
Describe the problem you are having
Hi 👋
Edits
07:38
Just for added clarity, the memlogs and most of the logs attached here points to FFmpeg, the frigate.capture leaks, can only be seen in the picture with btop taken with my phone because my whole system crashed unfortunately.
This is the only Memdump i have for ffmpeg.capture provided by journalctl logs (Click to expand)
TLDR
FFmpeg memory usage grow infinitely when a corrupted RTSP Stream is received.
This will "slowly" (hours) grow until a OOM is triggered.
Recording of said stream will also stop working until an OOM is triggered.
After fixing the memory leak in FFmpeg another bug in frigate.capture caught my attention.
ffmpeg.capture will randomly peak to 28GB RAM.
This report isn't perfect or complete, but due limited free time on my end, i'm going to post the current state of my findings
My current "fix" for FFmpeg
Building a "cutting edge" FFmpeg 8.0 statically linked with all libs/dependencies updated to latest available version (Dockerfile can be provided) and setting it as frigate ffmpeg
What i think causes frigate.capture to leak
WARNING : First time reading frigate code and only had 30 mins to look at it, so i might be misunderstanding this part
I also confirmed that this code block is still used in Release 0.16.1
In
frigate/video.pyin funccapture_framesline 155 ➜ 162 (0.16.0b2):Isn't this a memory leak when the Queue is full ?
I mean its skipped but the frame_manager is never decreased by calling close if the queue.put fails due to being full.
So that frame will never be freed, right ?
If that's the case a simple
finallywould fix it 👀System Info
System A (Triggered investigation in ffmpeg)
System B (Personal computer frigate.capture)
6.14.0-29-generic # 29-Ubuntu
Research
Introduction
After receiving multiple pings from Prometheus and Icinga about High memory usage and OOM being triggered; i started reviewing logs and found Frigate Container was the culprit.
During FFmpeg bug we can clearly see the slow increase over time.
After FFmpeg fix, frigate.capture leaks
Unfortunately no prometheus running on my main computer so i took a picture before my computer crashed
This specific leak grew gigabytes a second
PS example:
Other examples (click to expand)
We get it now i think ^^
FFmpeg buildarg & version
Note: Frigate uses the Raspberry build
FFprobe
CAM3 - ABNORMAL MEMORY
Go2RTC stream
Camera stream
CAM4 - NORMAL MEMORY (TLDR Exactly the same)Y
Go2RTC stream
Camera stream
Slabtop
kmalloc-1k: 12.4 GB ??
Click to expand
Meminfo
Command :
Click to expand
Memleak to process
For me this is the definitive proof that the process is leaking
Click to expand
Frigate.capture
Only observation i had the time to look into are in the TLDR
I would like to run some memory profiling tools to confirm it, hopufully soon. ⌛
Thanks for your time 👋
Steps to reproduce
FFmpeg Bug
This is not 100% reliable currently, pcaps and emulated video streams are being captured to effectively reproduce the bug(s)
Basically you need to find a corrupted RTSP Stream.
Example not ideal but should work.
I use mediamtx to proxy the stream configured like so : (I know we can skip this on frigate)
Relevant frigate config
Just wait ⌛ and monitor ram metrics.
frigate.capture Bug (Unrelated to ffmpeg)
For faster results use an High Res input with high bitrate.
To emulate an active server or weak cpu, we need to "stresstest" the CPU to cause CPU exhaustion to trigger the bug.
Version
0.16.0-1fa7ce54
In which browser(s) are you experiencing the issue with?
N/A
Frigate config file
docker-compose file or Docker CLI command
Relevant Frigate log output
Relevant go2rtc log output
Operating system
Debian
Install method
Docker Compose
Network connection
Wired
Camera make and model
N/A
Screenshots of the Frigate UI's System metrics pages
N/A
Any other information that may be helpful
N/A
Beta Was this translation helpful? Give feedback.
All reactions