Container runs out of process IDs
fork fails, the process cannot start threads or children.
ImpactRequests fail, the container may hang
Can it be seen coming?Yes
RupturaPlanned
How to detect it
Process count near the PID limit.
Metrics and signals to watch:
container_processesnode pid limits
Can you see it coming?
Yes Precursor: Thread or process count growing steadily.
What it hits next
callers of the workload, other pods of the node.
What to do
Fix the thread leak or raise the PID limit.
Sources
- Kubernetes node condition PIDPressure
Where Ruptura stands
This failure is on the Ruptura roadmap. It is not reported yet.
Related failures
- Memory leak ends in an OOM kill
- Memory growth accelerates toward the limit
- Sudden memory jump past the limit
- Container killed by OOM again and again
- Container crashes at start, again and again
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub