Kubernetes OOMKilled (exit code 137): why it happens and how to see the next one coming

Your pod restarted. kubectl describe pod shows this:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Exit code 137 means the container's main process received SIGKILL (128 + 9). Combined with the reason OOMKilled, it means the Linux kernel killed the container because it used more memory than its cgroup allows. This post covers why it happens, how to find the real cause in five minutes, and how to see the next one coming.

Why containers get OOMKilled

1. The limit is simply too low. Memory grows with traffic, steps up during a peak, and crosses resources.limits.memory. The usage curve tracks load, then plateaus right under the limit.

2. A memory leak. Memory grows steadily even when traffic is flat. The pod is killed, restarts at its baseline, and climbs again, often on a regular rhythm.

3. The runtime does not know the limit. A JVM or another runtime sized for the node instead of the container will happily grow past the cgroup limit. Make sure heap settings are derived from the container limit, not from the host.

4. Memory you did not count. An emptyDir with medium: Memory, /dev/shm, and page cache from heavy file I/O are charged to the same cgroup. The application heap looks fine while the container is killed.

5. The node itself ran out of memory. If the node is under pressure, the kubelet evicts pods, and the kernel can OOM kill containers that sit well under their own limit. Pods without requests and limits (QoS class BestEffort) go first.

Find the cause in five minutes

Check the last termination of each container:

kubectl get pod <pod> -n <namespace> \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.restartCount}{"\n"}{end}'

Compare usage with the limit:

kubectl top pod <pod> -n <namespace> --containers
kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].resources}'

Look at the shape of the curve in Prometheus over a few hours. This is the fastest way to tell a leak from a small limit:

container_memory_working_set_bytes{namespace="<namespace>", pod=~"<deployment>-.*", container!="", container!="POD"}

• A sawtooth that climbs, drops at each restart, and climbs again at the same slope: a leak.
• A curve that follows traffic and flattens near the limit: a limit that is too small.
• A sudden jump right after a deploy: a regression in the new release. Compare with the previous version.
• Pods killed while far from their own limits: check node memory pressure and evictions with kubectl get events -n <namespace>.

Get alerted when it happens

This rule fires when a container restarted in the last 10 minutes and its last termination was an OOM kill:

- alert: ContainerOOMKilled
  expr: |
    max by (namespace, pod, container) (increase(kube_pod_container_status_restarts_total[10m])) > 0
    and on (namespace, pod, container)
    max by (namespace, pod, container) (kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}) == 1
  labels:
    severity: warning

It tells you after the kill. To be warned before, forecast memory against the limit with predict_linear: I walk through a copy-paste rule, and where it breaks, in Predict Kubernetes OOM kills 30 minutes ahead with one PromQL alert.

Seeing the next one coming

The two rules above cover the basics. What they cannot tell you is which of the causes above you are facing: a leak in one replica, a release that leaks faster than the one before, or a limit that is simply too tight.

That is the problem Ruptura is built for. It is an open-source operator for Kubernetes and OpenShift that reads the same Prometheus metrics and, for each workload, reports the time left before the OOM kill with a range, the evidence, the services that will go down with it, and the next step, straight into Alertmanager. It is early, and feedback is very welcome on GitHub.