Analyzing OOM

When analyzing Out-of-Memory (OOM) events in Linux, the kernel logs critical details via dmesg and process-specific data in /proc/<pid>/status. These tools help pinpoint the process responsible for the OOM event, its memory usage, and the kernel’s decision-making process. Below is a structured approach to extract actionable insights.

Analyzing OOM Events with `dmesg`¶

The kernel logs OOM events in the ring buffer, which can be inspected using dmesg. These logs include the process ID (PID), memory usage metrics, and the kernel’s decision to terminate the process.

Run the following command to extract relevant entries:

dmesg | grep -i 'out of memory'

Example output might look like:

[12345.67890] Out of memory: Kill process 'nginx' (1234) score 123 or sacrifice child.
[12345.67890] Killed process 1234 (nginx) score 123.

- PID: The process ID (e.g., 1234) is critical for further analysis. - Score: The OOM score indicates the process’s priority for termination. Higher scores mean the process is more likely to be killed.

Step 2: Identify the Process¶

Use ps or top to find the process name associated with the PID:

ps -p 1234 -o pid,comm

Output:

  PID COMMAND
  1234 nginx

Examining `/proc/<pid>/status` for Memory Details¶

The /proc/<pid>/status file contains detailed memory and resource information for a process. Use it to analyze memory usage and OOM-related metrics.

Step 1: Check Memory Usage¶

Inspect the status file for the following fields:

cat /proc/1234/status | grep -E 'Vm[RS]SS|OOMKilled'

Example output:

VmRSS:        1024kB
VmData:       800kB
VmSwap:        0kB
OOMKilled:     1

- VmRSS: Resident Set Size (physical memory used by the process). - VmData: Data segment size (heap and stack memory). - VmSwap: Swap usage (non-zero indicates memory pressure). - OOMKilled: A value of 1 confirms the process was terminated by the OOM killer.

Step 2: Analyze Resource Limits¶

Check resource limits to identify potential constraints:

cat /proc/1234/limits

Look for Max memory size (soft limit) and Max swap settings. If the process exceeded these limits, it may have triggered the OOM killer.

Key Takeaways¶

Use dmesg | grep -i 'out of memory' to locate OOM events and extract PIDs.
Verify the process name with ps -p <PID> to contextualize the event.
Analyze /proc/<pid>/status to understand memory usage (VmRSS, VmData) and OOMKilled status.
Monitor resource limits in /proc/<pid>/limits to identify potential constraint violations.