Analyse and visualise Linux OOM output

OOMAnalyser is a small project to transform the OOM message of a Linux kernel into a more user-friendly format.

OOMAnalyser consists of a web page into whose input field the OOM message is copied. JavaScript code extracts the data from it and displays the details. All processing takes place in the browser. No data is transferred to external servers. This makes it possible to use a locally stored copy of the website for analysis. This project is written in Python and uses Transcrypt to translate Python code into JavaScript.

Step 1 - Enter your OOM message



Step 2 - Results

Go back to "Step 1 - Enter your OOM message" to run a new analysis.

Summary

The OOM killer was automatically triggered to free memory, because the system couldn't satisfy the memory request. The OOM killer calculates a score for each process and terminates the process with the highest score to satisfy the original memory request.

The process "" (PID ) requested () memory.

The OOM killer was manually triggered (e.g. with "echo f > /proc/sysrq-trigger") by a user with root privileges. There is no demand to free memory but the OOM killer started nevertheless. The OOM killer calculates a score for each process and terminates the process with the highest score.

The process "" (PID ) with an OOM score of has been terminated. It uses () of the resident memory.

The system has physical memory and swap space. That's total. ( out of ) physical memory and ( out of ) swap space are in use.

The system has physical memory and no swap space. ( out of ) physical memory are in use.

Details of analysis

The result of the analysis is displayed in three columns. The first column is used to name the property including the original OOM identifier in brackets. The extracted information is displayed in the second column. The last column contains further details and additional information.

Trigger Process
(PID ) This process requests memory and is triggering thereby the OOM situation.
Memory allocation flags
(gfp_mask)		 These flags are used to control the kernel internal memory allocation
GFP stands for __get_free_pages().
Node mask to show on which CPU Cores this process can run
(nodemask)		 Bit mask indicating the cores on which the process can run.
Requested memory
(order)		 (2) pages / The kernel specifies the requested number of pages as exponent of power of two.
Adjust oom-killer score
(oom_score_adj)		 This value is added to the badness score before it's used to determine the process to be killed.
Killed Process
(PID ) Process killed by Linux kernel to satisfy the memory request.
OOM Score
(score)		 Programs with the highest OOM score are terminated first.
Virtual Memory
(total_vm) 		Virtual memory used by this process.
Total resident anonymous memory
(rss)		 All virtual process memory mapped into RAM.
TotalRSS = anon-rss + file-rss + shmem-rss
Resident anonymous memory
(anon-rss)		 Resident anonymous pages
Part of the virtual process memory mapped into RAM.
Resident file mapping memory
(file-rss)		 Resident file mapping pages
Files which have been mapped into RAM (with mmap(2).)
Resident shared memory
(shmem-rss)		 Resident shared memory pages
This may include System V shared memory and shared anonymous memory.
Memory Usage
Graphs
RAM Summary
Swap Summary
Swap Usage
No swap space available
Swap Total Total amount of swap space available. [1]
Swap Free Amount of swap space that is currently unused. [1]
Swap Cached Memory that once was swapped out, is swapped back in but still also is in the swap file. (If memory pressure is high, these pages don't need to be swapped out again because they are already in the swap file. This saves I/O). [1]
Swap Used Amount of used swap space w/o cached swap
(SwapUsed = SwapTotal - SwapFree -SwapCache)
Memory Pages
RAM pages Total number of RAM pages
HighMem/MovableOnly Number of pages in the High Memory Area or marked movable for Contiguous Memory Allocator (CMA).
HighMem pages are also counted in the total page number.
Reserved pages Number of reserved pages
CMA reserved pages 0 Pages reserved for Contiguous Memory Allocator (CMA)
Pagetable Cache 0 Number of pages in pagetable cache
Number of pages with hardware errors 0 Pages with uncorrectable memory errors
Memory Usage Details
Active anonymous memory
(active_anon)		 Recently used anonymous memory.
These memory pages will usually not swapped out.
Inactive anonymous memory
(inactive_anon)		 Least recently used anonymous memory.
These memory pages can be swapped out.
Isolated anonymous memory
(isolated_anon)		 Memory isolation is used to separate memory between different virtual machines.
Active Pagecache
(active_file)		 Pagecache that has been used more recently and usually not reclaimed unless absolutely necessary.
Inactive Pagecache
(inactive_file)		 Pagecache which has been less recently used. It can be reclaimed without huge performance impact.
Isolated Pagecache
(isolated_file)		 Memory isolation is used to separate memory between different virtual machines.
Unevictable Pages
(unevictable)		 Unevictable memory. It can't be swapped out because the pages are owned by ramfs or protected by mlock(3) / shmctl(SHM_LOCK). Unevictable pages are managed by kernels LRU framework.
Dirty Pages
(dirty)		 Memory which is waiting to get written back to the disk. [1]
Writeback
(writeback)		 Memory which is actively being written back to the disk. [1]
Unstable
(unstable)		 Not yet committed to stable storage.
Slab Reclaimable
(slab_reclaimable)		 Slab is a in-kernel data structures cache. Part of Slab, that might be reclaimed, such as caches. [1]
Additional details are listed in slabinfo(5) also.
Slab Unreclaimable
(slab_unreclaimable)		 Part of Slab, that cannot be reclaimed on memory pressure. [1]
Mapped
(mapped)		 Files which have been mapped into memory (with mmap(2)), such as libraries. [1]
Shared Memory
(shmem)		 Amount of memory consumed in tmpfs(5) filesystems. [1]
Pagetables
(pagetables)		 Amount of memory dedicated to the lowest level of pagetables. [1]
Bounce
(bounce)		 Memory used for block device "bounce buffers". [1]
Free pages
(free)		 Free pages
Free per-CPU pages
(free_pcp)		 Free number of pages per CPU
Free CMA pages
(free_cma)		 Pages reserved but not used by Contiguous Memory Allocator (CMA)
Total Pagecache Total number of pages in pagecache
Operating System
Kernel
Distribution Guessed from the kernel version
Platform Guessed from the kernel version
Page size Guessed
Memory Chunks 

            		
        


        

            Process Table
        

        

            
            
                

                    
                        

                            pid
                                
                            
                            uid
                                
                            
                            tgid
                                
                            
                            total_vm
                                
                            
                            rss
                                
                            
                            nr_ptes
                                
                            
                            swapents
                                
                            
                            oom_score_adj
                                
                            
                            name
                                
                            
                            notes
                                
                            
                        

                    
                    
                    
                

            		
        


        

        

            Hardware Details
        

        

            
            
                

            		
        


        

        

            Kernel Call Trace
        

        

            
            
                

            		
        


        

        

            Entire OOM Message
                (click to hide)
            
        

        

            
            
                

            		
        

        
    


    

        Go back to
        "Step 1 - Enter your OOM message"
        to run a new analysis.
    


    
Footnotes

    
    
        
  1. 
	proc(5) - process information pseudo-filesystem (Go Back)
        
    2. 
    





Further Information


    
    
  1. 
        Linux man pages online
    
    2. 
    
  2. 
        
            Decoding the Linux kernel's page allocation failure messages
        
    
    3. 
    
  3. 
        Linux Kernel OOM Log Analysis
    
    4. 
    
  4. 
        Kernel Source code with
        mm/omm_kill.c
        as starting point at kernel.org
    
    5. 
    
  5. 
        A more comfortable kernel source code browser with
        mm/omm_kill.c at
        elixir.bootlin.com
    
    6. 





    
Changelog

    (click to show / hide)





    
Version 0.5.0 - 2021-XX-XX:


    
General

    
    
        
  1. Improve SVG chart colour palette
    2. 
        
  2. Add Selenium based unit tests
    3. 
        
  3. Fix to allow process names with spaces
    4. 
        
  4. Rework removal of unused information
    5. 
        
  5. Report uncaught errors to the user
    6. 
        
  6. Add support for manually triggered OOM (suggested by Mikko Rantalainen)
    7. 
        
  7. Add support for systems w/o swap (suggested by Mikko Rantalainen)
    8. 
        
  8. Add support for newer kernels (suggested by Mikko Rantalainen)
    9. 
        
  9. Add support for journalctl output (suggested by Mikko Rantalainen)
    10. 
        
  10. Add support for newer process table format
    11. 
        
  11. ...
    12. 
    


    
Note

    See the
    commit history
    of the repository for a full list of changes.

    
Version 0.4.0 - 2020-12-10:


    
General

    
    
        
  1. Add a textual summary of the analysis
    2. 
        
  2. Fix calculation of requested memory in kBytes
    3. 
        
  3. Fix issue that prevents units from being copied
    4. 
        
  4. Show additional information in process table
    5. 
        
  5. Add sorting process table
    6. 
        
  6. Fix: Trigger process isn't part of process table
    7. 
        
  7. Update to Transcrypt 3.7
    8. 
        
  8. Line "Killed process" can contain the process UID
    9. 
        
  9. Add drag-and-drop support for files
    10. 
        
  10. Add missing explanations
    11. 
        
  11. Allow more characters in program name
    12. 
        
  12. Bug fixes
    13. 
    


    
Note

    See the
    commit history
    of the repository for a full list of changes.

    
Version 0.3.0 - 2019-11-24:


    
General

    
    
        
  1. Improve presentation
    2. 
        
  2. Separate analysis and visualisation code
    3. 
        
  3. Use CSS classes to control the visibility
    4. 
        
  4. Strip columns left to the message automatically
    5. 
        
  5. Lot if internal improvements and restructuring
    6. 
        
  6. Bug fixes
    7. 
    


    
Note

    See the
    commit history
    of the repository for a full list of changes.





    
Local Installation

    (click to show / hide)





    Installing OOMAnalyser is quite easy since OOMAnalyser consists only of two files, a
    HTML file and a JavaScript file. Both can be stored locally to use OOMAnalyser
    without an Internet connection.

    
Installation steps

    
    
        
  1. Download the HTML file and the
            JavaScript file to the main directory
        
    2. 
        
  2. Open the file OOMAnalyser.html in your favourite browser.
    3. 
    









    OOMAnalyser  |
    Copyright (C) 2017-2021 Carsten Grohmann |
    License: MIT |
    Source Code on Sourcehut





    

        Copyright (c) 2017-2021 Carsten Grohmann mail <add at here> carstengrohmann.de
    

    

        Permission is hereby granted, free of charge, to any person obtaining a copy of
        this software and associated documentation files (the "Software"), to deal in
        the Software without restriction, including without limitation the rights to
        use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
        of the Software, and to permit persons to whom the Software is furnished to do
        so, subject to the following conditions:
    

    

        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
    

    

        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.