85a82a4770
Extract code to display details from code to analyse/split the OOM. This should simplify later extensions as well as later written unit tests.
633 lines
21 KiB
HTML
633 lines
21 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
<head>
|
||
<script src="__javascript__/OOMAnalyser.js" defer="defer"></script>
|
||
<meta charset="UTF-8">
|
||
<title>OOM Analyser</title>
|
||
|
||
<style type='text/css'>
|
||
h1 {
|
||
font-size: 2em;
|
||
margin: 0.67em 0;
|
||
}
|
||
|
||
#notify_box {
|
||
display: none;
|
||
width: 100%;
|
||
}
|
||
|
||
.kbytes {
|
||
text-align: right;
|
||
}
|
||
.kbytes::after {
|
||
content: " kBytes";
|
||
}
|
||
|
||
.pages {
|
||
text-align: right;
|
||
}
|
||
.pages::after {
|
||
content: " pages";
|
||
}
|
||
|
||
.align_right {
|
||
text-align: right;
|
||
}
|
||
|
||
.hide_tablerow {
|
||
display: none;
|
||
}
|
||
|
||
.click_msg_in_th {
|
||
font-size: small;
|
||
font-weight: unset;
|
||
padding: unset;
|
||
}
|
||
|
||
table {
|
||
border-collapse: collapse;
|
||
padding: 10px;
|
||
table-layout: fixed;
|
||
text-align: left;
|
||
width: 100%;
|
||
}
|
||
|
||
svg {
|
||
display: block;
|
||
max-height: 200px;
|
||
}
|
||
|
||
th {
|
||
font-weight: bold;
|
||
font-size: large;
|
||
padding: 5px;
|
||
}
|
||
|
||
th, td {
|
||
border: 1px solid black;
|
||
word-wrap: break-word;
|
||
}
|
||
|
||
.terminal {
|
||
font-family: monospace;
|
||
overflow: auto;
|
||
}
|
||
|
||
.notify-warning {
|
||
color: #9F6000;
|
||
background-color: #FEEFB3;
|
||
}
|
||
.notify-error {
|
||
color: #D8000C;
|
||
background-color: #FFD2D2;
|
||
}
|
||
|
||
/* Zebra-Layout */
|
||
tr:nth-child(odd) td:nth-child(2), tr:nth-child(odd) td:nth-child(3) {
|
||
background-color: #f2f2f2;
|
||
}
|
||
|
||
.license {
|
||
font-size: small;
|
||
display: none;
|
||
}
|
||
|
||
</style>
|
||
<script>
|
||
function goBack() {
|
||
window.history.back();
|
||
}
|
||
</script>
|
||
</head>
|
||
<body>
|
||
|
||
|
||
<h1>Analyse and visualise Linux OOM output</h1>
|
||
|
||
<p>
|
||
This web page analyses Linux Out Of Memory (OOM) messages and shows these
|
||
information in a human friendly presentation.
|
||
</p>
|
||
<p>
|
||
Your input will be analysed locally - in your browser - by an integrated
|
||
JavaScript script without transferring your data to foreign servers.
|
||
</p>
|
||
|
||
<div id="input">
|
||
<h2>Step 1 - Enter your OOM message</h2>
|
||
<textarea id="textarea_oom" cols="100" rows="20" autocomplete="off" title="OOM input field">Add your OOMhere</textarea>
|
||
<br/>
|
||
<button onclick="OOMAnalyser.OOMDisplayInstance.analyse_and_show()" title="Analyse the OOM from the input area and show it">Analyse</button>
|
||
<button onclick="OOMAnalyser.OOMDisplayInstance.reset_form()" title="Clean the input area">Reset</button>
|
||
<button onclick="OOMAnalyser.OOMDisplayInstance.copy_example_to_form()" title="Copy an example OOM into the input area">Insert example</button>
|
||
</div>
|
||
|
||
<p>
|
||
<div class="terminal" id="notify_box"></div>
|
||
</p>
|
||
|
||
<div id="analysis">
|
||
|
||
<h2>Step 2 - Results</h2>
|
||
<p>
|
||
Go back to
|
||
<a href="javascript:void(0);" onclick="OOMAnalyser.OOMDisplayInstance.reset_form()" title="Run a new analysis">"Step 1 - Enter your OOM message"</a>
|
||
to run a new analysis.
|
||
</p>
|
||
<p>
|
||
The result of the analysis is displayed in three columns. The first column is used to name the property
|
||
including the original OOM identifier in brackets. The extracted information is displayed in the second column.
|
||
The last column contains further details and additional information.
|
||
</p>
|
||
<table>
|
||
<!--
|
||
<thead>
|
||
<tr>
|
||
<th scope="col"></th>
|
||
<th scope="col">Entry</th>
|
||
<th scope="col">Description</th>
|
||
</tr>
|
||
</thead>
|
||
-->
|
||
<tbody>
|
||
|
||
<!-- Trigger process -->
|
||
|
||
<tr>
|
||
<th scope="row" colspan="3">Tigger Process</th>
|
||
</tr>
|
||
<tr>
|
||
<td></td>
|
||
<td class="align_right"><span id="trigger_proc_name"></span> (PID <span id="trigger_proc_pid"></span>)</td>
|
||
<td>This process requests memory and is triggering thereby the OOM situation</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Memory allocation flags<br>(gfp_mask)</td>
|
||
<td id="trigger_proc_gfp_mask" class="align_right"></td>
|
||
<td>These flags are used to control the kernel internal memory allocation<br>
|
||
GFP stands for <code>__get_free_pages()</code> </td>
|
||
</tr>
|
||
<tr>
|
||
<td>Node mask to show on which CPU Cores this process can run<br>(nodemask)</td>
|
||
<td id="trigger_proc_nodemask" class="align_right"></td>
|
||
<td>Bit mask indicating the cores on which the process can run</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Requested memory<br>(order)</td>
|
||
<td class="align_right">
|
||
<span id="trigger_proc_requested_memory"></span> (2^<sup><span id="trigger_proc_order"></span></sup>) pages /
|
||
<span class="kbytes" id="trigger_proc_requested_memory_kbytes"></span>
|
||
</td>
|
||
<td>The kernel specifies the requested number of pages as exponent of power of two.
|
||
</tr>
|
||
<tr>
|
||
<td>Adjust oom-killer score<br>(oom_score_adj)</td>
|
||
<td id="trigger_proc_oomscore" class="align_right"></td>
|
||
<td>
|
||
This value is added to the badness score before it's used to determine the process to be killed.
|
||
</td>
|
||
</tr>
|
||
|
||
<!-- Killed Process -->
|
||
|
||
<tr>
|
||
<th scope="row" colspan="3">Killed Process</th>
|
||
</tr>
|
||
<tr>
|
||
<td></td>
|
||
<td class="align_right"><span id="killed_proc_name"></span> (PID <span id="killed_proc_pid"></span>)</td>
|
||
<td>Process killed by Linux kernel to satisfy the memory request</td>
|
||
</tr>
|
||
<tr>
|
||
<td>OOM Score<br>(score)</td>
|
||
<td id="killed_proc_score" class="align_right"></td>
|
||
<td></td>
|
||
</tr>
|
||
<tr>
|
||
<td>Virtual Memory <br> (total-vm) </td>
|
||
<td id="killed_proc_vm_kb" class="kbytes"></td>
|
||
<td>Virtual memory used by this process</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Resident anonymous memory <br> (anon-rss) </td>
|
||
<td id="killed_proc_anon_rss_kb" class="kbytes"></td>
|
||
<td>Part of the virtual process memory mapped into RAM</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Resident file mapping memory <br> (file-rss) </td>
|
||
<td id="killed_proc_file_rss_kb" class="kbytes"></td>
|
||
<td>Files which have been mapped into RAM (with
|
||
<a href="http://man7.org/linux/man-pages/man2/mmap.2.html">mmap(2)</a>)
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Resident shared memory <br> (shmem-rss) </td>
|
||
<td id="killed_proc_shmem_rss_kb" class="kbytes"></td>
|
||
<td>This may include System V shared memory and shared anonymous memory</td>
|
||
</tr>
|
||
|
||
<!-- Memory Usage Graphs -->
|
||
|
||
<tr>
|
||
<th scope="row" colspan="3">Memory Usage Graphs</th>
|
||
</tr>
|
||
<tr>
|
||
<td style="font-weight: bold">RAM Summary</td>
|
||
<td colspan="2"><div id="svg_ram"></div></td>
|
||
</tr>
|
||
<tr>
|
||
<td style="font-weight: bold">Swap Summary</td>
|
||
<td colspan="2"><div id="svg_swap"></div></td>
|
||
</tr>
|
||
|
||
|
||
<!-- Page Usage -->
|
||
|
||
<tr>
|
||
<td style="font-weight: bold">Memory Pages</td>
|
||
<td></td>
|
||
<td></td>
|
||
</tr>
|
||
<tr>
|
||
<td>RAM pages</td>
|
||
<td id="ram_pages" class="pages"></td>
|
||
<td>Total number of RAM pages
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>HighMem/MovableOnly</td>
|
||
<td id="highmem_pages" class="pages"></td>
|
||
<td>Number of pages in the High Memory Area or marked movable for Contiguous Memory Allocator (CMA).
|
||
<br>
|
||
HighMem pages are also counted in the total page number.
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Reserved pages</td>
|
||
<td id="reserved_pages" class="pages"></td>
|
||
<td>Number of reserved pages
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>CMA reserved pages</td>
|
||
<td id="cma_pages" class="pages">0</td>
|
||
<td>Pages reserved for Contiguous Memory Allocator (CMA)
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Pagetable Cache</td>
|
||
<td id="pagetablecache_pages" class="pages">0</td>
|
||
<td>Number of pages in pagetable cache
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Number of pages with hardware errors</td>
|
||
<td id="hwpoisoned_pages" class="pages">0</td>
|
||
<td>Pages with uncorrectable memory errors
|
||
</td>
|
||
</tr>
|
||
|
||
<!-- Memory Usage Details -->
|
||
|
||
<tr>
|
||
<td style="font-weight: bold">Memory Usage Details</td>
|
||
<td></td>
|
||
<td></td>
|
||
</tr>
|
||
<tr>
|
||
<td>Active anonymous memory <br> (active_anon) </td>
|
||
<td id="active_anon_pages" class="pages"></td>
|
||
<td>Recently used anonymous memory.<br>
|
||
These memory pages will usually not swapped out.
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Inactive anonymous memory <br> (inactive_anon)</td>
|
||
<td id="inactive_anon_pages" class="pages"></td>
|
||
<td>Least recently used anonymous memory.<br>
|
||
These memory pages can be swapped out.
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Isolated anonymous memory <br> (isolated_anon)</td>
|
||
<td id="isolated_anon_pages" class="pages"></td>
|
||
<td>Memory isolation is used to separate memory between different virtual machines.</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Active Pagecache <br> (active_file)</td>
|
||
<td id="active_file_pages" class="pages"></td>
|
||
<td>Pagecache that has been used more recently and usually not reclaimed unless absolutely necessary.</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Inactive Pagecache <br> (inactive_file)</td>
|
||
<td id="inactive_file_pages" class="pages"></td>
|
||
<td>Pagecache which has been less recently used. It can be reclaimed without huge performance impact.</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Isolated Pagecache <br> (isolated_file)</td>
|
||
<td id="isolated_file_pages" class="pages"></td>
|
||
<td>Memory isolation is used to separate memory between different virtual machines.</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Unevictable Pages <br> (unevictable)</td>
|
||
<td id="unevictable_pages" class="pages"></td>
|
||
<td>Unevictable memory. It can't be swapped out because the pages are owned by ramfs or protected by
|
||
<a href="http://man7.org/linux/man-pages/man3/mlock.5.html" target="_blank">mlock(3)</a> /
|
||
<a href="http://man7.org/linux/man-pages/man2/shmctl.2.html" target="_blank">shmctl(SHM_LOCK)</a>.
|
||
Unevictable pages are managed by kernels LRU framework.
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Dirty Pages <br> (dirty)</td>
|
||
<td id="dirty_pages" class="pages"></td>
|
||
<td>Memory which is waiting to get written back to the disk.
|
||
<sup><a href="#footnote-proc5">[1]</a></sup>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Writeback <br> (writeback)</td>
|
||
<td id="writeback_pages" class="pages"></td>
|
||
<td>
|
||
Memory which is actively being written back to the disk.
|
||
<sup><a href="#footnote-proc5">[1]</a></sup>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Unstable <br> (unstable)</td>
|
||
<td id="unstable_pages" class="pages"></td>
|
||
<td>Not yet committed to stable storage.</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Slab Reclaimable <br> (slab_reclaimable)</td>
|
||
<td id="slab_reclaimable_pages" class="pages"></td>
|
||
<td>
|
||
Slab is a in-kernel data structures cache. Part of Slab, that might be reclaimed, such as caches.
|
||
<sup><a href="#footnote-proc5">[1]</a></sup>
|
||
<br>
|
||
Additional details are listed in
|
||
<a href="http://man7.org/linux/man-pages/man5/slabinfo.5.html" target="_blank">slabinfo(5)</a> also.
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Slab Unreclaimable <br> (slab_unreclaimable)</td>
|
||
<td id="slab_unreclaimable_pages" class="pages"></td>
|
||
<td>
|
||
Part of Slab, that cannot be reclaimed on memory pressure.
|
||
<sup><a href="#footnote-proc5">[1]</a></sup>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Mapped <br> (mapped)</td>
|
||
<td id="mapped_pages" class="pages"></td>
|
||
<td>
|
||
Files which have been mapped into memory (with
|
||
<a href="http://man7.org/linux/man-pages/man2/mmap.2.html">mmap(2)</a>), such as libraries.
|
||
<sup><a href="#footnote-proc5">[1]</a></sup>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Shared Memory <br> (shmem)</td>
|
||
<td id="shmem_pages" class="pages"></td>
|
||
<td>
|
||
Amount of memory consumed in
|
||
<a href="http://man7.org/linux/man-pages/man5/tmpfs.5.html">tmpfs(5)</a>
|
||
filesystems.
|
||
<sup><a href="#footnote-proc5">[1]</a></sup>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Pagetables <br> (pagetables)</td>
|
||
<td id="pagetables_pages" class="pages"></td>
|
||
<td>
|
||
Amount of memory dedicated to the lowest level of pagetables.
|
||
<sup><a href="#footnote-proc5">[1]</a></sup>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Bounce <br> (bounce)</td>
|
||
<td id="bounce_pages" class="pages"></td>
|
||
<td>
|
||
Memory used for block device "bounce buffers".
|
||
<sup><a href="#footnote-proc5">[1]</a></sup>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>free <br> (free)</td>
|
||
<td id="free_pages" class="pages"></td>
|
||
<td></td>
|
||
</tr>
|
||
<tr>
|
||
<td>free_pcp <br> (free_pcp)</td>
|
||
<td id="free_pcp_pages" class="pages"></td>
|
||
<td></td>
|
||
</tr>
|
||
<tr>
|
||
<td>free_cma <br> (free_cma)</td>
|
||
<td id="free_cma_pages" class="pages"></td>
|
||
<td></td>
|
||
</tr>
|
||
<tr>
|
||
<td>Total Pagecache</td>
|
||
<td id="pagecache_total_pages" class="pages"></td>
|
||
<td></td>
|
||
</tr>
|
||
|
||
<!-- Swap Usage -->
|
||
|
||
<tr>
|
||
<td style="font-weight: bold">Swap Usage</td>
|
||
<td></td>
|
||
<td></td>
|
||
</tr>
|
||
<tr>
|
||
<td>Swap Total</td>
|
||
<td id="swap_total_kb" class="kbytes"></td>
|
||
<td>Total amount of swap space available.
|
||
<sup><a href="#footnote-proc5">[1]</a></sup>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Swap Free</td>
|
||
<td id="swap_free_kb" class="kbytes"></td>
|
||
<td>Amount of swap space that is currently unused.
|
||
<sup><a href="#footnote-proc5">[1]</a></sup>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Swap Cached</td>
|
||
<td id="swap_cache_kb" class="kbytes"></td>
|
||
<td>Memory that once was swapped out, is swapped back in
|
||
but still also is in the swap file. (If memory pres‐
|
||
sure is high, these pages don't need to be swapped out
|
||
again because they are already in the swap file. This
|
||
saves I/O).
|
||
<sup><a href="#footnote-proc5">[1]</a></sup>
|
||
</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Swap Used</td>
|
||
<td id="swap_used_kb" class="kbytes"></td>
|
||
<td>Amount of used swap space w/o cached swap <br>
|
||
(<code>SwapUsed = SwapTotal - SwapFree -SwapCache</code>)
|
||
</td>
|
||
</tr>
|
||
|
||
<!-- Operating System -->
|
||
|
||
<tr>
|
||
<th scope="row" colspan="3">Operating System</th>
|
||
</tr>
|
||
<tr>
|
||
<td>Kernel</td>
|
||
<td id="kernel_version" class="align_right"></td>
|
||
<td></td>
|
||
</tr>
|
||
<tr>
|
||
<td>Distribution</td>
|
||
<td id="dist" class="align_right"></td>
|
||
<td>Guessed from the kernel version</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Platform</td>
|
||
<td id="platform" class="align_right"></td>
|
||
<td>Guessed from the kernel version</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Page size</td>
|
||
<td id="page_size" class="kbytes"></td>
|
||
<td>Guessed</td>
|
||
</tr>
|
||
|
||
|
||
<!-- Memory Chunks -->
|
||
|
||
<tr>
|
||
<th scope="row" colspan="3">Memory Chunks</th>
|
||
</tr>
|
||
<tr>
|
||
<td></td>
|
||
<td colspan="2" class="terminal">
|
||
<pre id="mem_node_info"></pre>
|
||
</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<th scope="row" colspan="3">Process Table</th>
|
||
</tr>
|
||
<tr>
|
||
<td></td>
|
||
<td colspan="2" class="terminal">
|
||
<pre id="process_table"></pre>
|
||
</td>
|
||
</tr>
|
||
|
||
<!-- Hardware Details -->
|
||
|
||
<tr>
|
||
<th scope="row" colspan="3">Hardware Details</th>
|
||
</tr>
|
||
<tr>
|
||
<td></td>
|
||
<td colspan="2" class="terminal">
|
||
<pre id="hardware_info"></pre>
|
||
</td>
|
||
</tr>
|
||
|
||
<!-- Kernel Call Tree -->
|
||
|
||
<tr>
|
||
<th scope="row" colspan="3">Kernel Call Trace</th>
|
||
</tr>
|
||
<tr>
|
||
<td></td>
|
||
<td colspan="2" class="terminal">
|
||
<pre id="call_trace"></pre>
|
||
</td>
|
||
</tr>
|
||
|
||
<!-- Initial OOM -->
|
||
|
||
<tr>
|
||
<th scope="row" colspan="3">Entire OOM Message
|
||
<span class="click_msg_in_th">
|
||
<a href="javascript:void(0);" id="oom_toogle_msg" onclick="OOMAnalyser.OOMDisplayInstance.toggle_oom()" title="Click to show/hide full OOM message">(click to hide)</a>
|
||
</span>
|
||
</th>
|
||
</tr>
|
||
<tr>
|
||
<td></td>
|
||
<td colspan="2" class="terminal">
|
||
<pre id="oom"></pre>
|
||
</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
|
||
<p>
|
||
Go back to
|
||
<a href="javascript:void(0);" onclick="OOMAnalyser.OOMDisplayInstance.reset()" title="Run a new analysis">"Step 1 - Enter your OOM message"</a>
|
||
to run a new analysis.
|
||
</p>
|
||
|
||
<h2>Footnotes</h2>
|
||
<ol>
|
||
<li id="footnote-proc5"><cite><a href="http://man7.org/linux/man-pages/man5/proc.5.html" target="_blank">
|
||
proc(5) - process information pseudo-filesystem</a></cite> (<a href="#" onclick="goBack()">Go Back</a>)
|
||
</li>
|
||
</ol>
|
||
</div>
|
||
|
||
<h2>Further Information</h2>
|
||
<ol>
|
||
<li>
|
||
<a href="https://man7.org/">Linux man pages online</a>
|
||
</li>
|
||
<li>
|
||
<a href="https://utcc.utoronto.ca/~cks/space/blog/linux/DecodingPageAllocFailures">
|
||
Decoding the Linux kernel's page allocation failure messages
|
||
</a>
|
||
</li>
|
||
<li>
|
||
<a href="http://elearningmedium.com/linux-kernel-oom-log-analysis/">Linux Kernel OOM Log Analysis</a>
|
||
</li>
|
||
</ol>
|
||
|
||
<hr/>
|
||
|
||
<div class="footer">
|
||
OOMAnalyser <span id="version"></span> |
|
||
Copyright (C) 2017-2018 Carsten Grohmann |
|
||
<a href="javascript:void(0);" onclick="OOMAnalyser.toggle('license')" title="Show / hide license text">License: MIT</a> |
|
||
<a href="https://github.com/CarstenGrohmann/OOMAnalyser" title="Source code on GitHub">Source Code on GitHub</a>
|
||
</div>
|
||
|
||
<div class="license" id="license">
|
||
<p>
|
||
Copyright (c) 2017-2018 Carsten Grohmann mail <add at here> carsten-grohmann.de
|
||
</p>
|
||
<p>
|
||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||
this software and associated documentation files (the "Software"), to deal in
|
||
the Software without restriction, including without limitation the rights to
|
||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
||
of the Software, and to permit persons to whom the Software is furnished to do
|
||
so, subject to the following conditions:
|
||
</p>
|
||
<p>
|
||
The above copyright notice and this permission notice shall be included in all
|
||
copies or substantial portions of the Software.
|
||
</p>
|
||
<p>
|
||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||
SOFTWARE.
|
||
</p>
|
||
</div>
|
||
|
||
</body>
|
||
</html>
|