Semicolon & Sons

Debugging a 100 percent CPU process

This is part of the Semicolon&Sons Code Diary - consisting of lessons learned on the job. You're in the unix category.

Last Updated: 2024-04-18

A ruby process 62461 is using 100% CPU.

What files is it accessing?

lsof -p 62461 shows it has like 20 pipes open.

COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF                NODE NAME


# Ones with links within this process

ruby    62461 jack    5   PIPE 0xa90da6eea6d9d9e2    16384                     ->0x28603f8608b5cee5
ruby    62461 jack    6   PIPE 0x28603f8608b5cee5    16384                     ->0xa90da6eea6d9d9e2
ruby    62461 jack    7   PIPE 0x2374675c4dde837d    16384                     ->0x632f23db24d442ab
ruby    62461 jack    8   PIPE 0x632f23db24d442ab    16384                     ->0x2374675c4dde837d
ruby    62461 jack    9   PIPE 0x275c79d087765990    16384                     ->0xf3661af9a1b1a84f
ruby    62461 jack   10   PIPE 0xf3661af9a1b1a84f    16384                     ->0x275c79d087765990
ruby    62461 jack   11   PIPE 0xd923c389e6087315    16384                     ->0x87a5f4ab3254ce0a
ruby    62461 jack   12   PIPE 0x87a5f4ab3254ce0a    16384                     ->0xd923c389e6087315

# Ones without links within this process

ruby    62461 jack   13   PIPE 0x133fc4dbb211d5ac    65536                     ->0x74a3f590379e18e
ruby    62461 jack   16   PIPE  0x658a59aa609b972    16384                     ->0xc2a5c21c06ab4f73
ruby    62461 jack   17   PIPE 0x4698523d30043181    16384                     ->0x349f9ef77d92611b
ruby    62461 jack   18   PIPE 0xcbe7385de741b2a8    16384                     ->0xfe057f18940158de
ruby    62461 jack   19   PIPE 0x1eea3de7c654428b    16384                     ->0x4c4fbae0d47b40b7
ruby    62461 jack   20   PIPE 0x7229689197020e20    16384                     ->0xfd279dd9328508e
ruby    62461 jack   21   PIPE 0xbb32609e662b9f22    16384                     ->0xf4ce2b44ab2a7425
ruby    62461 jack   22   PIPE 0x4a26be585b9f26a0    16384                     ->0xaabeac49c2088b5a
ruby    62461 jack   23   PIPE  0xcb786d951de115a    16384                     ->0xb7d71d6ea988101c
ruby    62461 jack   24   PIPE 0x4a2fdd60e5a3b1bf    16384                     ->0x1a8739b6f9454419
ruby    62461 jack   29   PIPE 0xcdbf6cc579e78491    16384                     ->0xf448741496d09034

When debugging pipes, it's important to see where each pipe leads to. According to the lsof man page, each one's endpoint is written in the NAME column (last one)

I can see above that many of these pipes connect to this same process. But more again lead elsewhere. How to figure out where, e.g. the last one leads?

Simply run lsof without any filters (all pipes on computer) and grep for the pipe memory address.

Staring with first column "device" pipe:

$ lsof | grep 0xcdbf6cc579e78491

fsevent_w 45891 jack    1      PIPE 0xf448741496d09034      16384                     ->0xcdbf6cc579e78491
ruby      62461 jack   29      PIPE 0xcdbf6cc579e78491      16384                     ->0xf448741496d09034

We see that this pipe gets its input from fsevent a famously problematic tool.

If you want to see the endpoint PID etc. for every single pipe in lsof use lsof +E (personally I could not get working in macos at time of writing)

What operations is the process performing?

In macos, dstruss to the rescue (strace in linux):

sudo dtruss -p  62461

SYSCALL(args)        = return
sigprocmask(0x1, 0x0, 0x700002B39D60)        = 0x0 0
sigaltstack(0x0, 0x700002B39D50, 0x0)        = 0 0
open_nocancel("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/util\0", 0x1100004, 0x24B400)      = 14 0
dtrace: 36724 dynamic variable drops with non-empty dirty list
fstatfs64(0xE, 0x700002B394D8, 0x0)      = 0 0
lseek(0xE, 0x0, 0x0)         = 0 0
fgetattrlist(0xE, 0x700002B39EA8, 0x700002B39EC0)        = 0 0
getdirentries64(0xE, 0x7FEAD29F4000, 0x2000)         = 272 0
close_nocancel(0xE)      = 0 0
sigprocmask(0x1, 0x0, 0x700002B39980)        = 0x0 0
sigaltstack(0x0, 0x700002B39970, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/util/stripComments.js\0", 0x700002B39A58, 0x0)       = 0 0
sigprocmask(0x1, 0x0, 0x700002B397E0)        = 0x0 0
sigaltstack(0x0, 0x700002B397D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/util/stripComments.js\0", 0x700002B398B8, 0x0)       = 0 0
sigprocmask(0x1, 0x0, 0x700002B39980)        = 0x0 0
sigaltstack(0x0, 0x700002B39970, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/util/ensureObject.js\0", 0x700002B39A58, 0x0)        = 0 0
sigprocmask(0x1, 0x0, 0x700002B397E0)        = 0x0 0
sigaltstack(0x0, 0x700002B397D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/util/ensureObject.js\0", 0x700002B398B8, 0x0)        = 0 0
sigprocmask(0x1, 0x0, 0x700002B39980)        = 0x0 0
sigaltstack(0x0, 0x700002B39970, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/util/index.js\0", 0x700002B39A58, 0x0)       = 0 0
sigprocmask(0x1, 0x0, 0x700002B397E0)        = 0x0 0
sigaltstack(0x0, 0x700002B397D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/util/index.js\0", 0x700002B398B8, 0x0)       = 0 0
sigprocmask(0x1, 0x0, 0x700002B39980)        = 0x0 0
sigaltstack(0x0, 0x700002B39970, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/util/getProp.js\0", 0x700002B39A58, 0x0)         = 0 0
sigprocmask(0x1, 0x0, 0x700002B397E0)        = 0x0 0
sigaltstack(0x0, 0x700002B397D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/util/getProp.js\0", 0x700002B398B8, 0x0)         = 0 0
sigprocmask(0x1, 0x0, 0x700002B39980)        = 0x0 0
sigaltstack(0x0, 0x700002B39970, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/util/unesc.js\0", 0x700002B39A58, 0x0)       = 0 0
sigprocmask(0x1, 0x0, 0x700002B397E0)        = 0x0 0
sigaltstack(0x0, 0x700002B397D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/util/unesc.js\0", 0x700002B398B8, 0x0)       = 0 0
sigprocmask(0x1, 0x0, 0x700002B39D60)        = 0x0 0
sigaltstack(0x0, 0x700002B39D50, 0x0)        = 0 0
open_nocancel("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors\0", 0x1100004, 0x24B400)         = 14 0
fstatfs64(0xE, 0x700002B394D8, 0x0)      = 0 0
lseek(0xE, 0x0, 0x0)         = 0 0
fgetattrlist(0xE, 0x700002B39EA8, 0x700002B39EC0)        = 0 0
getdirentries64(0xE, 0x7FEAD29F4000, 0x2000)         = 792 0
close_nocancel(0xE)      = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/selector.js\0", 0x700002B399B8, 0x0)       = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/selector.js\0", 0x700002B39818, 0x0)       = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/pseudo.js\0", 0x700002B399B8, 0x0)         = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/pseudo.js\0", 0x700002B39818, 0x0)         = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/types.js\0", 0x700002B399B8, 0x0)      = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/types.js\0", 0x700002B39818, 0x0)      = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/className.js\0", 0x700002B399B8, 0x0)      = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/className.js\0", 0x700002B39818, 0x0)      = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/combinator.js\0", 0x700002B399B8, 0x0)         = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/combinator.js\0", 0x700002B39818, 0x0)         = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/nesting.js\0", 0x700002B399B8, 0x0)        = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/nesting.js\0", 0x700002B39818, 0x0)        = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/index.js\0", 0x700002B399B8, 0x0)      = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/index.js\0", 0x700002B39818, 0x0)      = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/guards.js\0", 0x700002B399B8, 0x0)         = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/guards.js\0", 0x700002B39818, 0x0)         = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/id.js\0", 0x700002B399B8, 0x0)         = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/id.js\0", 0x700002B39818, 0x0)         = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/comment.js\0", 0x700002B399B8, 0x0)        = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/comment.js\0", 0x700002B39818, 0x0)        = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/attribute.js\0", 0x700002B399B8, 0x0)      = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/attribute.js\0", 0x700002B39818, 0x0)      = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/namespace.js\0", 0x700002B399B8, 0x0)      = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/namespace.js\0", 0x700002B39818, 0x0)      = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/node.js\0", 0x700002B399B8, 0x0)       = 0 0
sigprocmask(0x1, 0x0, 0x700002B39740)        = 0x0 0
sigaltstack(0x0, 0x700002B39730, 0x0)        = 0 0
lstat64("/Users/jack/code/semicolon/webapp/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors/node.js\0", 0x700002B39818, 0x0)       = 0 0
sigprocmask(0x1, 0x0, 0x700002B398E0)        = 0x0 0
sigaltstack(0x0, 0x700002B398D0, 0x0)

This already shows us something very suspicious - many of the arguments to these kernel function are in node_modules. These should ABSOLUTELY be ignored!

Aside: What do some of these systems calls do?

With reference to their man pages:

lsstat64 - could find an analogue "lstat" - this gets file stat structure - e.g. time last modified, user id of owner, size

signalprocmask - examine and change the signal mask of the calling thread. The signal mask is the set of signals whose delivery is currently blocked for the caller. Returns 0 on success (which we see above)

signaltstack - Allows a process to define a new alternate signal stack and/or retrieve the state of an existing alternate one. An alternate signal stack is used during execution of a signal handler if the establishment of that handler requested it. The most common usage of an alternate signal stack is to handle the SIGSEGV signal that is generated if the space available for the normal process stack is exhausted: in this case, a signal handler for SIGSEGV cannot be invoked on the process stack; if we wish to handle it, we must use an alternate signal stack.

What is memory compression?

When the routines used for memory compression can be offloaded to other processor cores, you're not likely to notice any performance hit when memory needs to be compressed or decompressed.

Purpose: reduces virtual memoory usage and can increaes speed by avoiding paging to HD by effectively making more memory available

Example use: a photo editing app you stop focusing on because you want to watch Netflix. i.e works on inactive memory.

Memory compression is coming back thanks to inexpensive multiple core processors.

Resources

Get Episode Alerts and Freebies

We publish a video once every week, on Sundays. Every 2nd video is completely free to watch.

If you'd like, we can remind you via email when a new episode is released. We'll also keep you up to date about the top-secret game we're developing for learning programming.

Semicolon & Sons

© 2024 Semicolon & Sons. All rights reserved.