troubleshooting high cpu occupancy rate

Posted on | Edited on | In

20190327092247.png

find target thread

top find abnormal rate cpu usage process
top -p [pid] -H find this process’s abnormal thread
printf "%x\n" tid change thread number to hexadecimal
jstack [pid] > xxx.log dump thread information
vim xxx.log find target hex id thread in file

log analysis

The state in dump file

  1. Deadlock
  2. Runnable
  3. Waiting on condition
  4. Waiting on monitor entry
  5. Suspended
  6. Object.wait() or TIMED_WAITING
  7. Blocked
  8. Parked

Waiting to lock and Blocked

1
2
3
4
5
6
7
kafka-coordinator-heartbeat-thread | xxxxx" #106 daemon prio=5 os_prio=0 tid=0x00007fee64001800 nid=0x214 waiting for monitor entry [0x00007fee1b6f5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disableWakeups(ConsumerNetworkClient.java:409)
        - waiting to lock <0x00000000c2a1b340> (a org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:264)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:865)
        - locked <0x00000000c29fa1a8> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

Waiting for monitor entry and in Object.wait()

1
2
3
4
5
6
7
8
9
10
11
12
pool-1-thread-2" #18 prio=5 os_prio=0 tid=0x00007fee5da7c000 nid=0x7f85 waiting on condition [0x00007fee9cdd6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0b6db48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

in Obejct.wait() and TIMED_WAITING

1
2
3
4
5
kafka-coordinator-heartbeat-thread | xxxxx" #89 daemon prio=5 os_prio=0 tid=0x00007fee2c007000 nid=0x202 in Object.wait() [0x00007fee289c8000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:884)
        - locked <0x00000000c2cac000> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

https://www.cnblogs.com/heyanan/p/9261695.html
https://www.cnblogs.com/paul8339/p/7464206.html