Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > futex high system utilization

Reply
Thread Tools

futex high system utilization

 
 
Jaydeep Chovatia
Guest
Posts: n/a
 
      07-17-2012
Hi,

In my c++ (Linux) application I am seeing high CPU utilization (almost 90%)and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run “strace” command and found that around 80% of time is being spent in “futex”system call, please see snippet of strace here:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
80.66 100.007743 1567 63806 22761 futex
9.15 11.340274 96925 117 4 restart_syscall
5.60 6.939598 4401 1577 poll

Initially I though this is because of pthread_lock, unlock, condition timedwait, condition wait, etc. system calls I am using in the application, to prove that I have overridden pthread_lock, unlock, condition timedwait, condition wait, unlock, etc. using LD_PRELOAD functionality and concluded that not many threads are waiting here (mutex->__data.__nusers remains less than5 most of the time), so it doesn’t seem that these sys calls are part ofthe problem.

Now I am running out of clue about how to find out code location/system call which is causing this futex high utilization, any help on this would be appreciated.

Thank you,
Jaydeep
 
Reply With Quote
 
 
 
 
Jorgen Grahn
Guest
Posts: n/a
 
      07-17-2012
On Tue, 2012-07-17, Jaydeep Chovatia wrote:
> Hi,
>
> In my c++ (Linux) application I am seeing high CPU utilization
> (almost 90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried
> to run ?strace? command and found that around 80% of time is being
> spent in ?futex? system call,


I'm not sure you can use strace to find that out -- strace makes
syscalls much more expensive.

> please see snippet of strace here:


Please break your paragraphs into lines, so I don't have to do it for
you.

> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 80.66 100.007743 1567 63806 22761 futex
> 9.15 11.340274 96925 117 4 restart_syscall
> 5.60 6.939598 4401 1577 poll
>


> Initially I though this is because of pthread_lock, unlock,
> condition timedwait, condition wait, etc. system calls I am using in
> the application, to prove that I have overridden pthread_lock, unlock,
> condition timedwait, condition wait, unlock, etc. using LD_PRELOAD
> functionality and concluded that not many threads are waiting here
> (mutex->__data.__nusers remains less than 5 most of the time), so it
> doesn?t seem that these sys calls are part of the problem.


> Now I am running out of clue about how to find out code
> location/system call which is causing this futex high utilization, any
> help on this would be appreciated.


This is a question which you should ask in some Linux newsgroup.

I have only one suggestion to offer: try ltrace too. It traces
library calls instead of system calls. Both strace and ltrace are
very useful on Linux.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
 
Reply With Quote
 
 
 
 
Miquel van Smoorenburg
Guest
Posts: n/a
 
      07-17-2012
In article <(E-Mail Removed)>,
Jaydeep Chovatia <(E-Mail Removed)> wrote:
>In my c++ (Linux) application I am seeing high CPU utilization (almost
>90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run


Is this a machine that has been running for a few weeks at least,
and runs ntpd? If so the system might be suffering from a
leap-second-adjustment bug .. as root run date -s "`date`" or
reboot, then see if your problem is gone.

Mike.
 
Reply With Quote
 
Jorgen Grahn
Guest
Posts: n/a
 
      07-17-2012
On Tue, 2012-07-17, Miquel van Smoorenburg wrote:
> In article <(E-Mail Removed)>,
> Jaydeep Chovatia <(E-Mail Removed)> wrote:
>>In my c++ (Linux) application I am seeing high CPU utilization (almost
>>90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run

>
> Is this a machine that has been running for a few weeks at least,
> and runs ntpd? If so the system might be suffering from a
> leap-second-adjustment bug .. as root run date -s "`date`" or
> reboot, then see if your problem is gone.


Do you have any URL for that? I was aware of the recent leap second,
but haven't heard about that triggering ntpd bugs. Also I'm always
interested in new ways to botch time calculations in software

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
 
Reply With Quote
 
Miquel van Smoorenburg
Guest
Posts: n/a
 
      07-17-2012
In article <(E-Mail Removed)>,
Jorgen Grahn <(E-Mail Removed)> wrote:
>On Tue, 2012-07-17, Miquel van Smoorenburg wrote:
>> In article <(E-Mail Removed)>,
>> Jaydeep Chovatia <(E-Mail Removed)> wrote:
>>>In my c++ (Linux) application I am seeing high CPU utilization (almost
>>>90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run

>>
>> Is this a machine that has been running for a few weeks at least,
>> and runs ntpd? If so the system might be suffering from a
>> leap-second-adjustment bug .. as root run date -s "`date`" or
>> reboot, then see if your problem is gone.

>
>Do you have any URL for that? I was aware of the recent leap second,
>but haven't heard about that triggering ntpd bugs. Also I'm always
>interested in new ways to botch time calculations in software


It isn't a bug in ntpd, it's just that if you run ntpd it would
have told the kernel to handle the extra 2012-06-30 23:59:60 leapsecond,
and that triggered a bug in the kernels timer implementation.
It crashed older kernels, and on newer kernels it caused applications
using futexes to use 100% cpu ..

See for example

http://www.devand.com/index.php/tech...2-tech/11-leap

or Google for "linux leap second bug 2012".

Mike.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
blocked on futex msoulier Python 5 03-29-2009 02:12 PM
RandomAccessFile and futex ETIMEDOUT ghostwhoowalks@gmail.com Java 4 12-18-2007 07:32 AM
high CPU utilization Miguel Moreno Cisco 4 01-30-2004 12:39 AM
2600XM, Frame Relay, and High CPU Utilization Donald Zelenak Jr. Cisco 5 12-06-2003 01:18 AM
High utilization on network with Cisco 4500M LAN Interface Sarah Cisco 3 12-01-2003 10:09 PM



Advertisments