Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > Evaluating static analysis and Dynamic analysis tools for C/C++

Reply
Thread Tools

Evaluating static analysis and Dynamic analysis tools for C/C++

 
 
ssubbarayan
Guest
Posts: n/a
 
      10-29-2009
Dear all,
Whats the general norm people use to evaluate static code analysis
and dynamic code analysis tools in your experience.I am confused on
the best tool to choose from,given that so many tools are available
over the net.I believe
following are the criteria to test the usefullness of these tools:
1)User friendliness
2)Ability to detect bugs
3)Ability to enforce coding guidelines
4)Ability to generate userfriendly reports
5)Speed of detection
6)Should not be providing lot of false defects.
7)Easily customisable

Are there any test suites available to evaluate these analysis tools?
How do people who use them evaluate?Are there any test programs
available for eg a sample C code which can be run with analysis tool
to see how it reports?

Please let me know your thoughts.Advance thanks for all your inputs.
Are there any bench marks available for these analysis tools?

Looking farward for your inputs and advanced thanks,
Regards,
s.subbarayan
 
Reply With Quote
 
 
 
 
romain.gaucher@gmail.com
Guest
Posts: n/a
 
      10-29-2009
On Oct 29, 8:07*am, ssubbarayan <(E-Mail Removed)> wrote:
> Dear all,
> Whats the general norm people use to evaluate static code *analysis
> and dynamic code analysis tools in your experience.I am confused on
> the best tool to choose from,given that so many tools are available
> over the net.I believe
> following are the criteria to test the usefullness of these tools:
> 1)User friendliness
> 2)Ability to detect bugs
> 3)Ability to enforce coding guidelines
> 4)Ability to generate userfriendly reports
> 5)Speed of detection
> 6)Should not be providing lot of false defects.
> 7)Easily customisable
>
> Are there any test suites available to evaluate these analysis tools?
> How do people who use them evaluate?Are there any test programs
> available for eg a sample C code which can be run with analysis tool
> to see how it reports?
>
> Please let me know your thoughts.Advance thanks for all your inputs.
> Are there any bench marks available for these analysis tools?
>
> Looking farward for your inputs and advanced thanks,
> Regards,
> s.subbarayan


In regards of static analysis tools:

NIST SAMATE project has some test suites for C, C++, Java...
You can reach it here:
http://samate.nist.gov/SRD

These are syntactic test cases, so they do not represent properly the
result of a tool on your code base. It just gives you an idea of the
weaknesses coverage of the tool (tools should also provide a list of
weaknesses they support, you can make sense of it with the CWE -
http://cwe.mitre.org)

As criteria to select a tool, I think it depends on lot on how you
plan to use to tool.
For example, if only few people (software security folks) use the
tool, then usability shouldn't be such a big deal; it is if many
developers will use the tool.

Otherwise, I would recommend few things:
- proper detection with few false-positive rate on selected test cases
- take some of your code (restrict the scope of the scan), and compare
tool results and look for false-negative/false-positive on your
code... (tools are sensitive to the code constructs/API used in the
code)
- customization (especially if you see an important FP/FN rate) might
be considered as important too; I suppose it depends on how you want
to use the tool...

Romain
 
Reply With Quote
 
 
 
 
Boudewijn Dijkstra
Guest
Posts: n/a
 
      10-29-2009
Op Thu, 29 Oct 2009 13:07:55 +0100 schreef ssubbarayan <(E-Mail Removed)>:
> Dear all,
> Whats the general norm people use to evaluate static code analysis
> and dynamic code analysis tools in your experience.I am confused on
> the best tool to choose from,given that so many tools are available
> over the net.I believe
> following are the criteria to test the usefullness of these tools:
> 1)User friendliness
> 2)Ability to detect bugs
> 3)Ability to enforce coding guidelines
> 4)Ability to generate userfriendly reports
> 5)Speed of detection
> 6)Should not be providing lot of false defects.
> 7)Easily customisable
>
> Are there any test suites available to evaluate these analysis tools?
> How do people who use them evaluate?Are there any test programs
> available for eg a sample C code which can be run with analysis tool
> to see how it reports?


Most companies who evaluate such a tool, use their own code base or a
representative, well-known portion of it.

> Please let me know your thoughts.Advance thanks for all your inputs.
> Are there any bench marks available for these analysis tools?


1)You cannot benchmark user friendliness.
2)There is no list of a respectable portion of all known bugs.
3)Coding guidelines have test suites, but sometimes interpretation differs.
4)You cannot benchmark user friendliness.
5)Speed of detection depends on code structure and style and is nearly
irrelevant for night-time testing.
6)There is no list of a respectable portion of all known false positives
of all coding guidelines.
7)You cannot benchmark user experience.


--
Gemaakt met Opera's revolutionaire e-mailprogramma:
http://www.opera.com/mail/
(remove the obvious prefix to reply by mail)
 
Reply With Quote
 
Hans-Bernhard Bröker
Guest
Posts: n/a
 
      10-29-2009
ssubbarayan wrote:

> Whats the general norm people use to evaluate static code analysis
> and dynamic code analysis tools in your experience.


There isn't one. People ask different things of such tools, and thus
value their properties differently. One person's strictly necessary
feature is another's gratuitous gimmick. One person's show-stopping
limitation is the next one's barely noticeable glitch.

> 1)User friendliness


As the saying goes: <insert program name here> is extremely
user-friendly. It's just a little picky who it makes friends with.

> 2)Ability to detect bugs


Depends on what kinds of bugs there are for the detecting.

> 3)Ability to enforce coding guidelines


Depends on the coding guideline in question.

> 4)Ability to generate userfriendly reports


There's no such thing as a user-friendly report.

> 5)Speed of detection


Depends on usage pattern. Unless the processing time runs considerably
north of one whole day, usage patterns will adapt to the tool, once its
usage has been prescribed.

> 6)Should not be providing lot of false defects.


See answer to 2).

> 7)Easily customisable


Depending on what you're trying to do, there's a point to be made that
the tools should not _need_ any "customization" ... it should find any
an all problems it can, period.

> Are there any test suites available to evaluate these analysis tools?


I rather much doubt it.

> How do people who use them evaluate?


Initially by throwing at it a significant amount of the worst, the best,
and the most typical code they can find at it. The real evaluation only
comes from actual long-time usage. The proof of the pudding, as they
say, is in the eating.
 
Reply With Quote
 
ssubbarayan
Guest
Posts: n/a
 
      10-30-2009
On Oct 30, 4:11*am, Hans-Bernhard Bröker <(E-Mail Removed)>
wrote:
> ssubbarayan wrote:
> > Whats the general norm people use to evaluate static code *analysis
> > and dynamic code analysis tools in your experience.

>
> There isn't one. *People ask different things of such tools, and thus
> value their properties differently. *One person's strictly necessary
> feature is another's gratuitous gimmick. *One person's show-stopping
> limitation is the next one's barely noticeable glitch.
>
> > 1)User friendliness

>
> As the saying goes: <insert program name here> is extremely
> user-friendly. *It's just a little picky who it makes friends with.
>
> > 2)Ability to detect bugs

>
> Depends on what kinds of bugs there are for the detecting.
>
> > 3)Ability to enforce coding guidelines

>
> Depends on the coding guideline in question.
>
> > 4)Ability to generate userfriendly reports

>
> There's no such thing as a user-friendly report.
>
> > 5)Speed of detection

>
> Depends on usage pattern. *Unless the processing time runs considerably
> north of one whole day, usage patterns will adapt to the tool, once its
> usage has been prescribed.
>
> > 6)Should not be providing lot of false defects.

>
> See answer to 2).
>
> > 7)Easily customisable

>
> Depending on what you're trying to do, there's a point to be made that
> the tools should not _need_ any "customization" ... it should find any
> an all problems it can, period.
>
> > Are there any test suites available to evaluate these analysis tools?

>
> I rather much doubt it.
>
> > How do people who use them evaluate?

>
> Initially by throwing at it a significant amount of the worst, the best,
> and the most typical code they can find at it. *The real evaluation only
> comes from actual long-time usage. *The proof of the pudding, as they
> say, is in the eating.


Hi,
I love your quote "As the saying goes: <insert program name here> is
extremely
user-friendly. It's just a little picky who it makes friends with. "
Thanks for making me smile.

I realise like what others say,It depends on the targeted audience and
no single tool can satisfy all.

Regards,
s.subbarayan
 
Reply With Quote
 
Dave Hansen
Guest
Posts: n/a
 
      11-03-2009
On Oct 29, 6:07*am, ssubbarayan <(E-Mail Removed)> wrote:
> Dear all,
> Whats the general norm people use to evaluate static code *analysis
> and dynamic code analysis tools in your experience.I am confused on
> the best tool to choose from,given that so many tools are available
> over the net.I believe
> following are the criteria to test the usefullness of these tools:


First question: do you lint your code?

Some of the new static analysis tools are wondrous things, that detect
subtle bugs in convoluted code. But they tend to be expensive,
require extensive setup and tuning, and can take hours to run.

Lint, on the other hand, is a wondrous thing, detects subtle bugs in
convoluted, code, is cheap, easy to use, and faster than your
compiler. <insert standard praise of Gimpel's PC-lint here>.

The newer tools can do more, certainly, but if you can't be bothered
to lint your code, you're not going to be willing to put in the effort
required by the new tools to get the best results. No tool can give
good results if you don't use it.

On the other hand, if you're already an enthusiastic user of lint, and
you are looking for a tool that digs deeper, carry on with your
search.

Lint early. Lint often. Lint is your friend.

Regards,

-=Dave
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Building Static Analysis Tools for C++ Scott Meyers C++ 13 06-29-2011 09:12 PM
Please recommend some tools for c++ static analysis and memory leak detecting Brian Jiang C++ 11 12-30-2010 11:24 PM
Blog: Tools for C++ Static Analysis Richard C++ 10 09-06-2009 03:15 PM
Any good static analysis tools out there? Derek C++ 2 03-04-2005 01:31 PM
Static Code Analysis Tools Don C Programming 1 07-25-2003 05:28 PM



Advertisments