Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   C++ (http://www.velocityreviews.com/forums/f39-c.html)
-   -   Unit testing of expected failures -- what do you use? (http://www.velocityreviews.com/forums/t716251-unit-testing-of-expected-failures-what-do-you-use.html)

Alf P. Steinbach 02-27-2010 06:56 PM

Unit testing of expected failures -- what do you use?
 
OK, this displays my ignorance of what's out there (it's been a long time since
I developed for a living), and also my laziness not googling. :-)

However.

I want to unit-test some library code I'm sort of extracting from some old code
I have.

The things that should work without error are easy to test, and it's currently
not so much code that I've considered a testing framework, although the code
size increases. I'm thinking that perhaps the popular frameworks don't support
my needs: there are cases where the code /should/ assert at run time. And worse,
there are cases where the could should assert at compile time...

How do you deal with this kind of testing, testing that things fail as they
should (at compile time and at run time)?

I'm particularly interested in anything that works well with Visual Studio 7.x
or thereabouts, but I'm thinking that I most likely will have to create makefiles?


Cheers,

- Alf

Ian Collins 02-27-2010 10:22 PM

Re: Unit testing of expected failures -- what do you use?
 
Alf P. Steinbach wrote:
> OK, this displays my ignorance of what's out there (it's been a long
> time since I developed for a living), and also my laziness not googling.
> :-)
>
> However.
>
> I want to unit-test some library code I'm sort of extracting from some
> old code I have.
>
> The things that should work without error are easy to test, and it's
> currently not so much code that I've considered a testing framework,
> although the code size increases. I'm thinking that perhaps the popular
> frameworks don't support my needs: there are cases where the code
> /should/ assert at run time. And worse, there are cases where the could
> should assert at compile time...


I used to use cppUnit (and still do for older projects) but now I use
gtest (http://code.google.com/p/googletest/). Compile time asserts are
beyond the scope of a unit testing framework; if it won't compile, there
isn't a unit to test!

> How do you deal with this kind of testing, testing that things fail as
> they should (at compile time and at run time)?


For run time asserts, I interpose whatever function the system's assert
macro calls (__assert on Solaris) and have __assert throw an exception
with the file, line and expression passed by assert.

A boiled down (no framework) example:

#include <iostream>
#include <exception>
#include <assert.h>

struct AssertionException : std::runtime_error
{
const char* expression;
const char* file;
int line;

AssertionException(const char* expression, const char* file, int line)
: std::runtime_error( expression ),
expression(expression), file(file), line(line) {}
};

void __assert(const char* expression, const char* file, int line)
{
throw AssertionException( expression, file, line );
}

void fut( const char* p )
{
assert( NULL != p );
}

int main()
{
try
{
fut( NULL );
std::cerr << "Oops" << std::endl;
}
catch( const AssertionException& e )
{
std::cerr << "Caught" << ": " << e.what() << std::endl;
}
}

--
Ian Collins

Jorgen Grahn 02-28-2010 11:49 AM

Re: Unit testing of expected failures -- what do you use?
 
On Sat, 2010-02-27, Ian Collins wrote:
> Alf P. Steinbach wrote:
>> OK, this displays my ignorance of what's out there (it's been a long
>> time since I developed for a living), and also my laziness not googling.
>> :-)
>>
>> However.
>>
>> I want to unit-test some library code I'm sort of extracting from some
>> old code I have.
>>
>> The things that should work without error are easy to test, and it's
>> currently not so much code that I've considered a testing framework,
>> although the code size increases. I'm thinking that perhaps the popular
>> frameworks don't support my needs: there are cases where the code
>> /should/ assert at run time. And worse, there are cases where the could
>> should assert at compile time...

>
> I used to use cppUnit (and still do for older projects) but now I use
> gtest (http://code.google.com/p/googletest/). Compile time asserts are
> beyond the scope of a unit testing framework; if it won't compile, there
> isn't a unit to test!


I don't even know what a "unit" is in this context. But I disagree --
with C++ it's sometimes as important to assure that some things don't
compile or fail noisily at runtime, as it is that the "normal tests"
pass.

I seem to recall that *some* of them support static checks. But I
haven't used any, and it wouldn't surprise me if they were clumpsy.
It *does* mean stepping into the build system's territory after all.

I've never done it, but compile-time checks I think I would implement
myself, in my Makefile. The "make check" target would of course drive
both the normal unit tests and the compile-time things.

But I'm not on Windows, and I wouldn't hesitate to use Gnu
Make-specific features.

>> How do you deal with this kind of testing, testing that things fail as
>> they should (at compile time and at run time)?

>
> For run time asserts, I interpose whatever function the system's assert
> macro calls (__assert on Solaris) and have __assert throw an exception
> with the file, line and expression passed by assert.


/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

Robert Fendt 02-28-2010 01:28 PM

Re: Unit testing of expected failures -- what do you use?
 
And thus spake Jorgen Grahn <grahn+nntp@snipabacken.se>
28 Feb 2010 11:49:48 GMT:

> > I used to use cppUnit (and still do for older projects) but now I use
> > gtest (http://code.google.com/p/googletest/). Compile time asserts are
> > beyond the scope of a unit testing framework; if it won't compile, there
> > isn't a unit to test!

>
> I don't even know what a "unit" is in this context. But I disagree --
> with C++ it's sometimes as important to assure that some things don't
> compile or fail noisily at runtime, as it is that the "normal tests"
> pass.
>
> I seem to recall that *some* of them support static checks. But I
> haven't used any, and it wouldn't surprise me if they were clumpsy.
> It *does* mean stepping into the build system's territory after all.


There's some stuff on compile-time assertions in the
Alexandrescu book ('Modern C++ Design'). Have a look at chapter
2, 'techniques', right at the beginning.

Regards,
Robert


Vladimir Jovic 03-01-2010 06:43 AM

Re: Unit testing of expected failures -- what do you use?
 
Alf P. Steinbach wrote:
> OK, this displays my ignorance of what's out there (it's been a long
> time since I developed for a living), and also my laziness not googling.
> :-)
>
> However.
>
> I want to unit-test some library code I'm sort of extracting from some
> old code I have.
>


For unit testing, see this:
http://cxxtest.sourceforge.net/guide.html

> The things that should work without error are easy to test, and it's
> currently not so much code that I've considered a testing framework,
> although the code size increases. I'm thinking that perhaps the popular
> frameworks don't support my needs: there are cases where the code
> /should/ assert at run time. And worse, there are cases where the could
> should assert at compile time...
>
> How do you deal with this kind of testing, testing that things fail as
> they should (at compile time and at run time)?


For compile time testing, see this:
http://www.boost.org/doc/libs/1_42_0...ticassert.html

For run time testing, I am using a macro, which throws an exception if
the condition fails. The exception class prints backtrace and the failed
condition.

Alf P. Steinbach 03-01-2010 11:15 AM

Re: Unit testing of expected failures -- what do you use?
 
* Vladimir Jovic:
> Alf P. Steinbach wrote:
>> OK, this displays my ignorance of what's out there (it's been a long
>> time since I developed for a living), and also my laziness not
>> googling. :-)
>>
>> However.
>>
>> I want to unit-test some library code I'm sort of extracting from some
>> old code I have.
>>

>
> For unit testing, see this:
> http://cxxtest.sourceforge.net/guide.html
>
>> The things that should work without error are easy to test, and it's
>> currently not so much code that I've considered a testing framework,
>> although the code size increases. I'm thinking that perhaps the
>> popular frameworks don't support my needs: there are cases where the
>> code /should/ assert at run time. And worse, there are cases where the
>> could should assert at compile time...
>>
>> How do you deal with this kind of testing, testing that things fail as
>> they should (at compile time and at run time)?

>
> For compile time testing, see this:
> http://www.boost.org/doc/libs/1_42_0...ticassert.html
>
> For run time testing, I am using a macro, which throws an exception if
> the condition fails. The exception class prints backtrace and the failed
> condition.


Thanks, but I think you misunderstood the question.

E.g., the problem isn't to produce compile time asserts. The problem is testing
them, systematically. Preferably in an automated way.


Cheers, & thanks,

- Alf

Alf P. Steinbach 03-01-2010 08:37 PM

Re: Unit testing of expected failures -- what do you use?
 
* Alf P. Steinbach:
> OK, this displays my ignorance of what's out there (it's been a long
> time since I developed for a living), and also my laziness not googling.
> :-)
>
> However.
>
> I want to unit-test some library code I'm sort of extracting from some
> old code I have.
>
> The things that should work without error are easy to test, and it's
> currently not so much code that I've considered a testing framework,
> although the code size increases. I'm thinking that perhaps the popular
> frameworks don't support my needs: there are cases where the code
> /should/ assert at run time. And worse, there are cases where the could
> should assert at compile time...


OK, I made a unit test driver GUI in Python 3.x, <url:
http://pastebin.com/bB1jeP5Z>.

It reports success or failure depending on whether build is expected to succeed
or fail, and depending on whether running the executable is expected to succeed
or fail.

It's very unfinished but this is a first usable version.

Tests and subtests are identified by C++ macro symbols defined in an [.ini]
file. Running a test the GUI (1) places #define's of the selected symbols in
file [_config.h], (2) invokes a 'build' script which in Windows can simply be a
batch file (building a /test/ is ordinarily very fast), and (3) if the build
succeeds and is expected to succeed, it invokes a 'run' script.

My 'build' script looks like this:

<code file="build.bat">
@echo off
devenv /nologo my_msvc_solution.sln /project my_lib_project.vcproj /build Debug
</code>

And the 'run' script is simply:

<code file="run.bat">
@Debug\the_name_of_the_executable
</code>

In the Python code the script (batch file) directory is hardcoded as
build_dir = os.path.normpath( "../build/msvc_7_1" )
but it should be easy to change.

For a main test the build is always expected to succeed.

For a subtest the build is expected to fail if the subtest id starts with "CERR_".


Cheers,

- Alf

tonydee 03-02-2010 05:21 AM

Re: Unit testing of expected failures -- what do you use?
 
On Mar 2, 5:37*am, "Alf P. Steinbach" <al...@start.no> wrote:
> > The things that should work without error are easy to test, and it's
> > currently not so much code that I've considered a testing framework,
> > although the code size increases. I'm thinking that perhaps the popular
> > frameworks don't support my needs: there are cases where the code
> > /should/ assert at run time. And worse, there are cases where the could
> > should assert at compile time...

>
> OK, I made a unit test driver GUI in Python 3.x, <url:http://pastebin.com/bB1jeP5Z>.


I've done something similar (conceptually, didn't check out your
Python code). Just thought I'd mention another alternative (at least
on UNIX) ala "configure" scripts - they often check for success or
failure in compiling little test programs.

Cheers,
Tony

Vladimir Jovic 03-02-2010 01:28 PM

Re: Unit testing of expected failures -- what do you use?
 
Alf P. Steinbach wrote:
> * Alf P. Steinbach:
>> OK, this displays my ignorance of what's out there (it's been a long
>> time since I developed for a living), and also my laziness not
>> googling. :-)
>>
>> However.
>>
>> I want to unit-test some library code I'm sort of extracting from some
>> old code I have.
>>
>> The things that should work without error are easy to test, and it's
>> currently not so much code that I've considered a testing framework,
>> although the code size increases. I'm thinking that perhaps the
>> popular frameworks don't support my needs: there are cases where the
>> code /should/ assert at run time. And worse, there are cases where the
>> could should assert at compile time...

>
> OK, I made a unit test driver GUI in Python 3.x, <url:
> http://pastebin.com/bB1jeP5Z>.
>
> It reports success or failure depending on whether build is expected to
> succeed or fail, and depending on whether running the executable is
> expected to succeed or fail.
>
> It's very unfinished but this is a first usable version.
>
> Tests and subtests are identified by C++ macro symbols defined in an
> [.ini] file. Running a test the GUI (1) places #define's of the selected
> symbols in file [_config.h], (2) invokes a 'build' script which in
> Windows can simply be a batch file (building a /test/ is ordinarily very
> fast), and (3) if the build succeeds and is expected to succeed, it
> invokes a 'run' script.
>
> My 'build' script looks like this:
>
> <code file="build.bat">
> @echo off
> devenv /nologo my_msvc_solution.sln /project my_lib_project.vcproj
> /build Debug
> </code>
>
> And the 'run' script is simply:
>
> <code file="run.bat">
> @Debug\the_name_of_the_executable
> </code>
>
> In the Python code the script (batch file) directory is hardcoded as
> build_dir = os.path.normpath( "../build/msvc_7_1" )
> but it should be easy to change.
>
> For a main test the build is always expected to succeed.
>
> For a subtest the build is expected to fail if the subtest id starts
> with "CERR_".
>


Ok, now I get it. For this kind of testing, you can use this:
http://expect.nist.gov/

DeMarcus 03-03-2010 06:02 PM

Re: Unit testing of expected failures -- what do you use?
 
Ian Collins wrote:
> Alf P. Steinbach wrote:
>> OK, this displays my ignorance of what's out there (it's been a long
>> time since I developed for a living), and also my laziness not
>> googling. :-)
>>
>> However.
>>
>> I want to unit-test some library code I'm sort of extracting from some
>> old code I have.
>>
>> The things that should work without error are easy to test, and it's
>> currently not so much code that I've considered a testing framework,
>> although the code size increases. I'm thinking that perhaps the
>> popular frameworks don't support my needs: there are cases where the
>> code /should/ assert at run time. And worse, there are cases where the
>> could should assert at compile time...

>
> I used to use cppUnit (and still do for older projects) but now I use
> gtest (http://code.google.com/p/googletest/). Compile time asserts are
> beyond the scope of a unit testing framework; if it won't compile, there
> isn't a unit to test!
>
>> How do you deal with this kind of testing, testing that things fail as
>> they should (at compile time and at run time)?

>
> For run time asserts, I interpose whatever function the system's assert
> macro calls (__assert on Solaris) and have __assert throw an exception
> with the file, line and expression passed by assert.
>
> A boiled down (no framework) example:
>
> #include <iostream>
> #include <exception>
> #include <assert.h>
>
> struct AssertionException : std::runtime_error
> {
> const char* expression;
> const char* file;
> int line;
>
> AssertionException(const char* expression, const char* file, int line)
> : std::runtime_error( expression ),
> expression(expression), file(file), line(line) {}
> };
>
> void __assert(const char* expression, const char* file, int line)
> {
> throw AssertionException( expression, file, line );
> }
>
> void fut( const char* p )
> {
> assert( NULL != p );
> }
>
> int main()
> {
> try
> {
> fut( NULL );
> std::cerr << "Oops" << std::endl;
> }
> catch( const AssertionException& e )
> {
> std::cerr << "Caught" << ": " << e.what() << std::endl;
> }
> }
>


I'm not a complete expert, nor am I a sulky person, but before writing
assert exceptions one might consider following that I found in the book
C++ Coding Standards by Sutter & Alexandrescu, Item 68 - Assert
liberally to document internal assumptions and invariants.

Quote:
"It is not recommended to throw an exception instead of asserting, even
though the standard std::logic_error exception class was originally
designed for this purpose. The primary disadvantage of using an
exception to report a programming error is that you don't really want
stack unwinding to occur - you want the debugger to launch on the exact
line where the violation was detected, with the line's state intact."


/Daniel


All times are GMT. The time now is 08:18 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.