Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Test::Unit -- multiple errors in test method ???

Reply
Thread Tools

Test::Unit -- multiple errors in test method ???

 
 
Johan Holmberg
Guest
Posts: n/a
 
      09-15-2003

Hi !

I have been writing some unit tests with Test::Unit.

I've noted that when an assertion fails in a test method, the
remaining assertions in the same test method aren't even excuted.
Here is an example:

require 'test/unit'

class TC_example < Test::Unit::TestCase

def test_a
assert_equal 2, 1 + 1
assert_equal 5, 2 + 2 # gives error
assert_equal 7, 2 + 5 # never executed !!!
assert_equal 8, 3 + 4 # never executed !!!
end

end

When running this I get:

C:\> ruby TC_example.rb
Loaded suite TC_example
Started
F
Finished in 0.01 seconds.

1) Failure!!!
test_a(TC_example) [TC_example1.rb:8]:
<5> expected but was
<4>

1 tests, 2 assertions, 1 failures, 0 errors


The two last assertions aren't even executed.
Is this a but or an intentional "feature" ?

I've looked at the way JUNIT in Java does the same thing,
and there I got *all* errors reported. That seems much more useful
to mee.

With the current Test::Unit behaviour in Ruby, I can't know if
an error "hides" a number of other errors until I've fixed the first
error (it feels like having a C-compiler that only report the first
compilation error ...).

I have tested this with Ruby 1.8, and also with a more recent
CVS-snapshot.

Am I missing something obvious,
or should this be considered a bug in Test::Unit ?

/Johan Holmberg


 
Reply With Quote
 
 
 
 
David Corbin
Guest
Posts: n/a
 
      09-15-2003
On Monday 15 September 2003 08:08, Johan Holmberg wrote:
> Hi !
>
> I have been writing some unit tests with Test::Unit.
> The two last assertions aren't even executed.
> Is this a but or an intentional "feature" ?


"feature"

>
> I've looked at the way JUNIT in Java does the same thing,
> and there I got *all* errors reported. That seems much more useful
> to mee.


You'd better look more closely. JUnit stops execution of the test at the
first failed assertion. Of course, it doesn't report how many assertions
it's making, so maybe it's less obvious to you.

>
> With the current Test::Unit behaviour in Ruby, I can't know if
> an error "hides" a number of other errors until I've fixed the first
> error (it feels like having a C-compiler that only report the first
> compilation error ...).
>


If you've got a number of assertions in one test case, while not always, often
they are "ascendent". That is, the second asssertion can only really be
checked. if the first one is true. For example, so people might right

assert_not_nil(a)
assert_equals("foo",a.name)


>
> /Johan Holmberg


--
David Corbin <(E-Mail Removed)>


 
Reply With Quote
 
 
 
 
Johan Holmberg
Guest
Posts: n/a
 
      09-15-2003

On Mon, 15 Sep 2003, David Corbin wrote:
> >
> > I've looked at the way JUNIT in Java does the same thing,
> > and there I got *all* errors reported. That seems much more useful
> > to mee.

>
> You'd better look more closely. JUnit stops execution of the test at the
> first failed assertion. Of course, it doesn't report how many assertions
> it's making, so maybe it's less obvious to you.
>


Yes, I missed that.

>
> If you've got a number of assertions in one test case, while not
> always, often they are "ascendent". That is, the second
> asssertion can only really be checked. if the first one is true.
>


Yes, in case this was a feature I suspected the thinking was like
that. But how about these cases ?

- the Rubicon tests for Ruby. The test methods test a number of
things in the same method. As soon as *any* test fails the whole
test suite soon becomes increasingly useless (maybe one could say
that there should not be any errors, but that has always been
the case when I have tried to run Rubicon ...)

- I have tried to write some table driven tests. How should I do
this ? Currently I tried something in the following style:

...
@@data = [
[2, 1, 1],
[5, 2, 2], # should fail
[7, 2, 5],
[8, 3, 4], # should fail
]

def test_b
for facit, aa, bb in @@data
assert_equal facit, aa + bb
end
end
...

Here I have several tests that are not "increasing".
(just before I post this I saw David Alan Blacks answer mentioning
Test::Unit::AssertionFailedError. Should I catch that ? )


Thanks for the replies,
/Johan Holmberg


 
Reply With Quote
 
Simon Strandgaard
Guest
Posts: n/a
 
      09-15-2003
On Mon, 15 Sep 2003 22:08:22 +0900, Johan Holmberg wrote:

> I have been writing some unit tests with Test::Unit.
>
> I've noted that when an assertion fails in a test method, the
> remaining assertions in the same test method aren't even excuted.
> Here is an example:



It is considered good practize only to *one* assertion per testcase, this
way you won't have any assertions which never gets executed.

Even though I doesn't always follow that convention

--
Simon Strandgaard


 
Reply With Quote
 
ts
Guest
Posts: n/a
 
      09-15-2003
>>>>> "J" == Johan Holmberg <(E-Mail Removed)> writes:

J> - I have tried to write some table driven tests. How should I do
J> this ? Currently I tried something in the following style:

Something like this ?

svg% cat b.rb
#!/usr/bin/ruby
require 'test/unit'

class TC_example < Test::Unit::TestCase

@@data = [
[2, 1, 1, true],
[5, 2, 2, false], # should fail
[7, 2, 5, true],
[8, 3, nil, TypeError], # should fail
]

def test_a
for facit, aa, bb, res in @@data
case res
when true
assert_equal(facit, aa + bb)
when false
assert_not_same(facit, aa + bb)
else
assert_raises(res) { aa + bb }
end
end
end
end

svg%

svg% b.rb
Loaded suite ./b
Started
 
Reply With Quote
 
Robert Klemme
Guest
Posts: n/a
 
      09-15-2003

"Simon Strandgaard" <(E-Mail Removed)> schrieb im Newsbeitrag
news(E-Mail Removed) ...
> On Mon, 15 Sep 2003 22:08:22 +0900, Johan Holmberg wrote:
>
> > I have been writing some unit tests with Test::Unit.
> >
> > I've noted that when an assertion fails in a test method, the
> > remaining assertions in the same test method aren't even excuted.
> > Here is an example:

>
>
> It is considered good practize only to *one* assertion per testcase,

this
> way you won't have any assertions which never gets executed.


Did you mean to say "*one* assertion per test" (i.e. test method)? One
assertion per test case (i.e. class) would be a bit lavish...

But even if so: JUnit encourages the convention of one test per method,
which typically leads to multiple assertions per test.

> Even though I doesn't always follow that convention


Same here.

Regards

robert

 
Reply With Quote
 
ts
Guest
Posts: n/a
 
      09-15-2003
>>>>> "J" == Johan Holmberg <(E-Mail Removed)> writes:

J> No, not really.
J> When I wrote "should fail" I just meant that I had written a line in
J> the table that *as it stands* give an error.

Then this ?

svg% cat b.rb
#!/usr/bin/ruby
require 'test/unit'

class TC_example < Test::Unit::TestCase

@@data = [
[2, 1, 1],
[5, 2, 2], # should fail
[7, 2, 5],
[8, 3, 7], # should fail
]

def test_a
error = []
for facit, aa, bb in @@data
begin
assert_equal(facit, aa + bb)
rescue Exception
error << $!
end
end
error.each {|e| add_failure(e.message, e.backtrace) }
end
end

svg%

svg% b.rb
Loaded suite ./b
Started
FF
Finished in 0.002194 seconds.

1) Failure!!!
test_a(TC_example) [./b.rb:17]:
<5> expected but was
<4>

2) Failure!!!
test_a(TC_example) [./b.rb:17]:
<8> expected but was
<10>

1 tests, 4 assertions, 2 failures, 0 errors
svg%


Guy Decoux


 
Reply With Quote
 
Johan Holmberg
Guest
Posts: n/a
 
      09-15-2003

On Mon, 15 Sep 2003, ts wrote:
>
> >>>>> "J" == Johan Holmberg <(E-Mail Removed)> writes:

>
> J> No, not really.
> J> When I wrote "should fail" I just meant that I had written a line in
> J> the table that *as it stands* give an error.
>
> Then this ?
>


Yes !
Thanks, for the example code.

It seems to solve my problems in the case where I want to use
table-driven tests, and where each test (i.e. line in the table) is
independent of the others.

/Johan Holmberg



> svg% cat b.rb
> #!/usr/bin/ruby
> require 'test/unit'
>
> class TC_example < Test::Unit::TestCase
>
> @@data = [
> [2, 1, 1],
> [5, 2, 2], # should fail
> [7, 2, 5],
> [8, 3, 7], # should fail
> ]
>
> def test_a
> error = []
> for facit, aa, bb in @@data
> begin
> assert_equal(facit, aa + bb)
> rescue Exception
> error << $!
> end
> end
> error.each {|e| add_failure(e.message, e.backtrace) }
> end
> end
>
> svg%
>
> svg% b.rb
> Loaded suite ./b
> Started
> FF
> Finished in 0.002194 seconds.
>
> 1) Failure!!!
> test_a(TC_example) [./b.rb:17]:
> <5> expected but was
> <4>
>
> 2) Failure!!!
> test_a(TC_example) [./b.rb:17]:
> <8> expected but was
> <10>
>
> 1 tests, 4 assertions, 2 failures, 0 errors
> svg%
>
>
> Guy Decoux
>
>



 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
TEST TEST Test...Blah Blah Blah generalbatguano@pacbell.net Computer Support 2 09-15-2006 03:47 AM
TEST TEST Test...Blah Blah Blah Generalbatguano@pacbell.net Computer Support 6 09-13-2006 01:53 AM
Errors, errors, errors Mark Goldin ASP .Net 2 01-17-2004 08:05 PM
TEST TEST TEST Gazwad Computer Support 2 09-05-2003 07:32 PM
test test test test test test test Computer Support 2 07-02-2003 06:02 PM



Advertisments