Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: When is unit-testing bad? [was: Re: does lack of type...]

Reply
Thread Tools

Re: When is unit-testing bad? [was: Re: does lack of type...]

 
 
Cameron Laird
Guest
Posts: n/a
 
      07-01-2003
In article <(E-Mail Removed)>,
Edvard Majakari <(E-Mail Removed)> wrote:
>
>I'm rather interested in (and fond of) unit testing as well as TDD, so I
>though to share my opinions. I once discussed TDD with my friend in here,
>and the common agreement seemed to be that TDD is not a silver bullet -
>often there are types of computational modules that are very hard to unit
>test, eg. think of graphical user interfaces or complex systems, where
>forking or threading is concerned.

.
.
.
While we're all being rational enough to look at bundles of
costs and benefits, this observation hints at a different
insight: that it's possible to *design* for testability.

Yes, GUIs and forking and threading are hard to test. SO
DON'T DO THEM. Or, as I prefer, adopt models of GUI and
concurrency development that lend themselves to testing.

At this point, I should explain what I mean by that. I
haven't yet figured out how to say it in less than a book;
therefore, all I can contribute for today are the slogans
above.

If you're in a situation where unit-testing is bad, you're
probably in a bad situation. Change your situation.
--

Cameron Laird <(E-Mail Removed)>
Business: http://www.Phaseit.net
Personal: http://phaseit.net/claird/home.html
 
Reply With Quote
 
 
 
 
Richard Brodie
Guest
Posts: n/a
 
      07-02-2003

"Aurélien Géron" <(E-Mail Removed)> wrote in message
news:bdunoh$1pms$(E-Mail Removed)...

> The same goes for GUIs. I'd be interested to know about projects where
> they automate unit tests for GUIs, but I suspect it would be just as
> cumbersome.


I don't think one would necessary need to automate the GUI itself.
A lot of applications seem to have poor separation of the GUI, something
that the developers of GUI frameworks seem to encourage (not necessarily
as part of their evil plan). If the end result of designing a GUI application for
testing was that it was both easy to script and to port to another toolkit, as a
side effect, than I can see the possible benefit.


 
Reply With Quote
 
 
 
 
David Bolen
Guest
Posts: n/a
 
      07-02-2003
"Aurélien Géron" <(E-Mail Removed)> writes:

> There are things that are hard to unit test (I'm talking about automated
> unit tests here) and which you just can't do without in some projects. For
> instance, the database! I worked in a couple of projects with automatic
> unit tests for the DB related components, and having lived through it, I'm
> not sure that it was worth it at all! Manual unit tests would have made
> much more sense. (...)
> (...)
> I guess my point is that though you may be right in some specific contexts,
> I don't think you can generalize. That's IMHO of course.


I'd agree that there are elements that are much more difficult than
others to test in an automated fashion - sometimes impractically so
given the cost to benefits comparison. Personally, I'd agree that
databases, GUIs and asynchronous activities such as network or
multi-threading are areas that fall into that category. But I also
think that such areas can often be tested far more than some might
think within a given development effort.

The idea is to strip any layer that can't be (or for a given project
can't be justified to be) automatically tested down to its bare
minimum, because an automated test will almost always be of more value
over time (with repeated use, different developers, etc...) than a
manual test.

This is where using the tests during development (as opposed to
afterwards) can help ensure a test friendly design, and one that is
often more modular and structured than a typical developer might have
otherwise arrived at without the tests during development.

For example, when writing a DB interface layer or component (whether
for persistence or direct RDBMS access), if you're trying to construct
tests as you develop, you will more than likely arrive at a structure
where the final database access layer is isolated enough that you can
mock it up for most of your tests of the remainder of your code - if
only because without it you find yourself not able to construct tests

This might take the form of a specific component used to wrap raw
database access. Or, if it's a SQL database, you might find yourself
designing business components such that they serve as producers of the
actual SQL, for which a separate object consumes it and executes it
against a database. That interface (the SQL output) provides a test
point for the business logic without involving the database.

In the former case, tests would simulate the lowest level database
layer to function just as is necessary for the test (accepting and
returning appropriate data), thus being able to rapidly and
efficiently test the remainder of the code without a live database in
an automated and controlled fashion. In the latter case, the tests
would execute the business logic and compare the result to expected
SQL. A much smaller suite of tests to exercise the database layer
would use a test database (or equivalent) but would be focused
strictly on the raw database interface, or execution of known SQL test
statements, and should be much more containable (in time and
development effort), and probably still automatable, although perhaps
on a longer cycle time.

If the system wasn't designed with testing involved, the actual
database I/O might very well be embedded in a higher layer (such as
the persistence mechanism which might be tightly coupled to an object
layer) and hard to isolate from the rest of the logic. For example,
you might find business objects directly calling a persistence or
database I/O layer within their implementation, making it hard to test
the business logic without a live database.

Similar ideas apply to GUIs - keep the actual interface layer as
extremely thin as possible, and containing no business logic or
application decision processing. Whether you use MVC, MVP or an
equivalent approach, ensure that you can test all underlying business
logic without actually needing a GUI. This holds for applications and
web sites (in many respects, a web site can just be modeled as a
different form of view).

For example, in MVC/MVP, your automated test can call
controller/presenter methods to simulate any aspect of user
interaction with the UI, but without the need for a UI, and can mock
up a non-UI view to ensure that the model is notified of appropriate
changes and that queries to it reflect the new information.

At some point you need to verify that performing action X against UI
object Y generates the right requests to the other system components,
but that's far more constrained testing then trying to test the
business logic through that UI. And yes, that could probably involve
external UI testing tools, or some manual interaction with an
automated test mocking the underlying controller/presenter and model.

Of course, this is all easiest if you are designing a system with
testability in mind from the get go. Trying to retrofit unit tests
onto a GUI for example, where much of the business logic is attached
to specific UI elements and not reachable without going through the UI
is just plain old difficult. The idea of making up tests as you go
(whether TDD or equivalent) and using them as part of the development
process helps not only ensure that the final product is as tested, in
as automated a fashion, as possible, but that it ends up being
designed to be tested.

-- David
 
Reply With Quote
 
Peter Hansen
Guest
Posts: n/a
 
      07-02-2003
"Aurélien Géron" wrote:
>
> There are things that are hard to unit test (I'm talking about automated
> unit tests here) and which you just can't do without in some projects. For
> instance, the database! I worked in a couple of projects with automatic
> unit tests for the DB related components, and having lived through it, I'm
> not sure that it was worth it at all! Manual unit tests would have made
> much more sense. The same goes for GUIs. I'd be interested to know about
> projects where they automate unit tests for GUIs, but I suspect it would be
> just as cumbersome. And live without threads? A lot of projects would be
> better off not using them, I agree with that, but some projects just can't
> do without them! Web sites are also horrible to automatically unit test.
> In all those cases, I much prefer having a human being go out and do the
> unit tests manually.


Databases are generally considered much easier to do unit tests for than
GUIs.

GUIs are recognized as difficult, but there are already fairly well-known
approaches for simplifying the process, depending on what you need/expect
out of the testing. For example, one should generally count on the GUI
framework to do what it does correctly and not try to test, for example,
that a call to a drawText() method actually puts pixels on the canvas.
Instead, one should be testing that a call to drawText() is made, and
contains the correct text and positioning.

Web sites are equivalently simple to test, if you aren't trying to test
the presentation itself. For example, unit-testing web forms is pretty
trivial. In fact, in most cases for web stuff, you don't need to actually
run a server, but I believe here I'm getting into XP-specific terminology
where "unit" test refers to something quite different from "acceptance"
or "customer" test, which is where you would be running the actual server.

-Peter
 
Reply With Quote
 
Jeremy Bowers
Guest
Posts: n/a
 
      07-03-2003
On Wed, 02 Jul 2003 15:21:45 +0200, Aurélien Géron wrote:
> I'd be interested to know about
> projects where they automate unit tests for GUIs, but I suspect it would be
> just as cumbersome.


I'm doing one right now. It's designed from the ground up by me, so it's
testable. It almost has to be, since it's fairly complicated and Needs To
Work.

For the record, I've found Tkinter does *great* with this, because you can
drive the GUI without ever actually starting the event loop. It doesn't
seem perfect but largely works, much better then it has any right to.

wxPython doesn't work at all without an event loop, so I never did get it
tested, which has contributed to me dropping it. (I'm gonna guess that
never entered into anybody's head when they were designing toolkits...)

> In all those cases, I much prefer having a human being go out and do the
> unit tests manually.


The thing is, they won't. We've got about 50 years of collective
experience backing that up.
 
Reply With Quote
 
Changjune Kim
Guest
Posts: n/a
 
      07-03-2003

"Jeremy Bowers" <(E-Mail Removed)> wrote in message
news(E-Mail Removed)...
> On Wed, 02 Jul 2003 15:21:45 +0200, Aurélien Géron wrote:
> > I'd be interested to know about
> > projects where they automate unit tests for GUIs, but I suspect it

would be
> > just as cumbersome.

>
> I'm doing one right now. It's designed from the ground up by me, so it's
> testable. It almost has to be, since it's fairly complicated and Needs To
> Work.
>
> For the record, I've found Tkinter does *great* with this, because you

can
> drive the GUI without ever actually starting the event loop. It doesn't
> seem perfect but largely works, much better then it has any right to.
>
> wxPython doesn't work at all without an event loop, so I never did get it
> tested, which has contributed to me dropping it. (I'm gonna guess

that
> never entered into anybody's head when they were designing toolkits...)
>
> > In all those cases, I much prefer having a human being go out and do

the
> > unit tests manually.

>
> The thing is, they won't. We've got about 50 years of collective
> experience backing that up.


UnitTesting GUI is not that difficult. UnitTesting multi-threaded programs
is not that difficult. UnitTesting database programs is not that difficult.
I have experiences in all of them, of which the degree and depth might
vary.

For example, I follow a sequence(as in Christopher Alexander's style) for
TDDable GUI program:


Principles:
1. Test everything that is in P and M. Don't test V.
2. There is no GUI framework API call in P and M.
3. In V, there are stuffs that you can have enough confidence even without
testing.
4. No duplication.
5. Keep the cycle.

Sequence:
1. Start from one chunk. Think about what should be shown from the
abstract level, what concepts(M) should be there. Don't use GUI API calls.
2. You can take M and P apart. How it should be shown goes into P. What
goes into M.
3. P uses the interfaces of V. Firstly, test P with mock-V. Put into V
only those that can be considered as "need not tests, really sure" in GUI
aspects.
4. As mock-V getting its form, implement real-V along with it.
5. If there is anything you feel not confident pass it over to P.

These are what I get after years of experience in TDDing GUI programs.

see also(all highly recommended):
* http://martinfowler.com/articles/separation.pdf
* http://www.objectmentor.com/resource...eDialogBox.pdf
* http://www.sdmagazine.com/documents/sdm0206b/
*
http://www.object-arts.com/Education...wPresenter.htm

Best wishes,

June

 
Reply With Quote
 
Jeremy Bowers
Guest
Posts: n/a
 
      07-03-2003
On Thu, 03 Jul 2003 01:14:44 +0000, Jeremy Bowers wrote:

> On Wed, 02 Jul 2003 15:21:45 +0200, Aurélien Géron wrote:
>> In all those cases, I much prefer having a human being go out and do the
>> unit tests manually.

>
> The thing is, they won't. We've got about 50 years of collective
> experience backing that up.


It occurs I can expand on this. What I'm doing right now at my job (as
opposed to the posting I'm replying to which is my hobby), I'm writing
unittests for an existing mod_perl-based application that is several years
old now. Been an adventure.

Part of the reason I'm doing this is we're safely past the point where we
can just poke and stab at the system and get any sort of coverage. We've
got a role-based system (a user can have several types of roles, and
technically, it's a permission-based system where roles are bundles of
permissions...), and of course any number of things the user can do. Add
up 10-ish roles, any number of actions (lets just call it 50 for
simplicity, though there's no real way good way to put a number on
"distinct actions in this system"), and a couple of other dimensions, and
you've got a lot to cover. Well, actually, it's not addition, it's
multiplication, so even in this rather simple case, you've got 10*50*8 or
something things to cover.

Each of these actions are relatively easy to characterize as unit tests
(well, in proper XP terms these would be acceptance tests, which is still
a step up for the existing system and while I hope to see true unit tests
someday, a whole lotta refactoring would have to happen first), but we've
recently been getting *hammered* with bugs that only occur on one or two
cells in that (easily) 4000-cell matrix. (Of course they're the .5% fringe
cases, important enough to actually occur but not important enough to bang
on in poke-and-grunt testing.) We had a bug that only occurred with Role A
in situation B in environment-type C. Unfortunately, while none of us
ever though to test that exact combination of settings, it was reasonably
common in the real world. Fully automated testing with absolutely no human
interaction needed during the testing is really, really important,
and I can't believe it took me this long to realize that. Not to mention
the rest of the industry.

On that topic, I'm starting to develop a multi-level unit testing
philosophy, since it seems like every program I've ever worked on ends up
with these multi-dimensional tests, and *yes* the equivalent of (1,4,2,4)
will sometimes fail, all by itself. Is there any official XP dogma on
that? Running a complete suite would take too long to be practical, so I'm
starting to get this image of defining a subset of tests for development
purposes (each axis, selected diagonals, smattering of others), and
running the full suite, say, overnight for full coverage... or even
constantly on a spare machine. Would make a great demo for
tourists coming into the lab, with suitably verbose output, probably
impress customers too.
 
Reply With Quote
 
Jim Hefferon
Guest
Posts: n/a
 
      07-04-2003
"Jeremy Bowers" <(E-Mail Removed)> wrote
> We're using mod_perl, and I'm using Test::Unit, which is a testing
> framework for Perl deliberately set up to look like JUnit, as Python's
> unittest is. What I'm doing is implementing an Apache Request faked-up
> object (which I later found existed in the form of the Perl module
> Apache::FakeRequest, but I think my own solution is better suited to our
> needs anyhow), and then going ahead and calling the Apache handlers
> directly with that object. It's tedious to construct the object if you
> can't find one pre-made, but it's easy.
>

Does anyone know of a place where I could see a sample of doing this
under Python?

Thanks in advance
Jim
 
Reply With Quote
 
Edvard Majakari
Guest
Posts: n/a
 
      07-08-2003
On Sat, 05 Jul 2003, http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

> Part I
> http://www-106.ibm.com/developerwork...os-puffin.html
>
> Part II
> http://www-106.ibm.com/developerwork...s-puffin2.html


Having read part I, I really have have to say it looks quite nice tool. I
guess it's no use to starting my own - also, I'm more convinced now that I
should do the testing at lower layers, making presentation layer 'thin'
enough so that there's no need to test it thoroughly. Thanks for the links.

--
#!/usr/bin/perl -w
$h={23,69,28,'6e',2,64,3,76,7,20,13,61,8,'4d',24,7 3,10,'6a',12,'6b',21,68,14,
72,16,'2c',17,20,9,61,11,61,25,74,4,61,1,45,29,20, 5,72,18,61,15,69,20,43,26,
69,19,20,6,64,27,61,22,72};$_=join'',map{chr hex $h->{$_}}sort{$a<=>$b}
keys%$h;m/(\w).*\s(\w+)/x;$_.=uc substr(crypt(join('',60,28,14,49),join'',
map{lc}($1,substr $2,4,1)),2,4)."\n"; print;
 
Reply With Quote
 
Peter Hansen
Guest
Posts: n/a
 
      07-08-2003
Edvard Majakari wrote:
>
> On Sat, 05 Jul 2003, (E-Mail Removed) wrote:
>
> > Part I
> > http://www-106.ibm.com/developerwork...os-puffin.html
> >
> > Part II
> > http://www-106.ibm.com/developerwork...s-puffin2.html

>
> Having read part I, I really have have to say it looks quite nice tool. I
> guess it's no use to starting my own - also, I'm more convinced now that I
> should do the testing at lower layers, making presentation layer 'thin'
> enough so that there's no need to test it thoroughly. Thanks for the links.


A refinement of the above statement: make the presentation layer
"thin" enough that you need to test it thoroughly only *once*. Also
make it thin enough that you very rarely if ever have to make changes
to it, thereby making such even a manual test adequate.

You shouldn't _not_ test something... but it's okay sometimes to avoid
the writing fully automated tests, and this is a good example of when.

-Peter
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: does lack of type declarations make Python unsafe? Anton Vredegoor Python 4 07-10-2003 05:24 PM
Technologic transitions (was: does lack of type declarations make Python unsafe?) Cameron Laird Python 0 07-02-2003 03:05 PM
Re: does lack of type declarations make Python unsafe? Mark 'Kamikaze' Hughes Python 4 07-01-2003 04:38 PM
Re: does lack of type declarations make Python unsafe? David Abrahams Python 0 06-29-2003 11:13 PM
Re: When is unit-testing bad? [was: Re: does lack of type...] Christopher Blunck Python 0 06-24-2003 02:16 AM



Advertisments