«

On 01/07/2012 04:20 AM, Ian Kelly wrote:
> On Fri, Jan 6, 2012 at 10:01 AM, Lie Ryan<> wrote:
>> That unittest executes its tests in alphabetical order is implementation
>> detail for a very good reason, and good unittest practice dictates that
>> execution order should never be defined (some even argued that the execution
>> order should be randomized). If the test runner turns out to execute tests
>> concurrently, that should not cause problems for a well-designed test.
>> Displaying the test results in a more convenient order for viewing is what
>> you really wanted in 99.99% of the cases.
>
> Randomizing the order is not a bad idea, but you also need to be able
> to run the tests in a consistent order, from a specific random seed.
> In the real world, test conflicts and dependencies do happen, and if
> we observe a failure, make a change, rerun the tests and observe
> success, we need to be able to be sure that we actually fixed the bug,
> and that it didn't pass only because it was run in a different order.
>
> Concurrent testing is a bad idea for this reason -- it's not
> repeatable (testing concurrency, OTOH, is a perfectly fine thing to be
> thinking about).

Concurrent testing is perfectly fine strategy in the case where you have
thousands of tests and running them synchronously will just take too
long. Certainly it makes it harder to repeat the test if there is any
sort of dependency in the tests, but when you have the large number of
tests, the benefit may exceeds the drawbacks.

On 6 January 2012 11:44, Peter Otten <__peter__@web.de> wrote:
> class Loader(unittest.TestLoader):
> Â* Â*def getTestCaseNames(self, testCaseClass):
> Â* Â* Â* Â*"""Return a sequence of method names found within testCaseClass
> Â* Â* Â* Â*sorted by co_firstlineno.
> Â* Â* Â* Â*"""

That's a clever trick!

--
Arnaud

On 6 January 2012 13:40, Ulrich Eckhardt
<> wrote:
> Am 06.01.2012 12:44, schrieb Peter Otten:
>
>> Alternatively you can write your own test loader:
>
> [...CODE...]
>
> Well, actually you just did that for me and it works!
>
>
> Nonetheless, I'm still wondering if I could somehow replace the dict with an
> OrderedDict.

No, you can't.

--
Arnaud

On Sat, 07 Jan 2012 04:01:44 +1100, Lie Ryan wrote:

> On 01/07/2012 12:36 AM, Ulrich Eckhardt wrote:
>> True, perhaps, but doing it this way would be more fun and easier
>> reusable in other cases where the default order is not desirable. I can
>> also go and name the test functions test_000 to test_009 to get results
>> quickly, if that was the only goal.
>
> Fun and easier, perhaps. Except that it solves the wrong problem.

Fun and easier, and a terrible thing to do. Contrast:

"test_binsearch_tuple is failing. What does that mean?"
"Oh, that tests that binsearch works on a tuple. You need to look at the
BinSearch.search_tuple method and see what it's doing."

with

"test_047 is failing. What does that mean?"
"How the hell should I know?"


Debugging is hard enough without obfuscating the test names. You wouldn't
write a function called "func_009", why write one called "test_009"?



--
Steven

On Fri, 06 Jan 2012 10:20:28 -0700, Ian Kelly wrote:

> On Fri, Jan 6, 2012 at 10:01 AM, Lie Ryan <> wrote:
>> That unittest executes its tests in alphabetical order is
>> implementation detail for a very good reason, and good unittest
>> practice dictates that execution order should never be defined (some
>> even argued that the execution order should be randomized). If the test
>> runner turns out to execute tests concurrently, that should not cause
>> problems for a well-designed test. Displaying the test results in a
>> more convenient order for viewing is what you really wanted in 99.99%
>> of the cases.
>
> Randomizing the order is not a bad idea, but you also need to be able to
> run the tests in a consistent order, from a specific random seed. In the
> real world, test conflicts and dependencies do happen,

In the real world, test conflicts and dependencies are bugs in your test
suite that should be fixed, like any other bug in code. The fact that it
is test code that is failing is irrelevant.

Also in the real world, sometimes it is too hard to fix a bug and you
just live with it.


> and if we observe
> a failure, make a change, rerun the tests and observe success, we need
> to be able to be sure that we actually fixed the bug, and that it didn't
> pass only because it was run in a different order.

Every test should stand alone. You should be able to isolate each and
every test and run it without the others. If you can't, your test suite
is buggy. (Chances are good that most test suites are buggy. After all,
who writes tests for their tests? That's just the start of an infinite
regress with rapidly diminishing benefit.)

You may not be able to run tests *simultaneously*, due to clashes
involving external resources, but you should be able to run them in
random order.



--
Steven

On 01/07/2012 11:49 AM, Steven D'Aprano wrote:
> You may not be able to run tests*simultaneously*, due to clashes
> involving external resources, but you should be able to run them in
> random order.

tests that involves external resources should be mocked, although there
are always a few external resources that cannot be mocked, those are the
exceptions not the rule; a concurrent test runner might have mechanisms
to mark them to be run synchronously.

On Fri, Jan 6, 2012 at 5:49 PM, Steven D'Aprano
<steve+> wrote:
> In the real world, test conflicts and dependencies are bugs in your test
> suite that should be fixed, like any other bug in code. The fact that it
> is test code that is failing is irrelevant.

I agree 100%, but none of that changes my point, which is that when
that this sort of problem arises, you need to be able to test
consistently to know that the bug is actually fixed.

> Every test should stand alone. You should be able to isolate each and
> every test and run it without the others.

Ideally, yes. I'm talking about *unintentional* conflicts and dependencies.

i havnt read every post in great detail, but it doesnt seem like your
actual question has been answered, so ill give it a try.

AFAIK, changing __dict__ to be an ordereddict is fundamentally
impossible in python 2. __dict__ is a builtin language construct
hardcoded into the C API. There is no way to mess with it.

Apparently this is different in python 3, but I dont know much about
that.

On Jan 7, 2:06*am, Ian Kelly <ian.g.ke...@gmail.com> wrote:
> <ulrich.eckha...@dominolaser.com> wrote:
> > Nonetheless, I'm still wondering if I could somehow replace the dict with an
> > OrderedDict.
>
> In Python 3, yes. *This is pretty much the entire use case for the new
> __prepare__ method of metaclasses. *See the "OrderedClass" example[...]

This isn't accurate. The OrderedClass example uses an OrderedDict to
remember the method creation order:

def __new__(cls, name, bases, classdict):
result = type.__new__(cls, name, bases, dict(classdict))
result.members = tuple(classdict)
return result

The instantiated objects __dict__ will still be a regularly
dictionary, while the assignment order is stored in the class
attribute .members.

On 01/09/2012 09:03 AM, Eelco wrote:
> i havnt read every post in great detail, but it doesnt seem like your
> actual question has been answered, so ill give it a try.
>
> AFAIK, changing __dict__ to be an ordereddict is fundamentally
> impossible in python 2. __dict__ is a builtin language construct
> hardcoded into the C API. There is no way to mess with it.
>
> Apparently this is different in python 3, but I dont know much about
> that.

Actually the primary question has been answered by Ian Kelly which
suggested __prepare__ for Python 3, and Peter Otten posted a code for a
custom TestLoader that will essentially do what the OP wanted.

I was just suggesting that what the OP thinks he wants is quite likely
not what he actually wants.