Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Javascript (http://www.velocityreviews.com/forums/f68-javascript.html)
-   -   My Library TaskSpeed tests updated (http://www.velocityreviews.com/forums/t940440-my-library-taskspeed-tests-updated.html)

David Mark 02-12-2010 04:22 AM

My Library TaskSpeed tests updated
 
I've updated the TaskSpeed test functions to improve performance. This
necessitated some minor additions (and one change) to the OO interface
as well. I am pretty happy with the interface at this point, so will
set about properly documenting it in the near future.

http://groups.google.com/group/my-li...bcbd95caa03991

I had been thinking about what to do with cloneNode in the OO interface.
All of the talk about PureDom's non-use of it gave me some ideas on how
it could fit in and speed up these tests as well. The change is that
E.prototype.clone no longer returns a wrapped node (not sure why I had
it doing that in the first place as it has been a while since I slapped
these objects together).

Objects design is considerably less slap-dash now too. Almost all
methods are on the prototypes now. There were a dozen or so that were
being added at construction. It wasn't a big issue, but required that
inheriting constructors call the "super" constructor on construction
(which isn't always desirable). For example, F (form) inherits from E,
but implements custom element and load methods. C (control) inherits
from E, but uses the stock methods.

F = function(i, docNode) {
var el;

if (this == global) {
return new F(i, docNode);
}

// Custom element/load methods get and set the element

function element() {
return el;
}

this.load = function(name, docNode) {
el = typeof name == 'object' ? name : getForm(name, docNode);
this.element = (el)?element:null;
return this;
};

this.load(i, docNode);
};

C = function(i, docNode) {

// Called without - new - operator

if (this == global) {
return new C(i, docNode);
}

// Uses stock E element/load methods

E.call(this, i, docNode);
};

I'm no OO guru (or even fan), but I am happy with the way interface has
turned out. One nagging incongruity is that (as noted in the
documentation), you _must_ use the - new - operator when calling from
outside the frame containing the My Library script (sort of like dialing
the area code for a long-distance call).

Functions are a bit more concise now too. The methods names are still
relatively verbose (which is not central to the issue IMO), but there
aren't as many calls to them now (which is what I consider the
determining factor for conciseness). Fewer calls makes for a lower
bill. ;)

Your move, Andrea. Do your worst. :)

David Mark 02-15-2010 04:45 AM

Re: My Library TaskSpeed tests updated
 
David Mark wrote:
> I've updated the TaskSpeed test functions to improve performance. This
> necessitated some minor additions (and one change) to the OO interface
> as well. I am pretty happy with the interface at this point, so will
> set about properly documenting it in the near future.
>
> http://groups.google.com/group/my-li...bcbd95caa03991
>
> I had been thinking about what to do with cloneNode in the OO interface.
> All of the talk about PureDom's non-use of it gave me some ideas on how
> it could fit in and speed up these tests as well. The change is that
> E.prototype.clone no longer returns a wrapped node (not sure why I had
> it doing that in the first place as it has been a while since I slapped
> these objects together).
>
> Objects design is considerably less slap-dash now too. Almost all
> methods are on the prototypes now. There were a dozen or so that were
> being added at construction. It wasn't a big issue, but required that
> inheriting constructors call the "super" constructor on construction
> (which isn't always desirable). For example, F (form) inherits from E,
> but implements custom element and load methods. C (control) inherits
> from E, but uses the stock methods.
>
> F = function(i, docNode) {
> var el;
>
> if (this == global) {
> return new F(i, docNode);
> }
>
> // Custom element/load methods get and set the element
>
> function element() {
> return el;
> }
>
> this.load = function(name, docNode) {
> el = typeof name == 'object' ? name : getForm(name, docNode);
> this.element = (el)?element:null;
> return this;
> };
>
> this.load(i, docNode);
> };
>
> C = function(i, docNode) {
>
> // Called without - new - operator
>
> if (this == global) {
> return new C(i, docNode);
> }
>
> // Uses stock E element/load methods
>
> E.call(this, i, docNode);
> };
>
> I'm no OO guru (or even fan), but I am happy with the way interface has
> turned out. One nagging incongruity is that (as noted in the
> documentation), you _must_ use the - new - operator when calling from
> outside the frame containing the My Library script (sort of like dialing
> the area code for a long-distance call).
>
> Functions are a bit more concise now too. The methods names are still
> relatively verbose (which is not central to the issue IMO), but there
> aren't as many calls to them now (which is what I consider the
> determining factor for conciseness). Fewer calls makes for a lower
> bill. ;)
>
> Your move, Andrea. Do your worst. :)


Opera 10.10, Windows XP on a very busy and older PC:-

2121 18624 9000 5172 22248 4846 4360 1109 1266 1189
6140 1876 843* 798*


I ran it a few times. This is representative. The two versions flip
flop randomly. Usually around a third of the purer tests. :)

The worst is always Prototype. Here roughly 30x slower than My Library.
And it failed to return anything for one of the tests (undefined
result). I remember that disqualification from the previous Operas as
well (among other things). jQuery is at roughly 8X slower.

The other three respectable scores were the two Dojo's and Qooxdoo.
YUI3 came close to respectability.

Of course, if you disallow all that sniff the UA string, the field
shrinks considerably (down to four IIRC), retaining none of the noted
respectable showings. If you do something wrong really fast, what are
you accomplishing? I plan to add a column style to highlight that.
Looking at results for the latest libraries in older (or odd) browsers
shows without a doubt that the sniffing "strategies" failed miserably
(and so can be predicted to fail miserably in the future).

Michael Wojcik 02-15-2010 04:56 PM

Re: My Library TaskSpeed tests updated
 
Andrew Poulos wrote:
>
> I'm not meaning to be offensive, I'm just wondering how one person can
> appear to achieve so much.


Surely if studies of software development have taught us anything,
they've taught us that there is no correlation between the size of a
team and the quality of the code.

I've been a professional developer for nearly a quarter of a century,
and I'm not surprised at all when one person delivers a better
solution than what a large team delivers. Even if the large team's
total effort is much greater - which may or may not be the case here.
Even if the large team has had ample opportunity to solicit feedback
and review and improve their design and implementation.

What matters more, in my opinion, are factors like understanding the
problem space, employing a robust and consistent design, rigorous
attention to quality, and preferring fewer well-implemented features
to many poorly-implemented ones. An individual can often do better in
those areas than a diverse and loosely-organized team can.

That doesn't mean that the open-source "bazaar" development model
necessarily produces bad code - just that it is vulnerable to
sloppiness, uneven quality, and conflicting approaches. Sometimes that
doesn't matter, if the project isn't meant to produce robust code.
(Some projects are prototypes or toys.) Sometimes a strong editorial
hand can enforce discipline on released code. But sometimes too many
cooks simply spoil the soup.

--
Michael Wojcik
Micro Focus
Rhetoric & Writing, Michigan State University

Scott Sauyet 02-15-2010 08:56 PM

Re: My Library TaskSpeed tests updated
 
On Feb 14, 11:45*pm, David Mark <dmark.cins...@gmail.com> wrote:
> David Mark wrote:
>> I've updated the TaskSpeed test functions to improve performance. *This
>> necessitated some minor additions (and one change) to the OO interface
>> as well. *I am pretty happy with the interface at this point, so will
>> set about properly documenting it in the near future.


So you've learned that test-driven development is not an
oxymoron? :-)

I actually am a fan of test-driven design, but I don't do it with
performance tests; that scares me.

>> [http://www.cinsoft.net/taskspeed.html]


> Opera 10.10, Windows XP on a very busy and older PC:-
>
> 2121 * * 18624 * 9000 * *5172 * *22248 * 4846 * *4360* *1109 * *1266 * *1189 *
> 6140 * * 1876 * *843* * *798*
>
> I ran it a few times. *This is representative. *The two versions flip
> flop randomly. *Usually around a third of the purer tests. *:)


I can confirm similar rankings (with generally faster speeds, of
course) on my modern machine in most recent browsers, with two
exceptions: First, in Firefox and IE, PureDOM was faster than My
Library. Second, in IE6, several tests fail in My Library ("setHTML",
"insertBefore", "insertAfter".) Also note that the flip-flopping of
the two versions might have to do with the fact that they are pointing
at the same exact version of My Library (the one with QSA) and the
same test code. You're running the same infrastructure twice! :-)

This is great work, David. I'm very impressed.

But I do have some significant caveats. Some of the tests seem to me
to be cheating, especially when it comes to the loops. For instance,
here is one of the functions specifications:

"append" : function(){
// in a 500 iteration loop:
// create a new <div> with the same critera as 'create'
// - NOTE: rel needs to be == "foo2"
// then append to body element (no caching)
//
// return then length of the matches from the selector
"div[rel^='foo2']"
},

My Library's implementation looks like this:

"append" : function() {
var myEl = E().loadNew('div', { 'rel':'foo2' }), body =
document.body;
for (var i = 500; i--;) {
myEl.loadClone().appendTo(body);
}
return $("div[rel^=foo2]").length;
},

This definitely involves caching some objects outside the loop. There
are a number of such instances of this among the test cases. My
Library is not alone in this, but most of the other libraries mainly
just have variable declarations outside the loop, not initialization.
In part, this is a problem with the Task Speed design and
specification. It would have been much better to have the testing
loop run each library's tests the appropriate number of times rather
than including the loop count in the specification. But that error
should not be an invitation to abuse. I ran a version with such
initializations moved inside the loop, and my tests average about a
15% performance drop for My Library, in all browsers but Opera, where
it made no significant difference.

But that is just one instance of a general problem. My Library is not
alone in coding its tests to the performance metrics. The spec has
this:

"bind" : function(){
// connect onclick to every first child li of ever ul
(suggested: "ul > li")
//
// return the length of the connected nodes
},

but the YUI3 tests perform this with event delegation:

"bind" : function(){
Y.one('body').delegate('click', function() {}, 'ul > li');
return Y.all('ul > li').size();
},

This might well be the suggested way to attach a behavior to a number
of elements in YUI. There's much to be said for doing it in this
manner. And yet it is pretty clearly not what was intended in the
specification; if nothing else, it's an avoidance of what was
presumably intended to be a loop. There's a real question of doing
things the appropriate way.

To test this, I limited myself to 15 minutes of trying to optimize the
jQuery tests in the same manner. I moved initialization outside the
loop and switched to event delegation. After this brief attempt, I
achieved speed gains between 54% and 169% in the various browsers.
And I did this without any changes to the underlying library. I'm
sure I could gain reasonable amounts of speed in some of the other
libraries as well, but this sort of manipulation is wrong-headed.

Perhaps an updated version of TaskSpeed is in order, but it's hard to
design a system that can't be gamed in this manner.

Does your host have PHP? It would suggest it would be better to host
a dynamic version of this, and not rely on static files. It's easy to
set up, and that also makes it almost trivial to add and remove
libraries from your tests.

Finally, the matter of IE6 is disappointing. This is still a widely
used browser; I'm surprised you didn't test there before releasing the
code. You've often pointed out how well My Library performed without
change when IE8 came out. Well the flip side is that it needs to keep
doing well at least in environments that are widely used, even as you
make changes. All the other libraries except qoodoox did fine in IE6,
even if all of them were ungodly slow.

So, overall, I'd give this a B+. It's solid work, but there are some
significant issues that still need to be resolved.

Cheers,

-- Scott


Scott Sauyet 02-16-2010 04:20 AM

Re: My Library TaskSpeed tests updated
 
On Feb 15, 7:21*pm, "Richard Cornford" <Rich...@litotes.demon.co.uk>
wrote:
> Scott Sauyet wrote:
>
> <snip>> * *"bind" : function(){
> > * * * *// *connect onclick to every first child li of ever ul
> >(suggested: "ul > li")

>
> <snip> * * * *^^^^^^^
>
> Surely that is not the CSS selector for "every first child li of ever
> ul". The specs for these tests are really bad, so it is not surprising
> that the implementations of the tests are all over the place.


And of course, if you do what's suggested by the text, you'll look
wrong, because all the other libraries are taking the "ul > li"
literally. I meant to mention this too in my last posting. There is
definitely some room for improvement to the test specs as well as to
the infrastructure.

> > * * * * // *return the length of the connected nodes

>
> You see, what is "the length of the connected nodes"? Is that "length"
> in terms of something like the pixel widths/height of the displayed
> nodes, or is "length" intended to imply the length of some sort of
> 'collected' array of nodes (i.e. some sort of 'query', and if so why are
> we timing that in a context that is not a test of 'selector'
> performance), or does the spec just call for *number* of nodes modified
> by the task to be returned?


This one does not bother me too much. In the context of the
collection of tests [1], it's fairly clear that they want the number
of nodes modified. Presumably they assume that the tools will collect
the nodes in some array-like structure with a "length" property.


-- Scott
___________________
[1] http://dante.dojotoolkit.org/taskspe...ample-tests.js

Scott Sauyet 02-16-2010 04:40 PM

Re: My Library TaskSpeed tests updated
 
On Feb 16, 2:51*am, "Richard Cornford" <Rich...@litotes.demon.co.uk>
wrote:
> Scott Sauyet wrote:
> >On Feb 15, 7:21 pm, Richard Cornford wrote:
> >> Scott Sauyet wrote:
> >> > // return the length of the connected nodes

>
> >> You see, what is "the length of the connected nodes"? Is
> >> that "length" in terms of something like the pixel
> >> widths/height of the displayed nodes, or is "length" intended
> >> to imply the length of some sort of 'collected' array of
> >> nodes (i.e. some sort of 'query', and if so why are we timing
> >> that in a context that is not a test of 'selector' performance),
> >> or does the spec just call for *number* of nodes modified
> >> by the task to be returned?

>
> > This one does not bother me too much. *In the context of the
> > collection of tests [1], it's fairly clear that they want the
> > number of nodes modified.

>
> Is that all they want, rather than, say, in some cases the number
> added/modified nodes as retrieved from the DOM post-modification (as a
> verification that the process has occurred as expected)?


I do think that in all of the tests, the result returned is supposed
to have to do with some count of elements after a certain
manipulation. But in some of them, such as bind, the manipulation
doesn't actually change the number of elements manipulated. As this
was based upon SlickSpeed, it inherits one of the main problems of
that system, namely that it tries to do one thing too many. It tries
to verify accuracy and compare speeds at the same time. This is
problematic, in my opinion.


> The various formulations used include:-
>
> | return the result of the selector ul.fromcode
>
> | return the length of the query "tr td"
>
> | return the lenght of the odd found divs
>
> | return the length of the destroyed nodes
>
> (without any definition of what "the selector" or "the query" mean).
>
> If you are saying that it is just the number of nodes that needs to be
> returned then in, for example, the YUI3 "bind" example you cited:-
>
> | *"bind" : function(){
> | * * * *Y.one('body').delegate('click', function() {}, 'ul > li');
> | * * * *return Y.all('ul > li').size();
> | *},
>
> - the last line could be written:-
>
> return 845; //or whatever number is correct for the document
>
> - and for that function (especially in a non-QSA environment) the bulk
> of the work carried out in that function has been removed.


I've actually thought of doing that, and loudly trumpeting that my
library is unbeatable at TaskSpeed! :-)

> > Presumably they assume that the tools will collect
> > the nodes in some array-like structure with a "length"
> > property.

>
> Maybe they do, but that is not what the spec is actually asking for. And
> if it was what is being asked for, why should that process be included
> in the timing for the tasks, as it is not really part of any realistic
> task.


Yes, if they want verification of counts, perhaps the test harness
itself could provide that.

-- Scott

Richard Cornford 02-16-2010 05:23 PM

Re: My Library TaskSpeed tests updated
 
On Feb 16, 4:40 pm, Scott Sauyet wrote:
> On Feb 16, 2:51 am, Richard Cornford wrote:
>> Scott Sauyet wrote:

<snip>
>>> This one does not bother me too much. In the context of the
>>> collection of tests [1], it's fairly clear that they want the
>>> number of nodes modified.

>
>> Is that all they want, rather than, say, in some cases the number
>> added/modified nodes as retrieved from the DOM post-modification
>> (as a verification that the process has occurred as expected)?

>
> I do think that in all of the tests, the result returned is
> supposed to have to do with some count of elements after a
> certain manipulation.


Which is not a very realistic test 'task' as the number of nodes
modified in operations on real DOMs is seldom of any actual interest.

> But in some of them, such as bind, the manipulation
> doesn't actually change the number of elements manipulated.


And for others, such as the list creation, the number of 'modified'
nodes is pre-determined by the number you have just created.

> As this was based upon SlickSpeed, it inherits one of the main
> problems of that system, namely that it tries to do one thing
> too many.


That is exactly where I attribute the cause of this flaw.

> It tries to verify accuracy and compare speeds at the same
> time. This is problematic, in my opinion.


Especially when it is timing the verification process with the task,
and applying different verification code to each 'library'. There you
have the potential for miscounting library code to combine with
misbehaving DOM modification code to give the impression that the
whole thing is working correctly, or for a correctly carried out task
to be labelled as failing because the counting process is off form
some reason.

<snip>
>> return 845; //or whatever number is correct for the document

>
>> - and for that function (especially in a non-QSA environment)
>> the bulk of the work carried out in that function has been
>> removed.

>
> I've actually thought of doing that, and loudly trumpeting
> that my library is unbeatable at TaskSpeed! :-)


It wouldn't do the pure DOM code any harm either.

>>> Presumably they assume that the tools will collect
>>> the nodes in some array-like structure with a "length"
>>> property.

>
>> Maybe they do, but that is not what the spec is actually asking
>> for. And if it was what is being asked for, why should that
>> process be included in the timing for the tasks, as it is not
>> really part of any realistic task.

>
> Yes, if they want verification of counts, perhaps the test
> harness itself could provide that.


Not just "perhaps". It should, and it should use the same verification
code for each test, and outside of any timing recording.

Richard.

Scott Sauyet 02-16-2010 08:57 PM

Re: My Library TaskSpeed tests updated
 
On Feb 16, 12:23*pm, Richard Cornford <Rich...@litotes.demon.co.uk>
wrote:
> On Feb 16, 4:40 pm, Scott Sauyet *wrote:
>
>> On Feb 16, 2:51 am, Richard Cornford wrote:
>>> Scott Sauyet wrote:
>>>> Presumably they assume that the tools will collect
>>>> the nodes in some array-like structure with a "length"
>>>> property.

>
>>> Maybe they do, but that is not what the spec is actually asking
>>> for. And if it was what is being asked for, why should that
> >> process be included in the timing for the tasks, as it is not
>>> really part of any realistic task.

>
>> Yes, if they want verification of counts, perhaps the test
>> harness itself could provide that.

>
> Not just "perhaps". It should, and it should use the same verification
> code for each test, and outside of any timing recording.


I think that testing the selector engine is part of testing the
library. Although this is not the same as the SlickSpeed selectors
test, it should subsume that one. So I don't object to testing
selector speed. The verification, though, is a different story. It's
quite easy to switch testing documents, but it is presumably not so
easy to verify all the results of all the manipulations. The
compromise that TaskSpeed inherits from SlickSpeed is, I think, fairly
reasonable. Make all the libraries report their results, and note if
there is any disagreement. They could, of course, all be wrong and
yet all have the same values, but that seems relatively unlikely.

There is an approach that I doubt I'd bother trying, but which is
quite interesting: Add a url query parameter, which would serve as a
seed for a randomizing function. If the server does not get one, it
chooses a random value and redirects to a page with that random seed.
Then, based upon random numbers derived from that seed, a document is
generated with some flexible structure, and a test script is generated
that runs a random some sequence of the predefined test cases against
each library. Verification might be tricky, but should be doable.
This might make it more difficult for libraries to design their tests
around the particulars of the document and/or the ordering of the
tests. While I think this would work, it sounds like more effort than
I'm willing to put in right now.

-- Scott

Richard Cornford 02-17-2010 04:04 PM

Re: My Library TaskSpeed tests updated
 
On Feb 16, 8:57 pm, Scott Sauyet wrote:
> On Feb 16, 12:23 pm, Richard Cornford wrote:
>> On Feb 16, 4:40 pm, Scott Sauyet wrote:
>>> On Feb 16, 2:51 am, Richard Cornford wrote:
>>>> Scott Sauyet wrote:
>>>>> Presumably they assume that the tools will collect
>>>>> the nodes in some array-like structure with a "length"
>>>>> property.

>
>>>> Maybe they do, but that is not what the spec is actually
>>>> asking for. And if it was what is being asked for, why
>>>> should that process be included in the timing for the
>>>> tasks, as it is not really part of any realistic task.

>
>>> Yes, if they want verification of counts, perhaps the test
>>> harness itself could provide that.

>
>> Not just "perhaps". It should, and it should use the same
>> verification code for each test, and outside of any
>> timing recording.

>
> I think that testing the selector engine is part of testing
> the library.


Obviously it is, if the 'library' has a selector engine, but that is a
separate activity from testing the library's ability to carry out
tasks as real world tasks don't necessitate any selector engine.
(Remember that common hardware and browser performance was not
sufficient for any sort of selector engine even to look like a viable
idea before about the middle of 2005, but (even quite extreme) DOM
manipulation was long established by that time.)

The 'pure DOM' tests, as a baseline for comparison, don't necessarily
need a selector engine to perform any given task (beyond the fact that
the tasks themselves have been designed around a notion of
'selectors'). So making selector engine testing part of the 'task'
tests acts to impose arbitrary restrictions on the possible code used,
biases the results, and ultimately negates the significance of the
entire exercise.

> Although this is not the same as the SlickSpeed
> selectors test,


Comparing the selector engines in libraries that have selector engines
seems like a fairly reasonable thing to do. Suggesting that a selector
engine is an inevitable prerequisite for carrying out DOM manipulation
tasks is self evident BS.

> it should subsume that one. So I don't object
> to testing selector speed. The verification, though, is a
> different story. It's quite easy to switch testing documents,
> but it is presumably not so easy to verify all the results of
> all the manipulations.


Why not (at least in most cases)? code could be written to record the
changes to a DOM that resulted from running a test function. You know
what you expect the test function to do so verifying that it did do it
shouldn't be too hard.

Granted there are cases like the use of - addEventListener - where
positive verification becomes a lot more difficult, but as it is the
existing tests aren't actually verifying that listeners were added.

> The compromise that TaskSpeed inherits
> from SlickSpeed is, I think, fairly reasonable.


I don't. TaskSpeed's validity is compromised in the process.

> Make all the libraries report their results, and note
> if there is any disagreement.


But reporting result is not part of any genuinely representative task,
and so it should not be timed along with any given task. The task
itself should be timed in isolation, and any verification employed
separately.

Whether some 'library' should be allowed to do its own verification is
another matter, but the verification definitely should not be timed
along with the task that it is attempting to verify.

> They could, of course, all be wrong and
> yet all have the same values, but that seems
> relatively unlikely.


Unlikely, but not impossible, and an issue that can easily be entirely
avoided.

> There is an approach that I doubt I'd bother trying, but
> which is quite interesting: Add a url query parameter,
> which would serve as a seed for a randomizing function.
> If the server does not get one, it chooses a random value
> and redirects to a page with that random seed. Then, based
> upon random numbers derived from that seed, a document is
> generated with some flexible structure, and a test script
> is generated that runs a random some sequence of the
> predefined test cases against each library.


I can see how this might make sense in selector speed testing (though
presumably you would run up against many cases where the reported
duration of the test would be zero millisecond, despite our knowing
that nothing happens in zero time) but for task testing randomly
generating the document acted upon would be totally the wrong
approach. If you did that you would bias against the baseline pure DOM
tests as then they would have to handle issues arising from the
general case, which are not issues inherent in DOM scripting because
websites are not randomly generated.

In any real web site/web application employment of scripting,
somewhere between something and everything is known about the
documents that are being scripted. Thus DOM scripts do not need to
deal with general issues in browser scripting, but rather only need to
deal with the issues that are known to exist in their specific
context.

In contrast, it is an inherent problem in general purpose library code
that they must address (or attempt to address) all the issues that
occur in a wide range of context (at minimum, all the common
contexts). There are inevitably overheads in doing this, with those
overheads increasing as the number of contexts accommodated increases.

With random documents and comparing libraries against some supposed
'pure DOM' baseline, you will be burdening the baseline with the
overheads that are only inherent in general purpose code. The result
would not be a representative comparison.

> Verification might be tricky, but should be doable.
> This might make it more difficult for libraries to
> design their tests around the particulars of the
> document and/or the ordering of the tests. While
> I think this would work, it sounds like more
> effort than I'm willing to put in right now.


Given that javascript source is available if anyone want to look at
it, any library author attempting to optimise for a specific test
(rather than, say, optimising for a common case) is likely to be
spotted doing so, and see their reputation suffer as a result.

Richard.

Scott Sauyet 02-17-2010 05:54 PM

Re: My Library TaskSpeed tests updated
 
On Feb 17, 11:04*am, Richard Cornford wrote:
> On Feb 16, 8:57 pm, Scott Sauyet wrote:


>> I think that testing the selector engine is part of testing
>> the library.

>
> Obviously it is, if the 'library' has a selector engine, but that is a
> separate activity from testing the library's ability to carry out
> tasks as real world tasks don't necessitate any selector engine.


Perhaps it's only because the test framework was built testing against
libraries that had both DOM manipulation and selector engines, but
these seem a natural fit. I don't believe this was meant to be a DOM
manipulation test in particular. My understanding (and I was not
involved in any of the original design, so take this with a grain of
salt) is that this was meant to be a more general test of how the
libraries were used, which involved DOM manipulation and selector-
based querying. If it seemed at all feasible, the framework would
probably have included event handler manipulation tests, as well. If
the libraries had all offered classical OO infrastructures the way
MooTools and Prototype do, that would probably also be tested.

Why the scare quotes around "library"? Is there a better term --
"toolkit"? -- that describes the systems being tested?

> (Remember that common hardware and browser performance was not
> sufficient for any sort of selector engine even to look like a viable
> idea before about the middle of 2005, but (even quite extreme) DOM
> manipulation was long established by that time.)


Really? Very interesting. I didn't realize that it was a system
performance issue. I just thought it was a new way of doing things
that people started trying around then.


> The 'pure DOM' tests, as a baseline for comparison, don't necessarily
> need a selector engine to perform any given task (beyond the fact that
> the tasks themselves have been designed around a notion of
> 'selectors'). So making selector engine testing part of the 'task'
> tests acts to impose arbitrary restrictions on the possible code used,


Absolutely. A pure selector engine would also not be testable, nor
would a drag-and-drop toolkit. We are restricted to systems that can
manipulate the DOM and find the size of certain collections of
elements.


> biases the results,


In what way?


> and ultimately negates the significance of the entire exercise.


I just don't see it. There is clearly much room for improvement, but
I think the tests as they stand have significant value.



>> Although this is not the same as the SlickSpeed
>> selectors test,

>
> Comparing the selector engines in libraries that have selector engines
> seems like a fairly reasonable thing to do. Suggesting that a selector
> engine is an inevitable prerequisite for carrying out DOM manipulation
> tasks is self evident BS.


Note that these results don't require that the library actually use a
CSS-style selector engine, only that it can for instance find the
number of elements of a certain type, the set of which if often most
easily described via a CSS selector. When the "table" function is
defined to return "the length of the query 'tr td'," we can interpret
that as counting the results of running the selector "tr td" in the
context of the document if we have a selector engine, but as "the
number of distinct TD elements in the document which descend from TR
elements" if not. Being able to find such elements has been an
important part of most of the DOM manipulation I've done.

PureDOM does all this without any particular CSS selector engine, so
it's clear that one is not required to pass the tests.


>> it should subsume that one. *So I don't object
>> to testing selector speed. *The verification, though, is a
>> different story. *It's quite easy to switch testing documents,
>> but it is presumably not so easy to verify all the results of
>> all the manipulations.

>
> Why not (at least in most cases)? code could be written to record the
> changes to a DOM that resulted from running a test function. You know
> what you expect the test function to do so verifying that it did do it
> shouldn't be too hard.


The document to test has been fairly static, and I suppose one could
go through it, analyzing its structure, and calculating the expected
results. But the document is included as a stand-alone file, used
with this PHP:

<?php include('../template.html');?>

Another file could easily be substituted, and it might well be
worthwhile doing. Adding this sort of analysis would make it much
more time-consuming to test against a different document.


> Granted there are cases like the use of - addEventListener - where
> positive verification becomes a lot more difficult, but as it is the
> existing tests aren't actually verifying that listeners were added.


Are there any good techniques you know of that would make it
straightforward to actually test this from within the browser's script
engine? It would be great to be able to test this.

>> The compromise that TaskSpeed inherits
>> from SlickSpeed is, I think, fairly reasonable.

>
> I don't. TaskSpeed's validity is compromised in the process.
>
>> Make all the libraries report their results, and note
>> if there is any disagreement.

>
> But reporting result is not part of any genuinely representative task,
> and so it should not be timed along with any given task. The task
> itself should be timed in isolation, and any verification employed
> separately. [ ... ]


I think this critique is valid only if you assume that the
infrastructure is designed only to test DOM Manipulation. I don't buy
that assumption.


>> They could, of course, all be wrong and
>> yet all have the same values, but that seems
>> relatively unlikely.

>
> Unlikely, but not impossible, and an issue that can easily be entirely
> avoided.


Easily for a single document, and even then only with some real work
in finding the expected results and devising a way to test them.


>> There is an approach that I doubt I'd bother trying, but
>> which is quite interesting: *Add a url query parameter,
>> which would serve as a seed for a randomizing function.
>> If the server does not get one, it chooses a random value
>> and redirects to a page with that random seed. Then, based
>> upon random numbers derived from that seed, a document is
>> generated with some flexible structure, and a test script
>> is generated that runs a random some sequence of the
>> predefined test cases against each library.

>
> I can see how this might make sense in selector speed testing (though
> presumably you would run up against many cases where the reported
> duration of the test would be zero millisecond, despite our knowing
> that nothing happens in zero time)


In another thread [1], I discuss an updated version of slickspeed,
which counts repeated tests over a 250ms span to more accurately time
the selectors.

> but for task testing randomly
> generating the document acted upon would be totally the wrong
> approach. If you did that you would bias against the baseline pure DOM
> tests as then they would have to handle issues arising from the
> general case, which are not issues inherent in DOM scripting because
> websites are not randomly generated.


I was not expecting entirely random documents. Instead, I would
expect to generate one in which the supplied tests generally have
meaningful results. So for this test

"attr" : function(){
// find all ul elements in the page.
// generate an array of their id's
// return the length of that array
},

I might want to randomly determine the level of nesting at which ULs
appear, randomly determine how many are included in the document, and
perhaps randomly choose whether some of them do not actually have
ids. There would probably be some small chance that there were no ULs
at all.

> In any real web site/web application employment of scripting,
> somewhere between something and everything is known about the
> documents that are being scripted. Thus DOM scripts do not need to
> deal with general issues in browser scripting, but rather only need to
> deal with the issues that are known to exist in their specific
> context.


Absolutely. I definitely wouldn't try to build entirely random
documents, only documents for which the results of the tests should be
meaningful. The reason I said I probably wouldn't do this is that,
while it is by no means impossible, it is also a far from trivial
exercise.


> In contrast, it is an inherent problem in general purpose library code
> that they must address (or attempt to address) all the issues that
> occur in a wide range of context (at minimum, all the common
> contexts). There are inevitably overheads in doing this, with those
> overheads increasing as the number of contexts accommodated increases.


Yes, this is true. But it is precisely these general purpose
libraries that are under comparison in these tests. Being able to
compare their performance and the code each one uses are the only
reason these tests exist.

> [ ... ]
>> Verification might be tricky, but should be doable.
>> This might make it more difficult for libraries to
>> design their tests around the particulars of the
>> document and/or the ordering of the tests. *While
>> I think this would work, it sounds like more
>> effort than I'm willing to put in right now.

>
> Given that javascript source is available if anyone want to look at
> it, any library author attempting to optimise for a specific test
> (rather than, say, optimising for a common case) *is likely to be
> spotted doing so, and see their reputation suffer as a result.


I would hope so, but as I said in the post to which you initially
responded, I see a fair bit of what could reasonably be considered
optomising for the test, and I only really looked at jQuery's, YUI's,
and My Library's test code. I wouldn't be surprised to find more in
the others.

-- Scott
____________________
[1] http://groups.google.com/group/comp....33d40588ae2ff0


All times are GMT. The time now is 05:13 AM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.