Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > looping in batches

Reply
Thread Tools

looping in batches

 
 
graham
Guest
Posts: n/a
 
      03-24-2011
This is annoying me cos I know I'm not doing it efficiently.

i need to collect some info from an obect that contains N elements.
However I'm not allowed to ask it for more than 1000 elements at a
time. So if N == 3400 i need to make 4 calls to the object for
elements;

0 .. 999
1000 ... 1999
2000 ... 2999
3000 ... 3399


Whats the fastest loop I can write to achieve this, can anybody
suggest? I have it coded but I know I'm being inefficient. I'm not
being lazy here, just want to see what you guys would do, cos its
gonna be better than my loop, I know.

G

 
Reply With Quote
 
 
 
 
Jorgen Grahn
Guest
Posts: n/a
 
      03-24-2011
On Thu, 2011-03-24, graham wrote:
> This is annoying me cos I know I'm not doing it efficiently.
>
> i need to collect some info from an obect that contains N elements.
> However I'm not allowed to ask it for more than 1000 elements at a
> time. So if N == 3400 i need to make 4 calls to the object for
> elements;
>
> 0 .. 999
> 1000 ... 1999
> 2000 ... 2999
> 3000 ... 3399
>
>
> Whats the fastest loop I can write to achieve this, can anybody
> suggest? I have it coded but I know I'm being inefficient.


That seems unlikely, if "asking for 1000" elements is some kind of
I/O, like database access or something.

> I'm not
> being lazy here, just want to see what you guys would do, cos its
> gonna be better than my loop, I know.


A better question: what's the clearest way to express it?

Your situation is a special case of a problem I seem to encounter a
lot: you need to read in batches, but iterate item by item, and it's
not acceptable to read "all" into a vector and then iterate.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
 
Reply With Quote
 
 
 
 
Michael Doubez
Guest
Posts: n/a
 
      03-24-2011
On 24 mar, 09:56, graham <(E-Mail Removed)> wrote:
> i need to collect some info from an obect that contains N elements.
> However I'm not allowed to ask it for more than 1000 elements at a
> time. So if N == 3400 i need to make 4 calls to the object for
> elements;
>
> 0 .. 999
> 1000 ... 1999
> 2000 ... 2999
> 3000 ... 3399
>
> Whats the fastest loop I can write to achieve this, can anybody
> suggest?


Without knowing the interface, the fastest loop to code looks like:

int sz = 3400;

const in batch_size = 1000;
int index = 0;
for( ; sz >= batch_size; sz -= batch_size ) {
object.collect(index, batch_size);
index += batch_size;
}
if( sz ) {
object.collect(index,sz);
}

--
Michael
 
Reply With Quote
 
robertwessel2@yahoo.com
Guest
Posts: n/a
 
      03-24-2011
On Mar 24, 4:33*am, Michael Doubez <(E-Mail Removed)> wrote:
> On 24 mar, 09:56, graham <(E-Mail Removed)> wrote:
>
> > i need to collect some info from an obect that contains N elements.
> > However I'm not allowed to ask it for more than 1000 elements at a
> > time. So if N == 3400 i need to make 4 calls to the object for
> > elements;

>
> > 0 .. 999
> > 1000 ... 1999
> > 2000 ... 2999
> > 3000 ... 3399

>
> > Whats the fastest loop I can write to achieve this, can anybody
> > suggest?

>
> Without knowing the interface, the fastest loop to code looks like:
>
> int sz = 3400;
>
> const in batch_size = 1000;
> int index = 0;
> for( *; sz >= batch_size; sz -= batch_size ) {
> * object.collect(index, batch_size);
> * index += batch_size;}
>
> if( sz ) {
> * object.collect(index,sz);
>
> }



I prefer a slightly different form:

index = 0;
while (sz)
{
// determine amount to be processed, this may be more complex in
general,
// but in this case:
batch = min(sz, batch_limit));
dowork(index, batch);
index += batch;
sz -= batch;
}

It supports cases where the batch size varies for reasons other than
being the last element (not an issue for this example), and doesn't
duplicate the "work."

As for fastest it would typically be faster to not have the batch
size determination inside the loop, but the cost in most cases this
very small compared to the amount of work done in a batch (and in this
case the OP was talking about processing 1000 items in a batch).
 
Reply With Quote
 
graham
Guest
Posts: n/a
 
      03-25-2011
Boy am I glad I didn't post you my approach.. I knew it was ugly but
that shows me just how much Lordy lord.

Thanks a million for that, I better update cvs before anybody see's my
code

thanks

Graham
On Mar 24, 10:33*am, Michael Doubez <(E-Mail Removed)> wrote:
> On 24 mar, 09:56, graham <(E-Mail Removed)> wrote:
>
> > i need to collect some info from an obect that contains N elements.
> > However I'm not allowed to ask it for more than 1000 elements at a
> > time. So if N == 3400 i need to make 4 calls to the object for
> > elements;

>
> > 0 .. 999
> > 1000 ... 1999
> > 2000 ... 2999
> > 3000 ... 3399

>
> > Whats the fastest loop I can write to achieve this, can anybody
> > suggest?

>
> Without knowing the interface, the fastest loop to code looks like:
>
> int sz = 3400;
>
> const in batch_size = 1000;
> int index = 0;
> for( *; sz >= batch_size; sz -= batch_size ) {
> * object.collect(index, batch_size);
> * index += batch_size;}
>
> if( sz ) {
> * object.collect(index,sz);
>
> }
>
> --
> Michael


 
Reply With Quote
 
Michael Doubez
Guest
Posts: n/a
 
      03-25-2011
On 24 mar, 18:41, "(E-Mail Removed)" <(E-Mail Removed)>
wrote:
> On Mar 24, 4:33*am, Michael Doubez <(E-Mail Removed)> wrote:
> > On 24 mar, 09:56, graham <(E-Mail Removed)> wrote:
> > > i need to collect some info from an obect that contains N elements.
> > > However I'm not allowed to ask it for more than 1000 elements at a
> > > time. So if N == 3400 i need to make 4 calls to the object for
> > > elements;

>
> > > 0 .. 999
> > > 1000 ... 1999
> > > 2000 ... 2999
> > > 3000 ... 3399

>
> > > Whats the fastest loop I can write to achieve this, can anybody
> > > suggest?

>
> > Without knowing the interface, the fastest loop to code looks like:

>
> > int sz = 3400;

>
> > const in batch_size = 1000;
> > int index = 0;
> > for( *; sz >= batch_size; sz -= batch_size ) {
> > * object.collect(index, batch_size);
> > * index += batch_size;}

>
> > if( sz ) {
> > * object.collect(index,sz);

>
> > }

>
> I prefer a slightly different form:


There is more than one way to skin a cat.

All depends on the abstraction you want ot express.

> index = 0;
> while (sz)
> {
> * // determine amount to be processed, this may be more complex in
> general,
> * // but in this case:
> * batch = min(sz, batch_limit));
> * dowork(index, batch);
> * index += batch;
> * sz -= batch;
> }


With this form, I'd prefer:
int index = 0;
while ( index != sz )
{
int const batch = std::min(sz-index,batch_limit));
dowork(index, batch);
index += batch;
}

--
Michael
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
looping in array vs looping in a dic giuseppe.amatulli@gmail.com Python 5 09-20-2012 11:58 PM
RFC - Recursive Array Method call acting on element batches rubydeckard@gmail.com Ruby 3 11-27-2007 01:50 PM
Scanner for large batches of Slides Toby Digital Photography 6 09-08-2007 08:36 AM
Moving batches of files with Rake Phrogz Ruby 2 04-01-2007 01:19 AM
Full-size DVD cases in batches of 20 (price check) Bruce P. Burrell DVD Video 5 01-24-2006 01:56 AM



Advertisments