Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > SyncEnumerator?

Reply
Thread Tools

SyncEnumerator?

 
 
Emmanuel Touzery
Guest
Posts: n/a
 
      07-10-2004
--Boundary-00=_IGG8AgQIywgLsRv
Content-Type: Multipart/Mixed;
boundary="Boundary-00=_IGG8AgQIywgLsRv"


--Boundary-00=_IGG8AgQIywgLsRv
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hello,

Just to share an experience of something that happened to me today; i wrote a
script and it took all the memory of my computer and I had no idea why...
Then after many hours, I had a flash.. Maybe SyncEnumerator was guilty, since
it (does it?) seems to use continuations...
Well.. in my code I actually could change the use of SyncEnumerator by an
Array#zip.. the result: the calculation took about no memory (instead of
several hundred megabytes) and finished almost instantly (as it always should
have)!

So, maybe just a warning: SyncEnumerator is not as cheap as it seems: prefer
Array#zip, and at the moment I can't think of a reason when SyncEnumerator
brings something over Array#zip since Array#zip is so much faster...

Attached a script.. it doesn't actually work to the end (the script is
working on data that I can't send here), but you can see the difference if
running with SyncEnumerator or Array#zip. Amazing!

At line 12 in the script:
# s = SyncEnumerator.new(persons, values)
s = persons.zip(values)

you can switch the comment between the two lines to see the change.

my conclusion for now: maybe SyncEnumerator will be one day more readable than
Array#zip, but in current implementations of ruby, it's way too slow..

emmanuel
--Boundary-00=_IGG8AgQIywgLsRv
Content-Type: text/plain;
charset="us-ascii";
name="raz.cvs"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment; filename="raz.cvs"

198 199 200 201 202 203 204 205 206 207
col1 M M M M M M M M M M
col2 25 24 25 23 25 22 20 18 25 28
col3 aaaaaaaaaaa aaaaaaa aaaaaa aa=C5=A1aaa =C5=A1aaaaaa aaaaaaaaaaa =C5=A1=
aaaaaa aaaaaaaaaaa a.a. aaaaaaa aaaaa
col4 b b b b b b b b.o. a b
col5 4 4 3 4 3 5 3 3 3 2
M N S N N N M M N M
aaaaa - - aa=C4=8Daaa aaaa=C5=A1a - - - - - aaaaaaa, aaaaaaa aaaaaaaaaa aaa=
aa -
=09
=09
=09
198 199 200 201 202 203 204 205 206 207
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
208 209 210 211 212 213 214 215 216 217
col1 M M M M M M =C5=BD =C5=BD M =C5=BD
col2 14 25 24 25 22 17 25 23 25 20
col3 aaaaaaa=C5=A1aaaa aaaaaaaaa aaaaaa aaaaaaa aaaaaaaaaaa aaaaaaaa aaaaaa=
a aaaaa aaaaaaa =C5=A1aaaaaaaa =C5=A1aaaaaa =C5=A1aaaaaaaa
col4 b a b b a a b b b b
col5 4 4 2 4 5 3 2 5 3 3
N N S N N N N N N N
208 209 210 211 212 213 214 215 216 217
208 209 210 211 212 213 214 215 216 217
=09
=09
=09
=09
218 219 220 221 222 223 224 225 226 227
col1 izlo=C4=8Den, opisan primer 38 letnega =C5=BD M M =C5=BD =C5=BD M M M =
=C5=BD
col2 21 25 25 25 19 20 20 18 24
col3 =C5=A1aaaaaaaa =C5=A1aaaaaa aaaaaaaa aaaaaa aaaaaaa aaaaaaaaa aaaaaa =
aaaaa=C4=8Daaa aaaaaaa =C5=A1aaa aaa=C4=8Daaa aaaaaaa =C5=A1aaa aaaa=C4=8Da=
aa aaaaaaaaaa =C5=A1aaaaaaaa
col4 b b a b b b a a b
col5 4 4 4 3 4 2 3 2 5
218 219 220 221 222 223 224 225 226 227
M N S M M M M
218 219 220 221 222 223 224 225 226 227
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
228 229 230 231 232 233 234 235 236 237
col1 M M M M M M M M =C5=BD M
col2 25 22 25 23 23 20 25 27 22 24
col3 aaaaaa=C4=8Daa aaaaaaa-aaaaaaaa aaaaaaaaaaa aaaaaaa aaaaaaaa aaaaaa aa=
aaaaaaaaa aaaaaaa=C4=8Daa aaaaaa aaaaa =C5=A1aaaaaaaa aaaaa aaaaaa
col4 b b.o. b a b b b b a b
col5 2 3 3 4 3 4 3 3 5 3
M M M N E M N N N S
228 229 230 231 232 233 234 235 236 237
228 229 230 231 232 233 234 235 236 237
=09
=09
=09
=09
238 239 240 241 242 243 244 245 246 247
col1 a =C5=BD a a aaaa=C4=8Daa, aaaaaa aaaaaa aa aaaaaaa aaaa=C4=8Daa, aaaa=
aaaaaaaaa aaaaa a a a a
col2 25 24 20 19 25 21 21 18
col3 =C5=BEaaaaaaaa a.a. a.a. aaaaa =C5=A1aaaaaa aaaaaaa =C5=A1aaa aaaaa =
aaaaa
col4 a b b b b b a b
col5 3 do4 2 3 2 4 3 3 do 4 4
238 239 240 241 242 243 244 245 246 247
E M N E S N E M
238 239 240 241 242 243 244 245 246 247
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
248 249 250 251 252 253 254 255 256 257
col1 =C5=BD M M M =C5=BD M M M M M
col2 21 24 22 25 25 21 24 22 19 24
col3 aaaaaaaaa aaaaaa aaaa aaaaaaa =C5=A1aaaaaa a.a. =C5=A1aaaaaaaa =C5=A1a=
aaaaa aaaaaaaaaa =C5=A1aaaaaa aaaaa aaaaaaaa aaaaaa
col4 b.o. brez poklica b a b b b.o. b a b
col5 4 2 4 3 5 4 do 5 3 4 4 2
M M M M M N S E N M
248 249 250 251 252 253 254 255 256 257
=09
=09
=09
=09
=09
=09
=09
258 259 260 261 262 263 264 265 266 267
col1 M M M M M M M M M M
col2 17 21-22 23 25 23 29 28 20 26 21
col3 aaaaaaa=C5=A1aaaa aaaaaaa aaaaaaaaaaa aaaaaaa, aaaaaaaa aa aaaa aaaaaa=
aaa =C5=A1aaaaaa aaaaaaa=C4=8Daa aaaaa aaaaaaa=09
col4 b b b b.o. b a b b b a
col5 1-6razred 5,7-8 pa2 3 3 3 5 3 3 3 3 2
M M N S E S N S M S
258 259 260 261 262 263 264 265 266 267
258 259 260 261 262 263 264 265 266 267
=09
=09
268 269 270 271 272 273 274 275 276 277
col1 M M M M M M M =C5=BD M M
col2 22 20 21 22 22 24 21 17 25 15
col3 aa=C4=8Daaaaaa=C5=A1aa aaaaaa aaaaaaa aaaaa aa =C5=A1aaa aaaaa aa =C5=
=A1aaa aaaaa aaaaa=C4=8Daa aaaaaa aaaaaaaa aaaaaa aa aaaaa =C5=A1aaa aaa=C4=
=8Daaa a=C5=A0 aa aaaaa =C5=A1aaa
col4 b b b b b b b b b b
col5 b.o. 3 5 3 2 3 3 3 2 3
268 269 270 271 272 273 274 275 276 277
M M N N M N M N S E
268 269 270 271 272 273 274 275 276 277
=09
=09
=09
=09
278 279 280 281 282 283 284 285 286 287
col1 M =C5=BD =C5=BD =C5=BD M M M M M M
col2 21 22 18 26 23 21 24 23 25 24
col3 aa aaaaa =C5=A1aaa aa aaaaa =C5=A1aaa aa aaaaa =C5=A1aaa aa aaaaa =C5=
=A1aaa aaaaaaaa aaaaaa aaaaaaa aa aaaaa =C5=A1aaa aaaaaaaaaaa aaaaaa a.a. a=
aaa a aaaaaaaa a aaaaaaaa, aaa=C4=8Daaa aaaaaa
col4 b b b b b b b b b b
col5 3 5 b.o. 5 3 ali4 3 4 5 2 4
278 279 280 281 282 283 284 285 286 287
S N M E N M E N N M
278 279 280 281 282 283 284 285 286 287
=09
=09
=09
=09
=09
=09
288 289 290 291 292 293 294 295 296 297
col1 M M M M M M M M M M
col2 18 20 23 25 21 25 21 23 20 27
col3 aaaaa aa =C5=A1aaa a.a. aaaaaaaaaa aaaaaa aa aaaaa =C5=A1aaa aaaaaa aa=
aaa aaa=C4=8Daa a=C5=A0 aaaaa aa =C5=A1aaa aaaaaa
col4 b b b b.o. a b b a b b
col5 3 3 4 b.o. 3 b.o. 3 2 4 3
288 289 290 291 292 293 294 295 296 297
M M N M E M N M N N
288 289 290 291 292 293 294 295 296 297
=09
=09
=09
298 299 300 301 302 303 304 305 306 307
col1 M M M M M M M M M M
col2 25 25 18 20 21 28 22 25 22 24
col3 aaaaaaaaa aaaaaa aaaaaaa aaaaaaaaaaa aaaaa aa aaaaa =C5=A1aaa aaaaaa a=
=2Ea. aaaa aaaaaaa a.a. =C5=BEaaaaaaa aaaaaa aaaaaaaaa aaaaaa
col4 b b b b b b a a b b
col5 3 2 zadosten 5 3 3 2 2 2 4
N S N N E M M E S N
298 299 300 301 302 303 304 305 306 307
=09
=09
=09
=09
=09
=09
=09
=09
=09
308 309 310 311 312 313 314 315 316 317
col1 M M M M M M =C5=BD M =C5=BD ni ankete
col2 25 24 18 23 25 24 23 22 21=09
col3 aaaa=C4=8Daaaa=C4=8Daa a.a. aa aaaaa =C5=A1aaa aaaaa aaaaaa: aaaaaaa a=
aaaaaa aaaaaaaa aa aaaaa =C5=A1aaa aa =C5=A1aaa=09
col4 b b b b b b b b b=09
col5 3 2 5 3 3 4 4 5 2=09
N S M N N M M M E=09
308 309 310 311 312 313 314 315 316 317
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
318 319 320 321 322 323 324 325 326 327
col1 M M M =C5=BD M M M M M ni ankete
col2 19 21 26 20 20 23 23 24 23=09
col3 aaaaaaaaaaaaaaa aaaaaaa aaaaa aaaaaaaaa aa aaaaa =C5=A1aaa aaaaa aaaaa=
aa a aaaaaaa aaaaaa a.a. aaaaa=09
col4 b a b b b b b b b=09
col5 4 2 3 5 3 2 3 2 b.o.=09
M N N M M N E E M=09
318 319 320 321 322 323 324 325 326 327
=09
=09
=09
=09
=09
=09
=09
=09
=09
328 329 330 331 332 333 334 335 336 337
col1 M =C5=BD M =C5=BD M M =C5=BD =C5=BD M M
col2 24 23 24 17 22 22 21 24 19 24
col3 aa aaaaa =C5=A1aaa aa aaaaa =C5=A1aaa a.a. aa aaaaa =C5=A1aaa aaaaaaaa=
aaaaaaaa a.a. aa aaaaa =C5=A1aaa aaaaa-aaaa=C5=A1=C4=8Daa aaaaaa
col4 b b b.o. b.o. b b b b a b
col5 4 5 4 4 3 b.o. 4 4 3 4
N N N M N M N N N E
328 329 330 331 332 333 334 335 336 337
=09
=09
=09
=09
=09
338 339 340 341 342 343 344 345 346 347
col1 M M =C5=BD M =C5=BD =C5=BD =C5=BD M =C5=BD M
col2 24 17 20 24 18 25 21 15 16 20
col3 aa aaaaa =C5=A1aaa aa aaaaa =C5=A1aaa aa aaaaa =C5=A1aaa aaaaaaaaaaa a=
a aaaaa =C5=A1aaa aaaaaaaa aa aaaaa =C5=A1aaa aaaaa aa =C5=A1aaa aaaaa aa =
=C5=A1aaa aaaaa aa =C5=A1aaa
col4 b b b b b b b b b b
col5 4 3 4 3 4 3 5 3 4 5
M N M M M E S M N M
338 339 340 341 342 343 344 345 346 347
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
=09
348 349 350 351 352 353 354 355 356 357
col1 M =C5=BD M M =C5=BD M M M M M
col2 20 22 25 24 20 24 23 20 20 23
col3 aa aaaaa =C5=A1aaa aaaaa aa =C5=A1aaa aaaaaaaaa aaaaaa aaaaaaa a.a. aa=
=C4=8Daa aaaaaaaaaa aaaa aaaaaaaaa=C5=A1aaa a aaaaaaa aa aaaaa =C5=A1aaa aa=
aaaaa =C5=A1aaa
col4 b b b b b b b b b a
col5 5 5 3 3 4,5 3 3 4 2 3
E N N S N N N S N E
348 349 350 351 352 353 354 355 356 357
348 349 350 351 352 353 354 355 356 357
=09
=09
=09
=09
358 359 360 361 362 363 364 365 366 367
col1 M M =C5=BD M ni ankete M M M =C5=BD M
col2 22 20 18 17 20 23 22 20 25
col3 aaaaa a.a. aa aaaaa =C5=A1aaa aa aaaaa =C5=A1aaa aaaaaaaa aaaaa aaaaa=
aaaa aaaaaa a.a. aaaaaaa aaaaaaa
col4 a b b b b a b b a
col5 b.o. 3 5 3 3 3 3 2 2do 3
M E N M M N M M E
358 359 360 361 362 363 364 365 366 367
=09
=09
368 369 370 371 372 373 374 375 376 377
col1 =C5=BD =C5=BD =C5=BD M M M M =C5=BD M M
col2 18 20 23 18 25 21 22 15 23 24
col3 aa aaaaa =C5=A1aaa aaaaa aa =C5=A1aaa aa aaaaa =C5=A1aaa aa aaaaa =C5=
=A1aaa aaaa.aaaaaaaaa aa aaaaa =C5=A1aaa aa aaaaa =C5=A1aaa aa aaaaa =C5=
=A1aaa aa aaaaa =C5=A1aaa aaaaaaaaaaa aaaaaa
col4 b b b a b b b a b a
col5 4 5 3 2 5 3 4 4 3 4
N N D M M M N E N E
368 369 370 371 372 373 374 375 376 377
=09
=09
=09
=09
=09
=09
=09
378 379 380 381 382 383 384 385 386 387
col1 M M =C5=BD M M M M M M M
col2 25 20 20 23 23 22 25 19 25 24
col3 aaaaaaa aaaaaa aa aaaaa =C5=A1aaa aa aaaaa =C5=A1aaa aa aaaaa =C5=A1aa=
a aaaaa aaaaaaaaaaa aaaaaaaaa aa=C5=A1aaaaaaa aaaaa aa =C5=A1aaa aaaaaaaaaa=
a aaaaa
col4 b b b b b b a b a b
col5 4 4 4 3 do 4 2 2 3 3 3 4
M M M N N S N N M M
378 379 380 381 382 383 384 385 386 387
=09
=09
=09
=09
=09
=09
=09
=09
388 389 390 391 392 393 394 395 396 397
col1 ni ankete =C5=BD =C5=BD M M M M izlo=C4=8Den, anketiran 40 letni mo=C5=
=A1ki M napaka
col2 22 20 24 21 21 18 21=09
col3 aa aaaaa =C5=A1aaa aa aaaaaaa =C5=A1aaa aaaaaaaaaaaaa aaaaa aaaa aaa =
aaaaaaa aa aaaaa =C5=A1aaa aaaaaaa=09
col4 b b a b b b b=09
col5 4 4 5 3 5 3 3=09
b.o. M N M N N M=09
388 389 390 391 392 393 394 395 396 397
388 389 390 391 392 393 394 395 396 397
=09
=09
=09
=09
398 399 400 401 402 403 404 405 406 407
col1 a a aaaaaa a aaaaaa aaaa=C4=8Daa, aaaa aaaaaaaaa aaaaa aaaa=C4=8Daa-aa=
aaa aaaa aaaa=C4=8Daa-aaaaa aaaa aaaa=C4=8Daa-aaaaa aaaa a
col2 23 25 24 21
col3 aaaaaaaa aaaaaaaa aaaaaa aaaaaaa =C5=A1aaaaaa-aaaaa aa =C5=A1aaa
col4 a b b b
col5 3 3 3 5
M M N N
398 399 400 401 402 403 404 405 406 407
=09
=09
=09
=09
408 409 410 411 412 413 414 415 416 417
col1 =C5=BD M M M M M M M =C5=BD M
col2 24 25 20 22 25 23 21 21 23 25
col3 aaaa=C5=BEaaaaa aaaaaa aaaaaaa aaaa aaaaaaa aaaaa aa =C5=A1aaa-=C5=A1a=
aaaaa aaaaaaaa aaaaaa aaaaa=C4=8Daa aaaaaa a.a. aaaaa aa =C5=A1aaa-=C5=A1aa=
aaaa aaaaaaaaa aaaaaa aaaaaa
col4 b b brez poklica b b b b b b b
col5 5 3 2 5 3 4 3do4 b.o. 4 2
N M M S N E M S E S
408 409 410 411 412 413 414 415 416 417
=09
=09
=09
=09
=09
=09
=09
418 419 420 421 422 423 424 425 426 427
col1 =C5=BD =C5=BD =C5=BD M M M =C5=BD M M M
col2 16 21 25 20 25 24 21 20 24 21
col3 aaaaa aa =C5=A1aaa-aaaaaaaaa aaaaa aa =C5=A1aaa aaaaa aa =C5=A1aaa aaa=
aa aa =C5=A1aaa =C5=A1aaaaaa aaaaaaa aaaaaa aaaaaaa=C4=8Daaa a=C5=A0 aaaaaa=
aaaaa aaaaaaa aaaaaa
col4 b b b b a b b b b.o. b
col5 3 4 4 4 4 4 4 3 2 2
N M N M N N M S S N
418 419 420 421 422 423 424 425 426 427
=09
=09
=09
=09
=09
428 429 430 431 432 433 434 435 436 437
col1 M M M izlo=C4=8Den-ni anketnega lista =C5=BD M izlo=C4=8Den, ni anketn=
ega lista M M =C5=BD
col2 20 23 17 28 24 23 23 22
col3 aaaaa aa =C5=A1aaa aaaa=C5=BEaa aaaaa aaaaaaa-aaaaaaaaaaaaaaa a aaaaaa=
a-aaaaaaaaa aaaaa aa =C5=A1aaa aaaaaaaaaaa aaaaa=C4=8Daaaa aaaaaaa aaaaaa=
=2Daaaaaaaaa aa aaaaaaaaa =C4=8Daaaaaa aaaaa aaaaa-aaaaa aa =C5=A1aaa
col4 a b b b b a a a
col5 3 3 5 5 3 3 2 4
E N S b.o. M S E E
428 429 430 431 432 433 434 435 436 437
=09
=09
=09
=09
=09
=09
438 439 440 441 442 443 444 445 446 447
col1 M M M M =C5=BD =C5=BD =C5=BD M =C5=BD M
col2 24 25 26 24 22 16 25 23 24 24
col3 aaaaaaaaa aaaaaa aaaaaaaa aaaaaa aaaaaaaaaaaa =C5=A1aaaaaa-aaaaa aa =
=C5=A1aaa =C5=A1aaaaaa-aaaaa aa =C5=A1aaa aaaaaaaaa-aaaaa aa =C5=A1aaa aaaa=
aaaaaa,aaaaaaa aaaaaaa aaaaaaaaaa aaaa=C4=8Daaaa=C4=8Daa
col4 b b b b b b b a b a
col5 4 4 3 3 4 5 3 3 4 3
M N N N M N N N M M
438 439 440 441 442 443 444 445 446 447
438 439 440 441 442 443 444 445 446 447
=09
=09
=09
=09
448 449 450 451 452 453 454 455 456 457
col1 M =C5=BD M M M M M =C5=BD =C5=BD M
col2 23 23 24 25 23 22 23 24 16 24
col3 =C5=A1aaaaaa =C5=A1aaaaaa aaaa aaaaaaa aaaaaa aaaaaaaaaaaa aaaa aaaaa=
aa a.a. =C5=A1aaaaaaaa aaaaaaaaa aaaaaaa
col4 a b brez poklica b b brez poklica b.o. b iz b v a b
col5 3 4 4 b.o. 3 2 4 5 4 4
448 449 450 451 452 453 454 455 456 457
M M M N S S E N N N
448 449 450 451 452 453 454 455 456 457
=09
=09
=09
=09
=09
=09
=09
458 459 460 461 462 463 464 465 466=09
col1 M M M M M M =C5=BD M M=09
col2 25 24 20 25 25 20 20 24 21=09
col3 =C5=A1aaaaaa aaaaaa=C5=A1=C4=8Daaa aaaa aaaaaaa aaaaaaaa aaaaaaa aaaaa=
aa-aaaaaaa aaaaaaaaa aaaaaaaaaa aaaaa=09
col4 a a a b.o. a b b a b=09
col5 4,5 3 3 4 do 5 3 3 4 4 3=09
N M M M M E M E M=09
458 459 460 461 462 463 464 465 466=09
458 459 460 461 462 463 464 465 466=09

--Boundary-00=_IGG8AgQIywgLsRv
Content-Type: text/plain;
charset="us-ascii";
name="parser2.rb"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="parser2.rb"

require 'generator'

def usage(msg); puts msg; exit 1; end

def parse_line(persons, lines, member, col_name)
while (line = lines.next.chomp) !~ /^#{Regexp.escape(col_name)}/
#puts "#{line} doesn't match skipping it"
end
values = line.split /\t/
values.shift # remove the column name
# s = SyncEnumerator.new(persons, values)
s = persons.zip(values)
s.each { |person, value|
if person.send(member)
# not overwriting if value is not nil
next
end
_value = if block_given?; yield value; else value; end
person.send(member.to_s + '=', _value)
}
end

class Person
attr_accessor erson_id, :is_male, :age, :job, :state,
:education

def initialize(person_id)
@person_id = person_id
end
end

if $0 == __FILE__

fName = ARGV.shift || usage("missing filename of statistics file")
f = File.open(fName)
lines = Generator.new(f)

all_persons = []

while lines.next?

line = lines.next.chomp while (lines.next? && line !~ /^(\t\d+)+\s*$/)
break if line !~ /^(\t\d+)+\s*$/

# found a series of data. first line => series
# of person numbers
person_ids = line.scan(/\d+/).map {|i| i.to_i}
line = nil # clear for next iteration
persons = person_ids.map { |p_id| Person.new(p_id) }

# now the series of genders
parse_line(persons, lines, :is_male, 'col1') { |gender_s| (gender_s == 'M') }
# now the age
parse_line(persons, lines, :age, 'col2')
# now the job
parse_line(persons, lines, :job, 'col3')
# now the stanje
parse_line(persons, lines, :state, 'col4')
# now the education
parse_line(persons, lines, :education, 'col5')

all_persons.concat(persons)
puts "parsed #{all_persons.size} people"
File.open('/proc/meminfo') {|memi| puts memi.grep(/MemFree/)} if RUBY_PLATFORM =~ /linux/
end

f.close

File.open('data.dta', 'w') do |file|
Marshal.dump(all_persons, file)
end

end

--Boundary-00=_IGG8AgQIywgLsRv--

--Boundary-00=_IGG8AgQIywgLsRv--


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off




Advertisments