Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > fun with nested loops

Reply
Thread Tools

fun with nested loops

 
 
Daniel
Guest
Posts: n/a
 
      08-31-2011
Dear All,

I have some complicated loops of the following form

for c in configurations: # loop 1
while nothing_bad_happened: # loop 2
while step1_did_not_work: # loop 3
for substeps in step1 # loop 4a
# at this point, we may have to
-leave loop 1
-restart loop 4
-skip a step in loop 4
-continue on to loop 4b

while step2_did_not_work: # loop 4b
for substeps in step2:
# at this point, we may have to
-leave loop 1
-restart loop 2
-restart loop 4b
...
...many more loops...


I don't see any way to reduce these nested loops logically, they
describe pretty well what the software has to do.
This is a data acquisition application, so on ever line there is
a lot of IO that might fail or make subsequent steps useless or
require a
retry.

Now every step could need to break out of any of the enclosing loops.
So basically I have to transform every loop to be of the following
horror:

# general transformation of
# "for c in configurations..."
# to provide restart, break and continue
# callable from any nesting level inside of the loop

class loop1_restart(Exception): pass
class loop1_break(Exception): pass
class loop1_continue(Exception): pass

while True:
try:
for c in configurations:
while True:
try:
# inner loops go here, of course, they would have
to get
# all the boilerplate added, too
while nothing_bad_happened:
while step1_did_not_work:
if cond1:
raise loop1_restart()
elif cond3:
raise loop1_break()
elif cond3:
raise loop1_continue()

break

except loop1_continue:
pass
break
except loop1_restart:
pass
except loop1_break:
break

Of course this is extremely tedious and error prone.
If Python had named loops (PEP 3136, unfortunately rejected), this
would be trivial:
In Fortran I can continue (cycle), break (exit) and redo (goto label)
arbitrary
loops, which results in neat code:

10 loop1: do I=1,3
loop2: do J=1,4
print *,I,J
goto 10
cycle loop1
exit loop1
enddo loop2
enddo loop1


My question is, how do I obtain something that implements the
following logic:

@named_loop(fred) # I wish this would be possible
for i in range(10):
for j in range(10):
break fred # breaks out of outer loop
continue fred # continues outer loop
redo fred # resets outer loop and restarts with i=0


The best solution would be something along the Proposal D - Explicit
iterators
in PEP 3136, in this case it would even be possible to peek at the
next i or
advance/reverse the iterator a few steps.

Has anyone an idea on a nice way to write breaks/continues/redos for
deeply
nested loops?


Dan


 
Reply With Quote
 
 
 
 
aspineux
Guest
Posts: n/a
 
      08-31-2011
On Aug 31, 5:51*pm, Daniel <(E-Mail Removed)> wrote:
> Dear All,
>
> I have some complicated loops of the following form
>
> for c in configurations: # loop 1
> * * while nothing_bad_happened: # loop 2
> * * * * while step1_did_not_work: # loop 3
> * * * * * * for substeps in step1 # loop 4a
> * * * * * * * * # at this point, we may have to
> * * * * * * * * -leave loop 1
> * * * * * * * * -restart loop 4
> * * * * * * * * -skip a step in loop 4
> * * * * * * * * -continue on to loop 4b
>
> * * * * while step2_did_not_work: # loop 4b
> * * * * * * for substeps in step2:
> * * * * * * * * # at this point, we may have to
> * * * * * * * * -leave loop 1
> * * * * * * * * -restart loop 2
> * * * * * * * * -restart loop 4b
> * * * * * * * * ...
> * * * * ...many more loops...
>
> I don't see any way to reduce these nested loops logically, they
> describe pretty well what the software has to do.
> This is a data acquisition application, so on ever line there is
> a lot of IO that might fail or make subsequent steps useless or
> require a
> retry.
>
> Now every step could need to break out of any of the enclosing loops.
> So basically I have to transform every loop to be of the following
> horror:
>
> # general transformation of
> # "for c in configurations..."
> # to provide restart, break and continue
> # callable from any nesting level inside of the loop
>
> class loop1_restart(Exception): pass
> class loop1_break(Exception): pass
> class loop1_continue(Exception): pass
>
> while True:
> * * try:
> * * * * for c in configurations:
> * * * * * * while True:
> * * * * * * * * try:
> * * * * * * * * * * # inner loops go here, of course,they would have
> to get
> * * * * * * * * * * # all the boilerplate added, too
> * * * * * * * * * * while nothing_bad_happened:
> * * * * * * * * * * * * while step1_did_not_work:
> * * * * * * * * * * * * * *if cond1:
> * * * * * * * * * * * * * * * *raise loop1_restart()
> * * * * * * * * * * * * * *elif cond3:
> * * * * * * * * * * * * * * * *raise loop1_break()
> * * * * * * * * * * * * * *elif cond3:
> * * * * * * * * * * * * * * * *raise loop1_continue()
>
> * * * * * * * * * * break
>
> * * * * * * * * except loop1_continue:
> * * * * * * * * * * pass
> * * * * break
> * * except loop1_restart:
> * * * * pass
> * * except loop1_break:
> * * * * break
>
> Of course this is extremely tedious and error prone.
> If Python had named loops (PEP 3136, unfortunately rejected), this
> would be trivial:
> In Fortran I can continue (cycle), break (exit) and redo (goto label)
> arbitrary
> loops, which results in neat code:
>
> 10 loop1: do I=1,3
> * * loop2: do J=1,4
> * * * * print *,I,J
> * * * * goto 10
> * * * * cycle loop1
> * * * * exit loop1
> * * enddo loop2
> enddo loop1
>
> My question is, how do I obtain something that implements the
> following logic:
>
> @named_loop(fred) # I wish this would be possible
> for i in range(10):
> * * for j in range(10):
> * * * * break fred # breaks out of outer loop
> * * * * continue fred # continues outer loop
> * * * * redo fred # resets outer loop and restarts with i=0
>
> The best solution would be something along the Proposal D - Explicit
> iterators
> in PEP 3136, in this case it would even be possible to peek at the
> next i or
> advance/reverse the iterator a few steps.
>
> Has anyone an idea on a nice way to write breaks/continues/redos for
> deeply
> nested loops?


Hi Dan, it looks like you have already answered all your questions.

one more idea, a kind of named loop:

ic=0
op='what to do'
while ic<len(configurations):
c=configuration[ic]

if op=='restart':
ic=0
elif op=='next'
ic+=1
elif op=='terminate'
ic=len(configurations)

and at first explicit iterator was also a good idea, when you don't
need to iter in reverse order

When it become too complicate, I use state machine:
http://en.wikipedia.org/wiki/Finite-state_machine


Regards


>
> Dan


 
Reply With Quote
 
 
 
 
Chris Angelico
Guest
Posts: n/a
 
      08-31-2011
On Thu, Sep 1, 2011 at 1:51 AM, Daniel <(E-Mail Removed)> wrote:
>
> Has anyone an idea on a nice way to write breaks/continues/redos for
> deeply
> nested loops?
>


Do you only ever have one top-level loop that you would be naming? If
so, put that loop into a function and use return instead of break.
Unfortunately that doesn't work for continue.

ChrisA
 
Reply With Quote
 
Daniel
Guest
Posts: n/a
 
      08-31-2011

> one more idea, a kind of named loop:

interesting idea, thanks.


>
> When it become too complicate, I use state machine:http://en.wikipedia.org/wiki/Finite-state_machine

I unsuccessfully played a bit with a FSM, but there is a lot of data
that is passed around between the states and a lot of counting (like
trying a certain step n times), so the FSM turned out to be even more
complex. And I have to keep the code simple for non CS people to run
the actual experiment. The loops are kind of self-explanatory, this is
exactly how you would specify the experiment, even though I am really
hitting a wall at the moment.

Maybe I am really missing an obvious solution, because breaking out of
nested loops really doesn't seem like anything fancy. Fortran/c/c++/
Ruby/Perl all have that facility, even Java has named loops.
 
Reply With Quote
 
Daniel
Guest
Posts: n/a
 
      08-31-2011

> Do you only ever have one top-level loop that you would be naming? If

no, unfortunately not. The rough structure is several loops deep, and
I need to break/continue/restart many of them.
Continue is used more than break, because most of the time that I find
some strange value, I'd just _continue_ a few levels up
to restart the current measurements.


for some configurations
while not enough data collected
while not in the right state
for steps in steps to bring the system to the right state
if the system is bad, break out of all loops
if it just need to be reset, just redo the steps
if it is ok, go to the next while loop
while in the right state
steps to do some measurements
...



 
Reply With Quote
 
Chris Angelico
Guest
Posts: n/a
 
      08-31-2011
On Thu, Sep 1, 2011 at 5:07 AM, Daniel <(E-Mail Removed)> wrote:
>> Do you only ever have one top-level loop that you would be naming? If

> no, unfortunately not. The rough structure is several loops deep, and
> I need to break/continue/restart many of them.
> Continue is used more than break, because most of the time that I find
> some strange value, I'd just _continue_ a few levels up
> to restart the current measurements.
>


Ah well, was worth a try. Raising exceptions smells wrong for this,
but peppering your code with sentinel checks isn't much better. I
don't really know what would be a good solution to this... except
maybe this, which was proposed a few years ago and which I'd never
heard of until Google showed it to me just now:
http://entrian.com/goto/

ChrisA
 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      09-01-2011
Daniel wrote:

> And I have to keep the code simple for non CS people to run
> the actual experiment.


Do you think the software in the Apple iPod is "simple"? Or Microsoft
Windows? No. You need to keep the *interface* simple. The internal details
can be as complicated as they are needed to be.

Same applies to your data acquisition application. Unless you expect these
non-CS people to be hacking the source code, they only interact with the
interface, not the internals.

Earlier, back in your initial post, you said:

"I don't see any way to reduce these nested loops logically, they
describe pretty well what the software has to do.
This is a data acquisition application, so on ever line there is
a lot of IO that might fail or make subsequent steps useless or
require a retry."

Do you think you're the first person to have written a data acquisition
application in Python? Almost certainly you can simplify the structure of
the code by splitting it into functions appropriately, instead of the
spaghetti code you have (apparently) written with jumps all over the place.
To take the most obvious, simple example: any time you have a loop that you
might want to redo, the right solution is to put the loop inside a
function, and then "redo the loop" becomes "call the function again".

I suppose that, just possibly, your application really would benefit from
named labels to jump to. But if so, you've stumbled across something rarer
than iridium.



--
Steven

 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      09-01-2011
Chris Angelico wrote:

> Ah well, was worth a try. Raising exceptions smells wrong for this,
> but peppering your code with sentinel checks isn't much better. I
> don't really know what would be a good solution to this... except
> maybe this, which was proposed a few years ago and which I'd never
> heard of until Google showed it to me just now:
> http://entrian.com/goto/


You're a wicked, wicked man.




--
Steven

 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      09-01-2011
Steven D'Aprano wrote:

[...]
> Do you think you're the first person to have written a data acquisition
> application in Python?


Hmmm, on re-reading that it comes across as much harsher than it sounded in
my head. Sorry about that Daniel.


--
Steven

 
Reply With Quote
 
Daniel
Guest
Posts: n/a
 
      09-01-2011
Hi Steve,

Thanks for your comments, I appreciate any input.
> Do you think the software in the Apple iPod is "simple"? Or Microsoft

No, that's much more complicated that what I am doing.
But the iPod probably (?) doesn't get new algorithms based on a
specification discussed with non-programmers once a month.

I didn't explain enough of what I am doing. This is a fairly complex
application that has been running for a few years now,
I am just trying to improve it. The code runs a big
test machine, that runs >100000 individual tests in one run on
something like semiconductor chips.

The specification of these tests is already very complex, it has the
form of the nested loops,
for all these configurations try these steps, if they fail try them
again n times, if it still doesn't work give up
this configuration, if it works continue on to the next steps etc.
That's the form the specification is in, and it makes sense and is
very readable.
In pseudocode it looks like this, I am using @ to give loops a name:

@loop1
for c in configurations:
@loop2
while not_done:
@loop3
while step1_did_not_work:
@loop4
for substeps in step1 # loop 4a
if hopeless(): continue loop1 # run next configuration
if substepsFailed(): restart loop4 # try again
if substepsWorked(): break loop3 # go on to next
steps, like loop4

That format is fine, everyone involved can understand it, even the
people in charge. I'd like to make this executable without
changing too much of the form. It would be possible to do this as a
FSM, but then you'd loose the line to line correspondence with the
specification, and of course some errors always creep in.

> non-CS people to be hacking the source code, they only interact with the

This is a research setting, so the person running the machine will
have to change the source from time to time if
he gets a new specification. The specifications are far to complex to
be entered into a user interface because of all the loops.

> the code by splitting it into functions appropriately, instead of the
> spaghetti code you have (apparently) written with jumps all over the place.

I wouldn't call the example above spaghetti code in the sense of old
Fortran or Basic full of gotos.
In a language that can break out of nested loops this is highly
structured code.
I am not jumping forward to labels, not jumping into functions, not
using jumps to replace loops etc.

It is like the Fortran example (just to show the syntax, has an
infinite loop), everyone can understand that right away, even
non Fortran people:

10 loop1: do I=1,3
loop2: do J=1,4
print *,I,J
goto 10 ! redo loop1
cycle loop1
exit loop1
enddo loop2
enddo loop1

There is no wild jumping her. The only problem is that Fortran does
not allow to restart a loop, so instead of restart loop1 you have to
do a goto 10. Otherwise you could do entirely without gotos (like in
Ruby with the redo, which is of course much much better)

> To take the most obvious, simple example: any time you have a loop that you
> might want to redo, the right solution is to put the loop inside a
> function, and then "redo the loop" becomes "call the function again".

Doesn't work, because it can only redo one level, to break out of the
next loop, I'd need exceptions anyway.
And having all of these exceptions as gotos doesn't make it more
readable.
Replacing loop4 by a function makes it possible to replace the restart
loop4 by a return, but then I still need an exception to
continue loop1 and one to break out of loop4 to indicate that we can
go on to the next step.

> I suppose that, just possibly, your application really would benefit from
> named labels to jump to. But if so, you've stumbled across something rarer
> than iridium.


Don't think so. I am doing that all of the time in other languages,
and I am convinced the named loops (not raw labels+gotos, which are
just a necessary evil) are beautiful and clean things.
They have a lot of symmetry, break is always break, not sometimes
break, sometimes return and sometimes raise breakSomeLoopExcept().
Rewriting the simple Fortran example above in Python would be much,
much uglier and more difficult to comprehend.
You always call break and continue with a label, searching for that
label will tell you right away which loop the break breaks.
I am always doing that, even if there is only one loop. break and
continue (without label) are IMO (please no flame war about that)
worse than goto, at least the goto tells you where it goes, with break/
continue you always have to scan the surroundings to find the right
loop.

I know I am not the only one who is trying to solve that problem, I
was hoping someone has come up with a hack to solve it, like this goto
Chis has come up with. I have to play with that a bit.



Dan

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
3 PIX VPN questions - FUN FUN FUN frishack@gmail.com Cisco 3 03-16-2006 02:25 PM
Loops with loops using html-template Me Perl Misc 2 01-12-2006 05:07 PM
OT: Wednesday follow-up-to-Tuesday-Fun Fun Ken Briscoe MCSE 0 07-14-2004 01:41 PM
Programming is not as much fun/more fun than it used to be. Andy Fish Java 65 05-18-2004 08:24 PM
Fun fun fun Luke Computer Support 3 10-07-2003 03:45 PM



Advertisments