Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   How to pop the interpreter's stack? (http://www.velocityreviews.com/forums/t740148-how-to-pop-the-interpreters-stack.html)

kj 12-14-2010 09:14 PM

How to pop the interpreter's stack?
 



Consider this code:


def spam(*args, **kwargs):
args, kwargs = __pre_spam(*args, **kwargs)

# args & kwargs are OK: proceed
# ...


def __pre_spam(*args, **kwargs):
# validate args & kwargs;
# return canonicalized versions of args & kwargs;
# on failure, raise some *informative* exception
# ...

return canonicalized_args, canonicalized_kwargs


I write functions like __pre_spam for one reason only: to remove
clutter from a corresponding spam function that has a particularly
complex argument-validation/canonicalization stage. In effect,
spam "outsources" to __pre_spam the messy business of checking and
conditioning its arguments.

The one thing I don't like about this strategy is that the tracebacks
of exceptions raised during the execution of __pre_spam include one
unwanted stack level (namely, the one corresponding to __pre_spam
itself).

__pre_spam should be completely invisible and unobtrusive, as if
it had been textually "inlined" into spam prior to the code's
interpretation. And I want to achieve this without in any way
cluttering spam with try/catches, decorators, and whatnot. (After
all, the whole point of introducing __pre_spam is to declutter
spam.)

It occurs to me, in my innocence (since I don't know the first
thing about the Python internals), that one way to achieve this
would be to have __pre_spam trap any exceptions (with a try/catch
around its entire body), and somehow pop its frame from the
interpreter stack before re-raising the exception. (Or some
clueful/non-oxymoronic version of this.) How feasible is this?
And, if it is quite unfeasible, is there some other way to achieve
the same overall design goals described above?

TIA!

~kj



Ethan Furman 12-15-2010 02:31 AM

Re: How to pop the interpreter's stack?
 
kj wrote:
> The one thing I don't like about this strategy is that the tracebacks
> of exceptions raised during the execution of __pre_spam include one
> unwanted stack level (namely, the one corresponding to __pre_spam
> itself).
>
> __pre_spam should be completely invisible and unobtrusive


I am unaware of any way to accomplish what you desire. I also think
this is one of those things that's not worth fighting -- how often are
you going to see such a traceback? When somebody makes a coding
mistake? I would say change the name (assuming yours was a real
example) to something more meaningful like _spam_arg_verifier and call
it good.

Alternatively, perhaps you could make a more general arg_verifier that
could be used for all such needs, and then your traceback would have:

caller

spam

arg_verifier

and that seems useful to me (it is, in fact, how I have mine set up).

Hope this helps!

~Ethan~

Steven D'Aprano 12-15-2010 10:16 AM

Re: How to pop the interpreter's stack?
 
On Tue, 14 Dec 2010 21:14:35 +0000, kj wrote:

> Consider this code:
>
>
> def spam(*args, **kwargs):
> args, kwargs = __pre_spam(*args, **kwargs)
>
> # args & kwargs are OK: proceed
> # ...
>
>
> def __pre_spam(*args, **kwargs):
> # validate args & kwargs;
> # return canonicalized versions of args & kwargs; # on failure,
> raise some *informative* exception # ...
> return canonicalized_args, canonicalized_kwargs


Double leading underscores don't have any special meaning in the global
scope. Save yourself an underscore and call it _pre_spam instead :)

In fact, even if spam and __pre_spam are methods, it's probably a good
idea to avoid the double-underscore name mangling. It's usually more
trouble than it's worth.


> I write functions like __pre_spam for one reason only: to remove clutter
> from a corresponding spam function that has a particularly complex
> argument-validation/canonicalization stage. In effect, spam
> "outsources" to __pre_spam the messy business of checking and
> conditioning its arguments.


A perfectly sensible arrangement.


> The one thing I don't like about this strategy is that the tracebacks of
> exceptions raised during the execution of __pre_spam include one
> unwanted stack level (namely, the one corresponding to __pre_spam
> itself).


But why is it unwanted? The traceback shows where the error occurs -- it
occurs in __pre_spam, not spam, or __post_spam, or spam_caller, or
anywhere else. Even if it's possible, having the traceback *lie* about
where it occurs is a bad idea which will cause confusion to anyone trying
to maintain the software in the future.

I can't think of any way to do it, but frankly I haven't thought too hard
about it. I'm glad I can't think of any way of doing it, because the
thought of having tracebacks lie about where they come from gives me the
shivers. Imagine debugging when you've edited the source but are still
running the old version, and now the reported line numbers don't match up
with the source file -- it would be like that, only worse.



--
Steven

Tim Arnold 12-15-2010 05:38 PM

Re: How to pop the interpreter's stack?
 
"Ethan Furman" <ethan@stoneleaf.us> wrote in message
news:mailman.4.1292379995.6505.python-list@python.org...
> kj wrote:
>> The one thing I don't like about this strategy is that the tracebacks
>> of exceptions raised during the execution of __pre_spam include one
>> unwanted stack level (namely, the one corresponding to __pre_spam
>> itself).
>>
>> __pre_spam should be completely invisible and unobtrusive

>
> I am unaware of any way to accomplish what you desire. I also think this
> is one of those things that's not worth fighting -- how often are you
> going to see such a traceback? When somebody makes a coding mistake? I
> would say change the name (assuming yours was a real example) to something
> more meaningful like _spam_arg_verifier and call it good.
>
> Alternatively, perhaps you could make a more general arg_verifier that
> could be used for all such needs, and then your traceback would have:
>
> caller
>
> spam
>
> arg_verifier
>
> and that seems useful to me (it is, in fact, how I have mine set up).
>
> Hope this helps!
>
> ~Ethan~


I thought people would advise using a decorator for this one. Wouldn't that
work?
thanks,
--Tim



Ethan Furman 12-16-2010 03:29 PM

Re: How to pop the interpreter's stack?
 
Tim Arnold wrote:
> "Ethan Furman" <ethan@stoneleaf.us> wrote in message
> news:mailman.4.1292379995.6505.python-list@python.org...
>> kj wrote:
>>> The one thing I don't like about this strategy is that the tracebacks
>>> of exceptions raised during the execution of __pre_spam include one
>>> unwanted stack level (namely, the one corresponding to __pre_spam
>>> itself).
>>>
>>> __pre_spam should be completely invisible and unobtrusive

>> I am unaware of any way to accomplish what you desire. I also think this
>> is one of those things that's not worth fighting -- how often are you
>> going to see such a traceback? When somebody makes a coding mistake? I
>> would say change the name (assuming yours was a real example) to something
>> more meaningful like _spam_arg_verifier and call it good.
>>
>> Alternatively, perhaps you could make a more general arg_verifier that
>> could be used for all such needs, and then your traceback would have:
>>
>> caller
>>
>> spam
>>
>> arg_verifier
>>
>> and that seems useful to me (it is, in fact, how I have mine set up).
>>
>> Hope this helps!
>>
>> ~Ethan~

>
> I thought people would advise using a decorator for this one. Wouldn't that
> work?
> thanks,
> --Tim


A decorator was one of the items kj explicity didn't want. Also, while
it would have a shallower traceback for exceptions raised during the
__pre_spam portion, any exceptions raised during spam itself would then
be one level deeper than desired... while that could be masked by
catching and (re-?)raising the exception in the decorator, Steven had a
very good point about why that is a bad idea -- namely, tracebacks
shouldn't lie about where the error is.

~Ethan~

Steven D'Aprano 12-16-2010 04:23 PM

Re: How to pop the interpreter's stack?
 
On Thu, 16 Dec 2010 07:29:25 -0800, Ethan Furman wrote:

> Tim Arnold wrote:
>> "Ethan Furman" <ethan@stoneleaf.us> wrote in message
>> news:mailman.4.1292379995.6505.python-list@python.org...
>>> kj wrote:
>>>> The one thing I don't like about this strategy is that the tracebacks
>>>> of exceptions raised during the execution of __pre_spam include one
>>>> unwanted stack level (namely, the one corresponding to __pre_spam
>>>> itself).

[...]
> A decorator was one of the items kj explicity didn't want. Also, while
> it would have a shallower traceback for exceptions raised during the
> __pre_spam portion, any exceptions raised during spam itself would then
> be one level deeper than desired... while that could be masked by
> catching and (re-?)raising the exception in the decorator, Steven had a
> very good point about why that is a bad idea -- namely, tracebacks
> shouldn't lie about where the error is.



True, very true... but many hours later, it suddenly hit me that what KJ
was asking for wasn't *necessarily* such a bad idea. My thought is,
suppose you have a function spam(x) which raises an exception. If it's a
*bug*, then absolutely you need to see exactly where the error occurred,
without the traceback being mangled or changed in any way.

But what if the exception is deliberate, part of the function's
documented behaviour? Then you might want the exception to appear to come
from the function spam even if it was actually generated inside some
private sub-routine.

So, with qualifications, I have half changed my mind.



--
Steven

Robert Kern 12-16-2010 04:39 PM

Re: How to pop the interpreter's stack?
 
On 12/16/10 10:23 AM, Steven D'Aprano wrote:
> On Thu, 16 Dec 2010 07:29:25 -0800, Ethan Furman wrote:
>
>> Tim Arnold wrote:
>>> "Ethan Furman"<ethan@stoneleaf.us> wrote in message
>>> news:mailman.4.1292379995.6505.python-list@python.org...
>>>> kj wrote:
>>>>> The one thing I don't like about this strategy is that the tracebacks
>>>>> of exceptions raised during the execution of __pre_spam include one
>>>>> unwanted stack level (namely, the one corresponding to __pre_spam
>>>>> itself).

> [...]
>> A decorator was one of the items kj explicity didn't want. Also, while
>> it would have a shallower traceback for exceptions raised during the
>> __pre_spam portion, any exceptions raised during spam itself would then
>> be one level deeper than desired... while that could be masked by
>> catching and (re-?)raising the exception in the decorator, Steven had a
>> very good point about why that is a bad idea -- namely, tracebacks
>> shouldn't lie about where the error is.

>
> True, very true... but many hours later, it suddenly hit me that what KJ
> was asking for wasn't *necessarily* such a bad idea. My thought is,
> suppose you have a function spam(x) which raises an exception. If it's a
> *bug*, then absolutely you need to see exactly where the error occurred,
> without the traceback being mangled or changed in any way.
>
> But what if the exception is deliberate, part of the function's
> documented behaviour? Then you might want the exception to appear to come
> from the function spam even if it was actually generated inside some
> private sub-routine.


Obfuscating the location that an exception gets raised prevents a lot of
debugging (by inspection or by pdb), even if the exception is deliberately
raised with an informative error message. Not least, the code that decides to
raise that exception may be buggy. But even if the actual error is outside of
the function (e.g. the caller is passing bad arguments), you want to at least
see what tests the __pre_spam function is doing in order to decide to raise that
exception.

Tracebacks are inherently over-verbose. This is necessarily true because no
algorithm (or clever programmer) can know all the pieces of information that the
person debugging may want to know a priori. Most customizations of tracebacks
*add* more verbosity rather than reduce it. Removing one stack level from the
traceback barely makes the traceback more readable and removes some of the most
relevant information.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco


Steven D'Aprano 12-17-2010 12:33 AM

Re: How to pop the interpreter's stack?
 
On Thu, 16 Dec 2010 10:39:34 -0600, Robert Kern wrote:

> On 12/16/10 10:23 AM, Steven D'Aprano wrote:
>> On Thu, 16 Dec 2010 07:29:25 -0800, Ethan Furman wrote:
>>
>>> Tim Arnold wrote:
>>>> "Ethan Furman"<ethan@stoneleaf.us> wrote in message
>>>> news:mailman.4.1292379995.6505.python-list@python.org...
>>>>> kj wrote:
>>>>>> The one thing I don't like about this strategy is that the
>>>>>> tracebacks of exceptions raised during the execution of __pre_spam
>>>>>> include one unwanted stack level (namely, the one corresponding to
>>>>>> __pre_spam itself).

>> [...]
>>> A decorator was one of the items kj explicity didn't want. Also,
>>> while it would have a shallower traceback for exceptions raised during
>>> the __pre_spam portion, any exceptions raised during spam itself would
>>> then be one level deeper than desired... while that could be masked by
>>> catching and (re-?)raising the exception in the decorator, Steven had
>>> a very good point about why that is a bad idea -- namely, tracebacks
>>> shouldn't lie about where the error is.

>>
>> True, very true... but many hours later, it suddenly hit me that what
>> KJ was asking for wasn't *necessarily* such a bad idea. My thought is,
>> suppose you have a function spam(x) which raises an exception. If it's
>> a *bug*, then absolutely you need to see exactly where the error
>> occurred, without the traceback being mangled or changed in any way.
>>
>> But what if the exception is deliberate, part of the function's
>> documented behaviour? Then you might want the exception to appear to
>> come from the function spam even if it was actually generated inside
>> some private sub-routine.

>
> Obfuscating the location that an exception gets raised prevents a lot of
> debugging (by inspection or by pdb), even if the exception is
> deliberately raised with an informative error message. Not least, the
> code that decides to raise that exception may be buggy. But even if the
> actual error is outside of the function (e.g. the caller is passing bad
> arguments), you want to at least see what tests the __pre_spam function
> is doing in order to decide to raise that exception.


And how do you think you see that from the traceback? The traceback
prints the line which actually raises the exception (and sometimes not
even that!), which is likely to be a raise statement:

>>> import example
>>> example.func(42)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "example.py", line 3, in func
raise ValueError('bad value for x')
ValueError: bad value for x

The actual test is:

def func(x):
if x > 10 and x%2 == 0:
raise ValueError('bad value for x')

but you can't get that information from the traceback.

Python's exception system has to handle two different situations: buggy
code, and bad data. It's not even clear whether there is a general
distinction to be made between the two, but even if there's not a general
distinction, there's certainly a distinction which we can *sometimes*
make. If a function contains a bug, we need all the information we can
get, including the exact line that causes the fault. But if the function
deliberately raises an exception due to bad input, we don't need any
information regarding the internals of the function (assuming that the
exception is sufficiently detailed, a big assumption I grant you!). If I
re-wrote the above func() like this:

def func(x):
if !(x <= 10):
if x%2 != 0:
pass
else:
raise ValueError('bad value for x')
return

I would have got the same traceback, except the location of the exception
would have been different (line 6, in a nested if-block). To the caller,
whether I had written the first version of func() or the second is
irrelevant. If I had passed the input validation off to a second
function, that too would be irrelevant.

I don't expect Python to magically know whether an exception is a bug or
not, but there's something to be said for the ability to turn Python
functions into black boxes with their internals invisible, like C
functions already are. If (say) math.atan2(y, x) raises an exception, you
have no way of knowing whether atan2 is a single monolithic function, or
whether it is split into multiple pieces. The location of the exception
is invisible to the caller: all you can see is that atan2 raised an
exception.


> Tracebacks are inherently over-verbose. This is necessarily true because
> no algorithm (or clever programmer) can know all the pieces of
> information that the person debugging may want to know a priori. Most
> customizations of tracebacks *add* more verbosity rather than reduce it.
> Removing one stack level from the traceback barely makes the traceback
> more readable and removes some of the most relevant information.


Right. But I have thought of a clever trick to get the result KJ was
asking for, with the minimum of boilerplate code. Instead of this:


def _pre_spam(args):
if condition(args):
raise SomeException("message")
if another_condition(args):
raise AnotherException("message")
if third_condition(args):
raise ThirdException("message")

def spam(args):
_pre_spam(args)
do_useful_work()


you can return the exceptions instead of raising them (exceptions are
just objects, like everything else!), and then add one small piece of
boilerplate to the spam() function:


def _pre_spam(args):
if condition(args):
return SomeException("message")
if another_condition(args):
return AnotherException("message")
if third_condition(args):
return ThirdException("message")

def spam(args):
exc = _pre_spam(args)
if exc: raise exc
do_useful_work()




--
Steven

Carl Banks 12-17-2010 05:25 AM

Re: How to pop the interpreter's stack?
 
On Dec 15, 2:16*am, Steven D'Aprano <steve
+comp.lang.pyt...@pearwood.info> wrote:
> On Tue, 14 Dec 2010 21:14:35 +0000, kj wrote:
> > Consider this code:

>
> > def spam(*args, **kwargs):
> > * * args, kwargs = __pre_spam(*args, **kwargs)

>
> > * * # args & kwargs are OK: proceed
> > * * # ...

>
> > def __pre_spam(*args, **kwargs):
> > * * # validate args & kwargs;
> > * * # return canonicalized versions of args & kwargs; # on failure,
> > * * raise some *informative* exception # ...
> > * * return canonicalized_args, canonicalized_kwargs

>
> Double leading underscores don't have any special meaning in the global
> scope. Save yourself an underscore and call it _pre_spam instead :)
>
> In fact, even if spam and __pre_spam are methods, it's probably a good
> idea to avoid the double-underscore name mangling. It's usually more
> trouble than it's worth.
>
> > I write functions like __pre_spam for one reason only: to remove clutter
> > from a corresponding spam function that has a particularly complex
> > argument-validation/canonicalization stage. *In effect, spam
> > "outsources" to __pre_spam the messy business of checking and
> > conditioning its arguments.

>
> A perfectly sensible arrangement.
>
> > The one thing I don't like about this strategy is that the tracebacks of
> > exceptions raised during the execution of __pre_spam include one
> > unwanted stack level (namely, the one corresponding to __pre_spam
> > itself).

>
> But why is it unwanted? The traceback shows where the error occurs -- it
> occurs in __pre_spam, not spam, or __post_spam, or spam_caller, or
> anywhere else. Even if it's possible, having the traceback *lie* about
> where it occurs is a bad idea which will cause confusion to anyone trying
> to maintain the software in the future.


I don't agree with kj's usage, but I have customized the traceback to
remove items before. In my case it was to remove lines for endemic
wrapper functions.

The traceback lines showing the wrapper functions in the stack were
useless, and since pretty much every function was wrapped it meant
half the lines in that traceback were useless. (Really. I was scanning
the loaded modules and adding wrappers to every function found. Never
mind why.) I only printed the wrapper line if it was the very top of
the stack.


> I can't think of any way to do it,


You override sys.excepthook to print lines from the traceback
selectively.


Carl Banks

John Nagle 12-17-2010 07:07 AM

Re: How to pop the interpreter's stack?
 
On 12/14/2010 6:31 PM, Ethan Furman wrote:
> kj wrote:
>> The one thing I don't like about this strategy is that the tracebacks
>> of exceptions raised during the execution of __pre_spam include one
>> unwanted stack level (namely, the one corresponding to __pre_spam
>> itself).
>>
>> __pre_spam should be completely invisible and unobtrusive

>
> I am unaware of any way to accomplish what you desire. I also think this
> is one of those things that's not worth fighting -- how often are you
> going to see such a traceback? When somebody makes a coding mistake?


Right. If you are worried about what the user sees in a traceback,
you are doing it wrong.

Consider reporting detailed error information via the logging
module, for example.

John Nagle


All times are GMT. The time now is 11:18 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.