Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Yet another attempt at a safe eval() call

Reply
Thread Tools

Yet another attempt at a safe eval() call

 
 
Chris Angelico
Guest
Posts: n/a
 
      01-05-2013
On Sun, Jan 6, 2013 at 2:56 AM, Oscar Benjamin
<(E-Mail Removed)> wrote:
> On 4 January 2013 15:53, Grant Edwards <(E-Mail Removed)> wrote:
>> On 2013-01-04, Steven D'Aprano <(E-Mail Removed)> wrote:
>>> On Thu, 03 Jan 2013 23:25:51 +0000, Grant Edwards wrote:
>>>
>>> * But frankly, you should avoid eval, and write your own mini-integer
>>> arithmetic evaluator which avoids even the most remote possibility
>>> of exploit.

>>
>> That's obviously the "right" thing to do. I suppose I should figure
>> out how to use the ast module.

>
> Someone has already created a module that does this called numexpr. Is
> there some reason why you don't want to use that?
>
>>>> import numexpr
>>>> numexpr.evaluate('2+4*5')

> array(22, dtype=int32)
>>>> numexpr.evaluate('2+a*5', {'a':4})

> array(22L)


Is that from PyPI? It's not in my Python 3.3 installation. Obvious
reason not to use it: Unaware of it.

ChrisA
 
Reply With Quote
 
 
 
 
Oscar Benjamin
Guest
Posts: n/a
 
      01-05-2013
On 5 January 2013 16:01, Chris Angelico <(E-Mail Removed)> wrote:
> On Sun, Jan 6, 2013 at 2:56 AM, Oscar Benjamin
> <(E-Mail Removed)> wrote:
>> On 4 January 2013 15:53, Grant Edwards <(E-Mail Removed)> wrote:
>>> On 2013-01-04, Steven D'Aprano <(E-Mail Removed)> wrote:
>>>> On Thu, 03 Jan 2013 23:25:51 +0000, Grant Edwards wrote:
>>>>
>>>> * But frankly, you should avoid eval, and write your own mini-integer
>>>> arithmetic evaluator which avoids even the most remote possibility
>>>> of exploit.
>>>
>>> That's obviously the "right" thing to do. I suppose I should figure
>>> out how to use the ast module.

>>
>> Someone has already created a module that does this called numexpr. Is
>> there some reason why you don't want to use that?
>>
>>>>> import numexpr
>>>>> numexpr.evaluate('2+4*5')

>> array(22, dtype=int32)
>>>>> numexpr.evaluate('2+a*5', {'a':4})

>> array(22L)

>
> Is that from PyPI? It's not in my Python 3.3 installation. Obvious
> reason not to use it: Unaware of it.


My apologies. I should have at least provided a link:
http://code.google.com/p/numexpr/

I installed it from the ubuntu repo under the name python-numexpr. It
is also on PyPI:
http://pypi.python.org/pypi/numexpr

numexpr is a well established project intended primarily for memory
and cache efficient computations over large arrays of data. Possibly
as a side effect, it can also be used to evaluate simple algebraic
expressions involving ordinary scalar variables.


Oscar
 
Reply With Quote
 
 
 
 
matt.newville@gmail.com
Guest
Posts: n/a
 
      01-05-2013
On Saturday, January 5, 2013 8:17:16 AM UTC-8, Oscar Benjamin wrote:
> On 5 January 2013 16:01, Chris Angelico <(E-Mail Removed)> wrote:
>
> > On Sun, Jan 6, 2013 at 2:56 AM, Oscar Benjamin

>
> > <(E-Mail Removed)> wrote:

>
> >> On 4 January 2013 15:53, Grant Edwards <(E-Mail Removed)> wrote:

>
> >>> On 2013-01-04, Steven D'Aprano <(E-Mail Removed)>wrote:

>
> >>>> On Thu, 03 Jan 2013 23:25:51 +0000, Grant Edwards wrote:

>
> >>>>

>
> >>>> * But frankly, you should avoid eval, and write your own mini-integer

>
> >>>> arithmetic evaluator which avoids even the most remote possibility

>
> >>>> of exploit.

>
> >>>

>
> >>> That's obviously the "right" thing to do. I suppose I should figure

>
> >>> out how to use the ast module.

>
> >>

>
> >> Someone has already created a module that does this called numexpr. Is

>
> >> there some reason why you don't want to use that?

>
> >>

>
> >>>>> import numexpr

>
> >>>>> numexpr.evaluate('2+4*5')

>
> >> array(22, dtype=int32)

>
> >>>>> numexpr.evaluate('2+a*5', {'a':4})

>
> >> array(22L)

>
> >

>
> > Is that from PyPI? It's not in my Python 3.3 installation. Obvious

>
> > reason not to use it: Unaware of it.

>
>
>
> My apologies. I should have at least provided a link:
>
> http://code.google.com/p/numexpr/
>
>
>
> I installed it from the ubuntu repo under the name python-numexpr. It
>
> is also on PyPI:
>
> http://pypi.python.org/pypi/numexpr
>
>
>
> numexpr is a well established project intended primarily for memory
>
> and cache efficient computations over large arrays of data. Possibly
>
> as a side effect, it can also be used to evaluate simple algebraic
>
> expressions involving ordinary scalar variables.
>
>
>
>
>
> Oscar


The asteval module http://pypi.python.org/pypi/asteval/0.9 and
http://newville.github.com/asteval/ might be another alternative. It's not as fast as numexpr, but a bit more general. It uses the ast module to "compile" an expression into the AST, then walks through that, intercepting Name nodes and using a flat namespace of variables. It disallows imports anddoes not support all python constructs, but it is a fairly complete in supporting python syntax.

It makes no claim at actually being safe from malicious attack, but should be safer than a straight eval(), and prevent accidental problems when evaluating user-input as code. If anyone can find exploits within it, I'd be happy to try to fix them.

--Matt
 
Reply With Quote
 
matt.newville@gmail.com
Guest
Posts: n/a
 
      01-05-2013
On Saturday, January 5, 2013 8:17:16 AM UTC-8, Oscar Benjamin wrote:
> On 5 January 2013 16:01, Chris Angelico <(E-Mail Removed)> wrote:
>
> > On Sun, Jan 6, 2013 at 2:56 AM, Oscar Benjamin

>
> > <(E-Mail Removed)> wrote:

>
> >> On 4 January 2013 15:53, Grant Edwards <(E-Mail Removed)> wrote:

>
> >>> On 2013-01-04, Steven D'Aprano <(E-Mail Removed)>wrote:

>
> >>>> On Thu, 03 Jan 2013 23:25:51 +0000, Grant Edwards wrote:

>
> >>>>

>
> >>>> * But frankly, you should avoid eval, and write your own mini-integer

>
> >>>> arithmetic evaluator which avoids even the most remote possibility

>
> >>>> of exploit.

>
> >>>

>
> >>> That's obviously the "right" thing to do. I suppose I should figure

>
> >>> out how to use the ast module.

>
> >>

>
> >> Someone has already created a module that does this called numexpr. Is

>
> >> there some reason why you don't want to use that?

>
> >>

>
> >>>>> import numexpr

>
> >>>>> numexpr.evaluate('2+4*5')

>
> >> array(22, dtype=int32)

>
> >>>>> numexpr.evaluate('2+a*5', {'a':4})

>
> >> array(22L)

>
> >

>
> > Is that from PyPI? It's not in my Python 3.3 installation. Obvious

>
> > reason not to use it: Unaware of it.

>
>
>
> My apologies. I should have at least provided a link:
>
> http://code.google.com/p/numexpr/
>
>
>
> I installed it from the ubuntu repo under the name python-numexpr. It
>
> is also on PyPI:
>
> http://pypi.python.org/pypi/numexpr
>
>
>
> numexpr is a well established project intended primarily for memory
>
> and cache efficient computations over large arrays of data. Possibly
>
> as a side effect, it can also be used to evaluate simple algebraic
>
> expressions involving ordinary scalar variables.
>
>
>
>
>
> Oscar


The asteval module http://pypi.python.org/pypi/asteval/0.9 and
http://newville.github.com/asteval/ might be another alternative. It's not as fast as numexpr, but a bit more general. It uses the ast module to "compile" an expression into the AST, then walks through that, intercepting Name nodes and using a flat namespace of variables. It disallows imports anddoes not support all python constructs, but it is a fairly complete in supporting python syntax.

It makes no claim at actually being safe from malicious attack, but should be safer than a straight eval(), and prevent accidental problems when evaluating user-input as code. If anyone can find exploits within it, I'd be happy to try to fix them.

--Matt
 
Reply With Quote
 
Grant Edwards
Guest
Posts: n/a
 
      01-06-2013
On 2013-01-05, Oscar Benjamin <(E-Mail Removed)> wrote:
> On 4 January 2013 15:53, Grant Edwards <(E-Mail Removed)> wrote:
>> On 2013-01-04, Steven D'Aprano <(E-Mail Removed)> wrote:
>>> On Thu, 03 Jan 2013 23:25:51 +0000, Grant Edwards wrote:
>>>
>>> * But frankly, you should avoid eval, and write your own mini-integer
>>> arithmetic evaluator which avoids even the most remote possibility
>>> of exploit.

>>
>> That's obviously the "right" thing to do. I suppose I should figure
>> out how to use the ast module.

>
> Someone has already created a module that does this called numexpr. Is
> there some reason why you don't want to use that?


1) I didn't know about it, and my Googling didn't find it.

2) It's not part of the standard library, and my program needs to be
distributed as a single source file.

--
Grant
 
Reply With Quote
 
Oscar Benjamin
Guest
Posts: n/a
 
      01-07-2013
On 6 January 2013 15:12, Grant Edwards <(E-Mail Removed)> wrote:
> On 2013-01-05, Oscar Benjamin <(E-Mail Removed)> wrote:
>> On 4 January 2013 15:53, Grant Edwards <(E-Mail Removed)> wrote:
>>> On 2013-01-04, Steven D'Aprano <(E-Mail Removed)> wrote:
>>>> On Thu, 03 Jan 2013 23:25:51 +0000, Grant Edwards wrote:
>>>>
>>>> * But frankly, you should avoid eval, and write your own mini-integer
>>>> arithmetic evaluator which avoids even the most remote possibility
>>>> of exploit.
>>>
>>> That's obviously the "right" thing to do. I suppose I should figure
>>> out how to use the ast module.

>>
>> Someone has already created a module that does this called numexpr. Is
>> there some reason why you don't want to use that?

>
> 1) I didn't know about it, and my Googling didn't find it.
>
> 2) It's not part of the standard library, and my program needs to be
> distributed as a single source file.


That's an unfortunate restriction. It also won't be possible to reuse
the code from numexpr (for technical rather than legal reasons).
Perhaps asteval will be more helpful in that sense.

Otherwise presumably the shunting-yard algorithm comes out a little
nicer in Python than in C (it would be useful if something like this
were available on PyPI as a pure Python module):
http://en.wikipedia.org/wiki/Shuntin...ithm#C_example


Oscar
 
Reply With Quote
 
Ken Seehart
Guest
Posts: n/a
 
      03-15-2013
On 1/4/2013 5:33 AM, Steven D'Aprano wrote:
> On Fri, 04 Jan 2013 07:24:04 -0500, Terry Reedy wrote:
>
>> On 1/3/2013 6:25 PM, Grant Edwards wrote:
>>> I've written a small assembler in Python 2.[67], and it needs to
>>> evaluate integer-valued arithmetic expressions in the context of a
>>> symbol table that defines integer values for a set of names. The
>>> "right" thing is probably an expression parser/evaluator using ast, but
>>> it looked like that would take more code that the rest of the assembler
>>> combined, and I've got other higher-priority tasks to get back to.

>> Will ast.literal_eval do what you want?

> No. Grant needs to support variables, not just literal constants, hence
> the symbol table.
>
>

Apologies for the delayed response...

Seems like it would be a bit safer and easier to approach this problem
by stretching the capability of ast.literal_eval() rather than
attempting to sandbox eval().

How about ast.literal_eval after performing lexical substitution using
the symbol table?

Assignment into the symbol table, and error handling, are exercises left
to the reader.

Something vaguely like this:

/pseudocode:/

def safe_eval(s, symbols={}):
while search(s, r'\w+'):
replace match with '('+repr(symbols[match])+')' in s
return ast.literal_eval(s)

- Ken


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: yet another feeb attempt to escape quotes in XPaths Bjoern Hoehrmann XML 0 04-13-2008 07:23 PM
Can't locate object method "first" via package "attempt" (perhaps you forgot to load "attempt"?) at .... GMI Perl Misc 3 06-19-2005 10:44 PM
Yet another book recommendation, but for someone who can program and yet does not the terminology well Berehem C Programming 4 04-28-2005 05:25 PM
Re: Yet another Attempt at Disproving the Halting Problem Peter Olcott C++ 245 08-21-2004 04:48 PM
Double checked locking (yet another attempt to get around) Ed Java 9 07-18-2003 03:59 PM



Advertisments