Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > How to know that two pyc files contain the same code

Reply
Thread Tools

How to know that two pyc files contain the same code

 
 
Gelonida N
Guest
Posts: n/a
 
      03-10-2012
Hi,

I want to know whether two .pyc files are identical.

With identical I mean whether they contain the same byte code.

Unfortunately it seems, that .pyc files contain also something like the
time stamp of the related source file.

So though two pyc files contain the same byte code, they will not be
byte identical.

One option, that I found is to use
python -m unpyclib.application -d filename.pyc and check whether the
results are identical.


However even this will fail if the files were not compiled under the
same absolute path name as the source filename is contained twice (at
least for my trivial example) in the disassemblers output.


Thanks a lot for any other idea.



 
Reply With Quote
 
 
 
 
Steven D'Aprano
Guest
Posts: n/a
 
      03-10-2012
On Sat, 10 Mar 2012 15:48:48 +0100, Gelonida N wrote:

> Hi,
>
> I want to know whether two .pyc files are identical.
>
> With identical I mean whether they contain the same byte code.


Define "identical" and "the same".

If I compile these two files:


# file ham.py
x = 23
def func():
a = 23
return a + 19



# file = spam.py
def func():
return 42

tmp = 19
x = 4 + tmp
del tmp


do you expect spam.pyc and ham.pyc to count as "the same"?


--
Steven
 
Reply With Quote
 
 
 
 
Chris Angelico
Guest
Posts: n/a
 
      03-11-2012
On Sun, Mar 11, 2012 at 9:52 AM, Steven D'Aprano
<(E-Mail Removed)> wrote:
> On Sat, 10 Mar 2012 15:48:48 +0100, Gelonida N wrote:
> Define "identical" and "the same".
>
> If I compile these two files:
>
>
> # file ham.py
> x = 23
> def func():
> * *a = 23
> * *return a + 19
>
>
>
> # file = spam.py
> def func():
> * *return 42
>
> tmp = 19
> x = 4 + tmp
> del tmp
>
>
> do you expect spam.pyc and ham.pyc to count as "the same"?


They do not contain the same code. They may contain code which has the
same effect, but it is not the same code.

I don't think Python has the level of aggressive optimization that
would make these compile to the same bytecode, but if it did, then
they would _become identical_ per the OP's description - that they
contain identical bytecode. In fact, I think the OP defined it quite
clearly.

ChrisA
 
Reply With Quote
 
Gelonida N
Guest
Posts: n/a
 
      03-11-2012
Hi Steven,

On 03/10/2012 11:52 PM, Steven D'Aprano wrote:
> > On Sat, 10 Mar 2012 15:48:48 +0100, Gelonida N wrote:
> >
>> >> Hi,
>> >>
>> >> I want to know whether two .pyc files are identical.
>> >>
>> >> With identical I mean whether they contain the same byte code.

> >
> > Define "identical" and "the same".

Indeed! Identical is not that simple to define and depends on the context.

One definition of identical, that would suit me at the moment would be:

If I have two .pyc files, which were the result of a compilation of
two identical .py files, then I would like to treat these two .pyc files
as identical,
even if they were compiled at different times (absolutely necessary)
and with a different absolute path (would be nice)

Above definition of identical byte code would also mean, that any error
message about errors in a given line number would be identical for both
..pyc files

> >
> > If I compile these two files:
> >
> >
> > # file ham.py
> > x = 23
> > def func():
> > a = 23
> > return a + 19
> >
> >
> >
> > # file = spam.py
> > def func():
> > return 42
> >
> > tmp = 19
> > x = 4 + tmp
> > del tmp
> >
> >
> > do you expect spam.pyc and ham.pyc to count as "the same"?
> >

For most pythons I would not expect, that ham.py and spam.py would
result in the same byte code and would thus not even have the same
performance,

I agree, though that an efficient compiler might generate the same byte
code, though I wonder if an optimizing compiler would/should be allowed
to optimize away the global variable tmp, as it would be visible (though
only for a short time) in a multithreading environment.

If the byte code were different in two .pyc files. then I would
like to have them treated as different .pyc files.

If by coincidence, the generated btye code were the same, then I
wouldn't mind, if they were treated as identical, but I wouldn't insist.

Up to my knowledge Python (or at least C-python) stores line numbers in
the .pyc files, so that it can report exact line numbers refering to the
originating source code in case of an exception or for back traces

So there is the choice to say, that two pyc files with exactly the same
byte code would be treated identical if white spaces / line numbers of
their sources were different or the choice to say, that they are
different.

Being conservative I'd treat them as different.

Ideally I'd like to be able depending on my use case to distinguish
following cases.
a) .pyc files with identical byte code
b) .pyc files with identical byte code AND source code line numbers
c) same as b) AND identical source file names.






 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      03-11-2012
On Sun, 11 Mar 2012 12:15:11 +1100, Chris Angelico wrote:

> On Sun, Mar 11, 2012 at 9:52 AM, Steven D'Aprano
> <(E-Mail Removed)> wrote:
>> On Sat, 10 Mar 2012 15:48:48 +0100, Gelonida N wrote: Define
>> "identical" and "the same".
>>
>> If I compile these two files:
>>
>>
>> # file ham.py
>> x = 23
>> def func():
>> * *a = 23
>> * *return a + 19
>>
>>
>>
>> # file = spam.py
>> def func():
>> * *return 42
>>
>> tmp = 19
>> x = 4 + tmp
>> del tmp
>>
>>
>> do you expect spam.pyc and ham.pyc to count as "the same"?

>
> They do not contain the same code. They may contain code which has the
> same effect, but it is not the same code.


To me, they do: they contain a function "func" which takes no arguments
and returns 42, and a global "x" initialised to 23. Everything else is an
implementation detail.

I'm not being facetious. One should be asking what is the *purpose* of
this question -- is it to detect when two pyc files contain the same
*interface*, or to determine if they were generated from identical source
code files (and if the later, do comments and whitespace matter)?

What if one merely changed the order of definition? Instead of:

def foo(): pass
def bar(): pass

one had this?

def bar(): pass
def foo(): pass

It depends on why the OP cares if they are "identical". I can imagine use-
cases where the right solution is to forget ideas about identical code,
and just checksum the files (ignoring any timestamps).


--
Steven
 
Reply With Quote
 
Rick Johnson
Guest
Posts: n/a
 
      03-11-2012
On Mar 11, 2:06*am, Steven D'Aprano <steve
(E-Mail Removed)> wrote:

> I'm not being facetious. [...]


Just in case anybody does not know already: the "D'A" in "Steven
D'Aprano" stands for "Devils Advocate"; his real name is "Steven
Prano".
 
Reply With Quote
 
Gelonida N
Guest
Posts: n/a
 
      03-11-2012
On 03/11/2012 08:06 AM, Steven D'Aprano wrote:
> What if one merely changed the order of definition? Instead of:
>
> def foo(): pass
> def bar(): pass
>
> one had this?
>
> def bar(): pass
> def foo(): pass
>
> It depends on why the OP cares if they are "identical". I can imagine use-
> cases where the right solution is to forget ideas about identical code,
> and just checksum the files (ignoring any timestamps).


I guess this is what I will do for my use case
Perform a checksum ignoring the time stamp.

What I did not know though is where the time stamp was located.
it seems it's in bytes 4-7 for all C-python versions so far.

What is regrettable though is, that the absolute path name is part of
the .pyc file, as I do not care


Following snippet calculates the hash of a .pyc file by just ignoring
the time stamp:

import hashlib

def md5_for_pyc(fname):
hasher = hashlib.md5()
with open(fname, 'rb') as fin:
version = fin.read(4)
hasher.update(version)
_tstamp = fin.read(4)
bytecode = fin.read()
hasher.update(bytecode)
return hasher.hexdigest()





 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
problem in running a basic code in python 3.3.0 that includes HTML file Satabdi Mukherjee Python 1 04-04-2013 07:48 PM
How to know that two pyc files contain the same code Gelonida N Python 1 03-10-2012 03:08 PM
RCR: String#contain? and Array#contain? Roger Pack Ruby 3 09-28-2010 04:13 PM
find out whther byte two .pyc files contain the same byte code. gelonida Python 1 05-05-2010 11:04 PM



Advertisments