«

So I kind of wanted to ask this question on the pypy mailing list..
but there's only a pypy-dev list, and I don't want to put noise on the
dev list.

What's the point of RPython? By this, I don't mean "What is RPython"?
I get that. I mean, why?

The goals of the pypy project seems to be to create a fast python
implementation. I may be wrong about this, as the goals seem a little
amorphous if you look at their home page.

So, to do that this RPython compiler was created? Except that it
doesn't compile python, it compiles a static subset of python that has
type inference like ML.

This RPython thing seems like a totally unnecessary intermediate step.
Instead of writing an implementation of one language, there's an
implementation of two languages.

Actually, it seems like it harms the ultimate goal. For the
interpreted pyton to be fast, the interpreter has to be written in a
reasonably fast language. ML, and C++ compilers have had a lot of work
put into their optimization steps. A lot of the performance boost you
get from a static language is that knowing the types at compile time
lets you inline code like crazy, unroll loops, and even execute code
at compile time.

RPython is a statically typed language because I guess the developers
associate static languages with speed? Except that they use it to
generate C code, which throws away the type information they need to
get the speed increase. Huh? I thought the goal was to write a fast
dynamic language, not a slow static one?

Is this going anywhere or is this just architecture astronautics?

The RPython project seems kind of interseting to me and I'd like to
see more python implementations, but looking at the project I can't
help but think that they haven't really explained *why* they are doing
the things they are doing.

Anyway, I can tell this is the sort of question that some people will
interpret as rude. Asking hard questions is never polite, but it is
always necessary

Thanks,
Brendan

On Jan 16, 9:37*pm, "Brendan Miller" <catph...@catphive.net> wrote:
> So I kind of wanted to ask this question on the pypy mailing list..
> but there's only a pypy-dev list, and I don't want to put noise on the
> dev list.
>
> What's the point of RPython? By this, I don't mean "What is RPython"?
> I get that. I mean, why?
>
> The goals of the pypy project seems to be to create a fast python
> implementation. I may be wrong about this, as the goals seem a little
> amorphous if you look at their home page.
>
> So, to do that this RPython compiler was created? Except that it
> doesn't compile python, it compiles a static subset of python that has
> type inference like ML.
>
> This RPython thing seems like a totally unnecessary intermediate step.
> Instead of writing an implementation of one language, there's an
> implementation of two languages.
>
> Actually, it seems like it harms the ultimate goal. For the
> interpreted pyton to be fast, the interpreter has to be written in a
> reasonably fast language. ML, and C++ compilers have had a lot of work
> put into their optimization steps. A lot of the performance boost you
> get from a static language is that knowing the types at compile time
> lets you inline code like crazy, unroll loops, and even execute code
> at compile time.
>
> RPython is a statically typed language because I guess the developers
> associate static languages with speed? Except that they use it to
> generate C code, which throws away the type information they need to
> get the speed increase. Huh? I thought the goal was to write a fast
> dynamic language, not a slow static one?
>
> Is this going anywhere or is this just architecture astronautics?
>
> The RPython project seems kind of interseting to me and I'd like to
> see more python implementations, but looking at the project I can't
> help but think that they haven't really explained *why* they are doing
> the things they are doing.
>
> Anyway, I can tell this is the sort of question that some people will
> interpret as rude. Asking hard questions is never polite, but it is
> always necessary
>
> Thanks,
> Brendan

Check out this thread:
http://groups.google.com/group/comp....74f796cd687931

One of pypy's goals is having a more flexible implementation, easier
to experiment with than cpython.
Having all written in the same language facilitates this task.
Why Rpython? Because by avoiding the dinamicity of the whole python
language, it is possible to translate this code to a static one (c,
java, whatever).
The translated rpython interpreter aims to be something similar in
performance to cpython. The only difference is that cpython was
written from scratch in c, while pypy's interpreter was written in
rpython and then translated automatically to c.

The pypy team intends to achieve greater speed and performance by
adding a JIT compiler to this interpreter.
So pypy-c + JIT = faster python.

I hope this helps...

Luis

On Jan 17, 10:37*am, "Brendan Miller" <catph...@catphive.net> wrote:
> What's the point of RPython? By this, I don't mean "What is RPython"?
> I get that. I mean, why?

This is more or less covered in the FAQ:

"The restrictions are to ensure that type inference (and so,
ultimately, translation to other languages) of RPython programs is
possible. These restrictions only apply after the full import happens,
so at import time arbitrary Python code can be executed."

http://codespeak.net/pypy/dist/pypy/doc/faq.html#id25

> The goals of the pypy project seems to be to create a fast python
> implementation. I may be wrong about this, as the goals seem a little
> amorphous if you look at their home page.

The home page itself is ambiguous, and does oversell the performance
aspect. The *actual* goal as outlined by their official docs is to
implement Python in Python, at every level. If this means utilising a
less-dynamic subset of Python for the lower levels, its -still- at
least more-Python-than-C.

"PyPy is both:
* a reimplementation of Python in Python, and
* a framework for implementing interpreters and virtual machines for
programming languages, especially dynamic languages."

http://codespeak.net/pypy/dist/pypy/doc/faq.html#id11

The PyPy devs feel that this will allow them to more easily experiment
with optimising the interpreter for greater speeds, but that isn't one
of the stated goals (just, apparently, their 'secret' one).

> This RPython thing seems like a totally unnecessary intermediate step.
> Instead of writing an implementation of one language, there's an
> implementation of two languages.

My understanding is that the 'higher' level Python language features
are implemented on top of the restricted RPython features, so it's
more of an organic growth of one language than two separate
implementations.

> A lot of the performance boost you
> get from a static language is that knowing the types at compile time
> lets you inline code like crazy, unroll loops, and even execute code
> at compile time.
>
> RPython is a statically typed language because I guess the developers
> associate static languages with speed?

Doesn't the first paragraph above state the same thing that you
question in the next?

> Except that they use it to
> generate C code, which throws away the type information they need to
> get the speed increase. Huh? I thought the goal was to write a fast
> dynamic language, not a slow static one?

They're not just generating C code, they currently also target LLVM,
JVM and CLI.

> Is this going anywhere or is this just architecture astronautics?

Well, they actually seem to have achieved some substantial gains, so I
think it's unfair to imply that the project isn't based on pragmatic
objectives. Their initial JIT prototype failed to work as well as
expected, and they've spent some time re-thinking the approach; I'm
happier to wait for a stably performing JIT that can be improved over
time than a short term gain.

> The RPython project seems kind of interseting to me and I'd like to
> see more python implementations, but looking at the project I can't
> help but think that they haven't really explained *why* they are doing
> the things they are doing.

A lot of this is covered in the FAQ. Whether you agree with their
approach or not, they're the ones actively pushing this effort
forward.

It's been a few months since the last update, but the PyPy status blog
may have more information for you. At the very least, it's a venue to
discuss your concerns directly with the PyPy devs.

http://morepypy.blogspot.com

>> The goals of the pypy project seems to be to create a fast python
>> implementation. I may be wrong about this, as the goals seem a little
>> amorphous if you look at their home page.
>
> The home page itself is ambiguous, and does oversell the performance
> aspect. The *actual* goal as outlined by their official docs is to
> implement Python in Python, at every level.

Ok fair enough. In some ways I see that as more of a purely
intellectual exercise than the practical endeavor that I assumed the
project was originally.

However, one of the links I was sent had one of the devs talking about
using the translation process to make C/Java/LLVM implementations out
of the same interpreter code. I'll say that makes a lot of sense.

Another question I was wondering about is whether they plan on
maintaining good C bindings? Will existing bindings for third party
libraries be able to work?

Also, are they going to do away with the GIL? The python devs seem to
consider the GIL a non-issue, though they may change their mind in 3
years when we all have 32 core desktops, until then getting rid of the
GIL would make pypy pretty attractive in some quarters. I know the
scons project was running into GIL issues.

Finally, I'm pretty unclear on what versions of python that pypy is targeting.

On Jan 18, 9:55*am, "Brendan Miller" <catph...@catphive.net> wrote:
> > The *actual* goal as outlined by their official docs is to
> > implement Python in Python, at every level.
> Ok fair enough. In some ways I see that as more of a purely
> intellectual exercise than the practical endeavor that I assumed the
> project was originally.

Well, it opens up Python as a whole for experimentation & modification
by Python developers, which seems pretty practical to me

> Another question I was wondering about is whether they plan on
> maintaining good C bindings? Will existing bindings for third party
> libraries be able to work?

You really should run through the whole faq:

http://codespeak.net/pypy/dist/pypy/...ension-modules

(Spoiler alert: the answer is no.)

> Also, are they going to do away with the GIL?

One FAQ entry before the above:

http://codespeak.net/pypy/dist/pypy/...ules-that-work

> The python devs seem to
> consider the GIL a non-issue, though they may change their mind in 3
> years when we all have 32 core desktops, until then getting rid of the
> GIL would make pypy pretty attractive in some quarters. I know the
> scons project was running into GIL issues.

It's not a case of the Python devs digging their heels in and not
giving the world what it desperately wants. Here's an article by Guido
talking about the last attempt to remove the GIL and the performance
issues that arose:

"I'd welcome a set of patches into Py3k *only if* the performance for
a single-threaded program (and for a multi-threaded but I/O-bound
program) *does not decrease*."

http://www.artima.com/weblogs/viewpo...?thread=214235

> Finally, I'm pretty unclear on what versions of python that pypy is targeting.

I wonder where such a frequently asked question could be answered?

http://codespeak.net/pypy/dist/pypy/...pypy-implement

alex23 <> writes:
> Here's an article by Guido talking about the last attempt to remove
> the GIL and the performance issues that arose:
>
> "I'd welcome a set of patches into Py3k *only if* the performance for
> a single-threaded program (and for a multi-threaded but I/O-bound
> program) *does not decrease*."

The performance decrease is an artifact of CPython's rather primitive
storage management (reference counts in every object). This is
pervasive and can't really be removed. But a new implementation
(e.g. PyPy) can and should have a real garbage collector that doesn't
suffer from such effects.

Since this is a PyPy bashing thread, maybe it's an appropriate place
to suggest that the project has got a little bit waylaid by exploring
cool things instead of releasing a useful final result?

I am not questioning rpython directly - the case for something like
that is obvious. But there's a question of balance. It's possible to
go on building ever more complex systems which are theoretically
justified, but which postpone ever finishing the job. At some point
there has to be a "good enough".

To some extent I am playing devil's advocate here, but as an outside
who looked at PyPy a while back, my uninformed and naive impression
was that the project was suffering from the kid of issues I have
caricatured above....

Andrew

PS I guess you are aware of worse is better etc? I think this may
also be a US/Euro culture issue...

On Jan 18, 8:56*am, andrew cooke <and...@acooke.org> wrote:
> Since this is a PyPy bashing thread, maybe it's an appropriate place
> to suggest that the project has got a little bit waylaid by exploring
> cool things instead of releasing a useful final result?
>
> I am not questioning rpython directly - the case for something like
> that is obvious. *But there's a question of balance. *It's possible to
> go on building ever more complex systems which are theoretically
> justified, but which postpone ever finishing the job. *At some point
> there has to be a "good enough".
>
> To some extent I am playing devil's advocate here, but as an outside
> who looked at PyPy a while back, my uninformed and naive impression
> was that the project was suffering from the kid of issues I have
> caricatured above....
>
> Andrew
>
> PS I guess you are aware of worse is better etc? *I think this may
> also be a US/Euro culture issue...

It's worth adding that, regarding the goal of having a faster python,
we have had for a long time a solid and useful proof-of-concept in
psyco. Pypy rolls up on this concept and adds many more useful
features, not all of them related to speed but to flexibility.
I have no doubt the project will be succesful. The question is how
long we will have to wait...

Luis

On Sat, Jan 17, 2009 at 7:57 PM, Paul Rubin
<"http://phr.cx"@nospam.invalid> wrote:
> alex23 <> writes:
>> Here's an article by Guido talking about the last attempt to remove
>> the GIL and the performance issues that arose:
>>
>> "I'd welcome a set of patches into Py3k *only if* the performance for
>> a single-threaded program (and for a multi-threaded but I/O-bound
>> program) *does not decrease*."
>
> The performance decrease is an artifact of CPython's rather primitive
> storage management (reference counts in every object). This is
> pervasive and can't really be removed. But a new implementation
> (e.g. PyPy) can and should have a real garbage collector that doesn't
> suffer from such effects.
> --
> http://mail.python.org/mailman/listinfo/python-list
>

That's interesting, I hadn't heard the reference counting mechanism
was related to the GIL. Is it just that you need to lock the reference
count before mutating it if there's no GIL? Really, that shouldn't be
the case. Reference counting can be done with a lock free algorithm.

Garbage collection is definitely in vogue right now. However, people
tend to treat it more like a religion than an algorithm. Garbage
collection vs reference counting actually has some trade offs both
ways. GC gets you some amortized performance gains, and some space
gains because you don't need to hang on to a bunch of counts. However,
GC also has the problem of having a very loose memory profile and poor
interactive performance during compaction if the heap is large. In
some cases this discussion becomes complicated with python because
python has both reference counting and GC.

"Brendan Miller" <> writes:
> That's interesting, I hadn't heard the reference counting mechanism
> was related to the GIL. Is it just that you need to lock the reference
> count before mutating it if there's no GIL?

Yes. Someone tried inserting such a lock but it slowed down the
single-cpu case unacceptably.

> Really, that shouldn't be
> the case. Reference counting can be done with a lock free algorithm.

That's interesting, got a reference? (no pun intended)