python simply not scaleable enough for google?
 

http://groups.google.com/group/unlad...f544643e?pli=1

thoughts?

rday
--

================================================== ======================
Robert P. J. Day Waterloo, Ontario, CANADA

Linux Consulting, Training and Kernel Pedantry.

Web page: http://crashcourse.ca
Twitter: http://twitter.com/rpjday
================================================== ======================

Re: python simply not scaleable enough for google?
 

On Nov 11, 3:57*am, "Robert P. J. Day" <rpj...@crashcourse.ca> wrote:
> http://groups.google.com/group/unlad...hread/thread/4...
>
> * thoughts?

Google's already given us its thoughts:
http://developers.slashdot.org/story...mming-Language

Re: python simply not scaleable enough for google?
 

"Robert P. J. Day" <rpjday@crashcourse.ca> writes:
> http://groups.google.com/group/unlad...f544643e?pli=1
> thoughts?

I'd bet it's not just about multicore scaling and general efficiency,
but also the suitability of the language itself for large, complex
projects. It's just not possible to be everything for everybody.
Python is beginner-friendly, has a very fast learning curve for
experienced programmers in other languages, and is highly productive
for throwing small and medium sized scripts together, that are
debugged through iterated testing. One might say it's optimized for
those purposes. I use it all the time because a lot of my programming
fits the pattern. The car analogy is the no-frills electric commuter
car, just hop in and aim it where you want to go; if you crash it,
brush yourself off and restart. But there are times (large production
applications) when you really want the Airbus A380 with the 100's of
automatic monitoring systems and checkout procedures to follow before
you take off, even if the skill level needed to use it is much higher
than the commuter car.

Re: python simply not scaleable enough for google?
 

On 2009-11-13, at 19:53, Paul Rubin wrote:

> "Robert P. J. Day" <rpjday@crashcourse.ca> writes:
>> http://groups.google.com/group/unlad...f544643e?pli=1
>> thoughts?
>
> I'd bet it's not just about multicore scaling and general efficiency,
> but also the suitability of the language itself for large, complex
> projects. It's just not possible to be everything for everybody.
> Python is beginner-friendly, has a very fast learning curve for
> experienced programmers in other languages, and is highly productive
> for throwing small and medium sized scripts together, that are
> debugged through iterated testing. One might say it's optimized for
> those purposes. I use it all the time because a lot of my programming
> fits the pattern. The car analogy is the no-frills electric commuter
> car, just hop in and aim it where you want to go; if you crash it,
> brush yourself off and restart. But there are times (large production
> applications) when you really want the Airbus A380 with the 100's of
> automatic monitoring systems and checkout procedures to follow before
> you take off, even if the skill level needed to use it is much higher
> than the commuter car.

OK. The quoted link deals with Unladen Swallow, which is an attempt to deal with the
very real performance limitations of current Python systems. The remarks above deal with
productivity scalability, which is a totally different matter. So...

People can and do write large programs in Python, not just `throwing...medium sized
scripts together'. Unlike, say, Javascript, it has the necessary machinery to build very
large programs that are highly maintainable. One can reasonably compare it with Java, C#,
and Smalltalk; the facilities are comparable, and all of those (as well as Python) are
used for building enterprise systems.

I believe that the A380's control software is largely written in Ada, which is a
perfectly fine programming language that I would prefer not to write code in. For
approximately 10 years, US DOD pretty much required the use of Ada in military (and
aerospace) software (though a a couple of years ago I discovered that there is still
one remaining source of Jovial compilers that still sells to DOD). According to a
presentation by Lt. Colonel J. A. Hamilton, `Programming Language Policy in the DOD:
After The Ada Mandate', given in 1999, `We are unlikely to see a return of a programming
language mandate' (www.drew-hamilton.com/stc99/stcAda_99.pdf). As I understand it,
the removal of the Ada mandate came from the realization (20 years after many computer
scientists *told* DOD this) that software engineering processes contribute more to
reliability than do programming language structures (c.f. Fred Brooks, `No Silver
Bullet').

So: to sum up, there are lots of large systems where Python might be totally appropriate,
especially if complemented with processes that feature careful specification and strong
automated testing. There are some large systems where Python would definitely NOT be
the language of choice, or even usable at all, because different engineering processes
were in place.

From a productivity viewpoint, there is no data to say that Python is more, less, or equally
scalable than <Language X> in that it produces correctly-tested, highly-maintainable programs
at a lower, higher, or equal cost. I would appreciate it if people who wanted to comment on
Python's scalability or lack thereof would give another programming language that they would
compare it with.

-- v

Re: python simply not scaleable enough for google?
 

In article <mailman.224.1257933469.2873.python-list@python.org>,
Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>
>http://groups.google.com/group/unlad...f544643e?pli=1
>
> thoughts?

Haven't seen this elsewhere in the thread:

http://dalkescientific.com/writings/..._tasklets.html
--
Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it." --Brian W. Kernighan

Re: python simply not scaleable enough for google?
 

Aahz wrote:
> In article <mailman.224.1257933469.2873.python-list@python.org>,
> Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>> http://groups.google.com/group/unlad...f544643e?pli=1
>>
>> thoughts?
>
> Haven't seen this elsewhere in the thread:
>
> http://dalkescientific.com/writings/..._tasklets.html


I looked at this and it looks very good in that stackless appears twice as fast
as go(lang) (I used to be in the department of computing at Imperial so I
suppose I have to side with McCabe).

Anyhow, my reading of why Pike was so proud of his set up and tear down of the
tasks example was that these were real threads.

Presumably that means they could potentially run in parallel on the 100000 cpu
machines of the future.

I'm not so clear on whether the threadless tasklets will run on separate cpus.
--
Robin Becker

Re: python simply not scaleable enough for google?
 

On 20 Nov, 11:12, Robin Becker <ro...@reportlab.com> wrote:

> Presumably that means they could potentially run in parallel on the 100000 cpu
> machines of the future.
>
> I'm not so clear on whether the threadless tasklets will run on separate cpus.

You can make a user-space scheduler and run a 100000 tasklets on a
threadpool. But there is a GIL in stackless as well.

Nobody wants 100000 OS threads, not with Python, not with Go, not with
C.

Also note that Windows has native support for "taskelets", regardless
of language. They are called "fibers" (as opposed to "threads") and
are created using the CreateFiber system call. I would not be
surprised if Unix'es has this as well. We do not need Stackless for
light-weight threads. We can just take Python's threading modules' C
code and replace CreateThread with CreateFiber.

Re: python simply not scaleable enough for google?
 

In article <de86d30e-c9c1-4d30-9e62-d043b78ea070@a31g2000yqn.googlegroups.com>,
sturlamolden <sturlamolden@yahoo.no> wrote:
>
>Also note that Windows has native support for "taskelets", regardless
>of language. They are called "fibers" (as opposed to "threads") and are
>created using the CreateFiber system call. I would not be surprised if
>Unix'es has this as well. We do not need Stackless for light-weight
>threads. We can just take Python's threading modules' C code and
>replace CreateThread with CreateFiber.

Are you advocating a high-fiber diet?
--
Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it." --Brian W. Kernighan

Re: python simply not scaleable enough for google?
 

On 20 Nov, 22:45, a...@pythoncraft.com (Aahz) wrote:

> Are you advocating a high-fiber diet?

Only if you are a ruminant.

No really...

Windows has user-space threads natively. But you must reserve some
stack space for them (from virtual memory), which mainly makes them
useful on 64 bit systems.

Re: python simply not scaleable enough for google?
 

On Fri, 20 Nov 2009 09:51:49 -0800, sturlamolden wrote:

> You can make a user-space scheduler and run a 100000 tasklets on a
> threadpool. But there is a GIL in stackless as well.
>
> Nobody wants 100000 OS threads, not with Python, not with Go, not with
> C.
>
> Also note that Windows has native support for "taskelets", regardless
> of language. They are called "fibers" (as opposed to "threads") and
> are created using the CreateFiber system call. I would not be
> surprised if Unix'es has this as well. We do not need Stackless for
> light-weight threads. We can just take Python's threading modules' C
> code and replace CreateThread with CreateFiber.

POSIX.1-2001 and POSIX.1-2004 have makecontext(), setcontext(),
getcontext() and swapcontext(), but obsoleted by POSIX.1-2008.

They are available on Linux; I don't know about other Unices.