«

I just quit an interactive session using Python 2.7 on Linux. It took in
excess of twelve minutes to exit, with the load average going well past 9
for much of that time.

I think the reason it took so long was that Python was garbage-collecting
a giant dict with 10 million entries, each one containing a list of the
form [1, [2, 3], 4]. But still, that's terribly slow -- ironically, it
took longer to dispose of the dict (12+ minutes) than it took to create
it in the first place (approx 3 minutes, with a maximum load of 4).

Can anyone explain why this was so painfully slow, and what (if anything)
I can do to avoid it in the future?

I know there is a function os._exit which effectively kills the Python
interpreter dead immediately, without doing any cleanup. What are the
consequences of doing this? I assume that the memory used by the Python
process will be reclaimed by the operating system, but other resources
such as opened files may not be.


--
Steven

On 2013-02-28, Steven D'Aprano <steve+> wrote:
> Can anyone explain why this was so painfully slow, and what (if
> anything) I can do to avoid it in the future?

I think your explanation makes sense. Maybe the nested nature of
the strings was causing it to churn looking for circular
references?

Disabling gc before exiting might do the trick, assuming you're
assiduously managing other resources with context managers.

gc.disable()
exit()


--
Neil Cerutti

On Fri, Mar 1, 2013 at 3:49 AM, Steven D'Aprano
<steve+> wrote:
> I think the reason it took so long was that Python was garbage-collecting
> a giant dict with 10 million entries, each one containing a list of the
> form [1, [2, 3], 4]. But still, that's terribly slow -- ironically, it
> took longer to dispose of the dict (12+ minutes) than it took to create
> it in the first place (approx 3 minutes, with a maximum load of 4).

Leaving the question of just *why* you have so much in your dict.....
but anyway.

Is it any different if you create a deliberate reference loop and then
stuff it into some module somewhere? That would force it to be kept
until interpreter shutdown, and then a cyclic garbage collection after
that, which quite probably would be never run. A stupid trick,
perhaps, but it might work; I tested it with a dummy class with a
__del__ method and it wasn't called. Putting it into some other module
may not be necessary, but I don't know what happens with the
interactive interpreter and what gets freed up when.

ChrisA

On Thu, Feb 28, 2013 at 12:06 PM, Chris Angelico <> wrote:
> Is it any different if you create a deliberate reference loop and then
> stuff it into some module somewhere? That would force it to be kept
> until interpreter shutdown, and then a cyclic garbage collection after
> that, which quite probably would be never run. A stupid trick,
> perhaps, but it might work; I tested it with a dummy class with a
> __del__ method and it wasn't called. Putting it into some other module
> may not be necessary, but I don't know what happens with the
> interactive interpreter and what gets freed up when.

__del__ is never called for cyclic references.

-- Devin

On Thu, Feb 28, 2013 at 12:31 PM, Devin Jeanpierre
<> wrote:
> On Thu, Feb 28, 2013 at 12:06 PM, Chris Angelico <> wrote:
>> Is it any different if you create a deliberate reference loop and then
>> stuff it into some module somewhere? That would force it to be kept
>> until interpreter shutdown, and then a cyclic garbage collection after
>> that, which quite probably would be never run. A stupid trick,
>> perhaps, but it might work; I tested it with a dummy class with a
>> __del__ method and it wasn't called. Putting it into some other module
>> may not be necessary, but I don't know what happens with the
>> interactive interpreter and what gets freed up when.
>
> __del__ is never called for cyclic references.

Sorry, I posted too early. Not only is __del__ never called, but
__del__ is the reason the cycles aren't collected. I don't know if
your trick will work without __del__.

-- Devin

On Fri, Mar 1, 2013 at 4:31 AM, Devin Jeanpierre <> wrote:
> On Thu, Feb 28, 2013 at 12:06 PM, Chris Angelico <> wrote:
>> Is it any different if you create a deliberate reference loop and then
>> stuff it into some module somewhere? That would force it to be kept
>> until interpreter shutdown, and then a cyclic garbage collection after
>> that, which quite probably would be never run. A stupid trick,
>> perhaps, but it might work; I tested it with a dummy class with a
>> __del__ method and it wasn't called. Putting it into some other module
>> may not be necessary, but I don't know what happens with the
>> interactive interpreter and what gets freed up when.
>
> __del__ is never called for cyclic references.

D'oh. Test is flawed, then. But is the theory plausible? That the
cycle detector won't be called on exit after other modules get freed?

ChrisA

On 2013-02-28, Steven D'Aprano <steve+> wrote:

> I know there is a function os._exit which effectively kills the
> Python interpreter dead immediately, without doing any cleanup. What
> are the consequences of doing this?

You loose any data you haven't saved to disk.

> I assume that the memory used by the Python process will be reclaimed
> by the operating system, but other resources such as opened files may
> not be.

All open files (including sockets, pipes, serial ports, etc) will be
flushed (from an OS standpoint) and closed. If you've closed all the
files you've written to, there should be no danger in just pulling the
plug.

--
Grant

Steven D'Aprano <steve+comp.lang.python <at> pearwood.info> writes:
>
> I just quit an interactive session using Python 2.7 on Linux. It took in
> excess of twelve minutes to exit, with the load average going well past 9
> for much of that time.
>
> I think the reason it took so long was that Python was garbage-collecting
> a giant dict with 10 million entries, each one containing a list of the
> form [1, [2, 3], 4]. But still, that's terribly slow -- ironically, it
> took longer to dispose of the dict (12+ minutes) than it took to create
> it in the first place (approx 3 minutes, with a maximum load of 4).
>
> Can anyone explain why this was so painfully slow, and what (if anything)
> I can do to avoid it in the future?

You are basically asking people to guess where your performance problem
comes from, without even providing a snippet so that people can reproduce

> I know there is a function os._exit which effectively kills the Python
> interpreter dead immediately, without doing any cleanup. What are the
> consequences of doing this? I assume that the memory used by the Python
> process will be reclaimed by the operating system, but other resources
> such as opened files may not be.

The OS always disposes of per-process resources when the process terminates
(except if the OS is buggy ). However, file buffers will not be flushed,
atexit handlers and other destructors will not be called, database
transactions will be abandoned (rolled back), etc.

Regards

Antoine.

Grant Edwards <invalid <at> invalid.invalid> writes:
>
> > I assume that the memory used by the Python process will be reclaimed
> > by the operating system, but other resources such as opened files may
> > not be.
>
> All open files (including sockets, pipes, serial ports, etc) will be
> flushed (from an OS standpoint) and closed.

According to POSIX, no, open files will not be flushed:

“The _Exit() and _exit() functions shall not call functions registered with
atexit() nor any registered signal handlers. Open streams shall not be flushed.
Whether open streams are closed (without flushing) is implementation-defined.”

http://pubs.opengroup.org/onlinepubs...ons/_exit.html

(under the hood, os._exit() calls C _exit())

Regards

Antoine.

On 03/01/2013 02:10 PM, Antoine Pitrou wrote:
> Grant Edwards <invalid <at> invalid.invalid> writes:
>> <snip>
>> All open files (including sockets, pipes, serial ports, etc) will be
>> flushed (from an OS standpoint) and closed.
>
> According to POSIX, no, open files will not be flushed:
>
> “The _Exit() and _exit() functions shall not call functions registered with
> atexit() nor any registered signal handlers. Open streams shall not be flushed.
> Whether open streams are closed (without flushing) is implementation-defined.”
>

Note he didn't say the python buffers would be flushed. It's the OS
buffers that are flushed.


--
DaveA