«

In very rare cases a program crashes (hard to reproduce) :

* several threads work on an object tree with dict's etc. in it. Items
are added, deleted, iteration over .keys() ... ). The threads are "good"
in such terms, that this core data structure is changed only by atomic
operations, so that the data structure is always consistent regarding
the application. Only the change-operations on the dicts and lists
itself seem to cause problems on a Python level ..

* one thread periodically pickle-dumps the tree to a file:
>>> cPickle.dump(obj, f)

"RuntimeError: dictionary changed size during iteration" is raised by
..dump ( or a similar "..list changed ..." )

What can I do about this to get a stable pickle-dump without risiking
execution error or even worse - errors in the pickled file ?

Is a copy.deepcopy ( -> "cPickle.dump(copy.deepcopy(obj),f)" ) an
atomic opertion with a guarantee to not fail?

Or can I only retry several times in case of RuntimeError? (which would
apears to me as odd gambling; retry how often?)

Robert


PS: Zope dumps thread exposed data structes regularly. How does the ZODB
in Zope handle dict/list changes during its pickling operations?


---
Python 2.4.1 (#2, May 5 2005, 11:32:06)
[GCC 3.3.5 (Debian 1:3.3.5-12)] on linux2

> Is a copy.deepcopy ( -> "cPickle.dump(copy.deepcopy(obj),f)" ) an
> atomic opertion with a guarantee to not fail?
>
> Or can I only retry several times in case of RuntimeError? (which would
> apears to me as odd gambling; retry how often?)

For an intermediate solution, I'm playing roulette:

for i in 1,2,3:
try:
cPickle.dump(obj, f)
break
except RuntimeError,v:
pass


I hope this works for some million years ...



> PS: Zope dumps thread exposed data structes regularly. How does the ZODB
> in Zope handle dict/list changes during its pickling operations?

robert wrote:

>
>> Is a copy.deepcopy ( -> "cPickle.dump(copy.deepcopy(obj),f)" ) an
>> atomic opertion with a guarantee to not fail?
>>
>> Or can I only retry several times in case of RuntimeError? (which
>> would apears to me as odd gambling; retry how often?)
>
>
> For an intermediate solution, I'm playing roulette:
>
> for i in 1,2,3:
> try:
> cPickle.dump(obj, f)
> break
> except RuntimeError,v:
> pass
>

hmm..

for i in 1,2,3:
try:
cPickle.dump(obj, f)
break
except RuntimeError,v:
f.seek(0);f.truncate(0)


Meanwhile I think this is a bug of cPickle.dump: It should use .keys()
instead of free iteration internally, when pickling elementary dicts.
I'd file a bug if no objection.

Robert


> I hope this works for some million years ...
>
>
>
>> PS: Zope dumps thread exposed data structes regularly. How does the
>> ZODB in Zope handle dict/list changes during its pickling operations?

Em Sáb, 2006-03-11 Ã*s 12:49 +0100, robert escreveu:
> Meanwhile I think this is a bug of cPickle.dump: It should use .keys()
> instead of free iteration internally, when pickling elementary dicts.
> I'd file a bug if no objection.

AFAICS, it's a problem with your code. You should lock your object while
using it. That's what Threading.Lock is supposed to work for. If you
want to use threads, you have to know in what parts of your code there
should be locks.

Cya,
Felipe.

--
"Quem excele em empregar a força militar subjulga os exércitos dos
outros povos sem travar batalha, toma cidades fortificadas dos outros
povos sem as atacar e destrói os estados dos outros povos sem lutas
prolongadas. Deve lutar sob o Céu com o propósito primordial da
'preservação'. Desse modo suas armas não se embotarão, e os ganhos
poderão ser preservados. Essa é a estratégia para planejar ofensivas."

-- Sun Tzu, em "A arte da guerra"

robert si è profuso/a a scrivere su comp.lang.python tutte queste
elucubrazioni:

[cut]

I don't know what's your code like, but a similar error occurred in some of
my software and it was my fault indeed. I think you should either use a
lock, or implement a deepcopy method of your own.

--
EleSSaR^ <usenetpublic->
--
Togli .xyz dalla mia email per contattarmi.

Felipe Almeida Lessa wrote:

> Em Sáb, 2006-03-11 Ã*s 12:49 +0100, robert escreveu:
>
>>Meanwhile I think this is a bug of cPickle.dump: It should use .keys()
>>instead of free iteration internally, when pickling elementary dicts.
>>I'd file a bug if no objection.
>
>
> AFAICS, it's a problem with your code. You should lock your object while
> using it. That's what Threading.Lock is supposed to work for. If you
> want to use threads, you have to know in what parts of your code there
> should be locks.

99.99% no. I would have to use a lock everywhere, where I add or remove
something into a dict or list of the struct. Thats not the purpose of
big thread locks. Such simple operations are already atomic by the
definition of Python - and thanks to the global interpreter lock.
(Otherwise I would leave the Python language, God beware ... )

I'm of course aware, where to use locks for resons of the application.
But this is an issue on Python level. And it can be solved gracly and
simple in Python - I guess:

If cPickle.dump (and maybe also copy/deepcopy?) is corrected to work
atomic on dicts (use .keys()) and list-copies or locks python threads)
the problem is solved gracely and generally.

Robert

EleSSaR^ wrote:

> robert si è profuso/a a scrivere su comp.lang.python tutte queste
> elucubrazioni:
>
> [cut]
>
> I don't know what's your code like, but a similar error occurred in some of
> my software and it was my fault indeed. I think you should either use a
> lock, or implement a deepcopy method of your own.

100s of locks? no (see other message). It should be

own deepcopy: thus, do you already know if the existing deepcopy has the
same problem as cPickle.dump ? (as the problem araises rarely, it is
difficult for me to test it out)

Robert

PS: how does ZODB work with this kind of problem? I thought is uses cPickle?

robert <no-> wrote:
...
> 99.99% no. I would have to use a lock everywhere, where I add or remove
> something into a dict or list of the struct. Thats not the purpose of
> big thread locks. Such simple operations are already atomic by the
> definition of Python - and thanks to the global interpreter lock.
> (Otherwise I would leave the Python language, God beware ... )

You have misread the Python Language Reference -- if you can give the
URL on which you have read any such promise of atomicity, I will be glad
to fix the docs to make that unambiguous.

There is no such promise (there may be implementation accidents in some
specific implementation which happen to make some operation atomic, but
NO guarantee even there that the next bugfix won't break that).

Farwell and best of luck in finding other languages which support
threads in a way that is more to your liking than Python -- maybe Ruby
suits you, I don't know for sure though.


Alex

robert si è profuso/a a scrivere su comp.lang.python tutte queste
elucubrazioni:

> own deepcopy: thus, do you already know if the existing deepcopy has the
> same problem as cPickle.dump ? (as the problem araises rarely, it is
> difficult for me to test it out)

I don't know the exact specs of your object, and I don't know what
operations are you performing on that object, nor the way they're atomic.

It seems like you're trying to save periodically the state of such object
while it is being modified (a sort of backup?), and Python complains about
that. A self-implemented deepcopy might raise anomalies (i.e. your dumped
object may be partly a 'before' object and partly an 'after' object ) as
well.

By the way, you could try employing locks from other threads to dump the
object as well... this would prevent additional locking.

> PS: how does ZODB work with this kind of problem? I thought is uses cPickle?

I have no idea about this.


--
EleSSaR^ <usenetpublic->
--
Togli .xyz dalla mia email per contattarmi.

robert si è profuso/a a scrivere su comp.lang.python tutte queste
elucubrazioni:

[cut]

P.S.
I'm very bad at threaded programming. Please verify any of my suggestions
^_^


--
EleSSaR^ <usenetpublic->
--
Togli .xyz dalla mia email per contattarmi.