Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   Daemon loses __file__ reference after a while. (http://www.velocityreviews.com/forums/t948484-daemon-loses-__file__-reference-after-a-while.html)

ivdneut@gmail.com 07-24-2012 11:48 AM

Daemon loses __file__ reference after a while.
 
Hello,

I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:

Exception info: Traceback (most recent call last):
File "scheduler.py", line 376, in applyrule
result = execrule(rule_code)
File "scheduler.py", line 521, in execrule
rulepath = os.path.dirname(__file__)+"/"+'/'.join(rule['modules'])+"/"+rule['rulename']
NameError: name '__file__' is not defined

This section of the code is executed in this process *all the time*, but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the script interactively. This is not the case here.

I am running python from a virtual-env installation from a stock Red Hat EL6.2 installation:

(virtual-env)[user@host ~]$ python --version
Python 2.6.6
(virtual-env)[user@host ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.2 (Santiago)

I would greatly appreciate any pointers on where to start looking to find the problem.

Ian.

Ervin Hegedüs 07-24-2012 12:24 PM

Re: Daemon loses __file__ reference after a while.
 
hello,

On Tue, Jul 24, 2012 at 04:48:42AM -0700, ivdneut@gmail.com wrote:
> Hello,
>
> I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
>
> Exception info: Traceback (most recent call last):
> File "scheduler.py", line 376, in applyrule
> result = execrule(rule_code)
> File "scheduler.py", line 521, in execrule
> rulepath = os.path.dirname(__file__)+"/"+'/'.join(rule['modules'])+"/"+rule['rulename']
> NameError: name '__file__' is not defined
>
> This section of the code is executed in this process *all the time*, but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the script interactively. This is not the case here.


could you send the relevant part of the code?

I mean: how do you daemonize your process?

> I am running python from a virtual-env installation from a stock Red Hat EL 6.2 installation:
>
> (virtual-env)[user@host ~]$ python --version
> Python 2.6.6
> (virtual-env)[user@host ~]$ cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 6.2 (Santiago)


If you use fork(), it drops all file descriptors, and creates new
ones - may be then loss the __file__...?


a.


--
I � UTF-8

Laszlo Nagy 07-24-2012 12:31 PM

Re: Daemon loses __file__ reference after a while.
 

> If you use fork(), it drops all file descriptors, and creates new
> ones - may be then loss the __file__...?
>

I don't think this is the case. He wrote that the process runs for weeks
without problems, and code using __file__ is being executed all the time.

ivdneut@gmail.com 07-24-2012 12:41 PM

Re: Daemon loses __file__ reference after a while.
 
On Tuesday, July 24, 2012 2:24:31 PM UTC+2, Ervin Hegedüs wrote:
> hello,
>
> On Tue, Jul 24, 2012 at 04:48:42AM -0700, ivdneut@gmail.com wrote:
> > Hello,
> >
> > I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
> >
> > Exception info: Traceback (most recent call last):
> > File "scheduler.py", line 376, in applyrule
> > result = execrule(rule_code)
> > File "scheduler.py", line 521, in execrule
> > rulepath = os.path.dirname(__file__)+"/"+'/'..join(rule['modules'])+"/"+rule['rulename']
> > NameError: name '__file__' is not defined
> >
> > This section of the code is executed in this process *all the time*,but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the scriptinteractively. This is not the case here.
>
> could you send the relevant part of the code?
>
> I mean: how do you daemonize your process?


It's done by a double fork:

## First fork()
pid = os.fork()
if pid != 0: sys.exit(0) # parent exits.

## create new session
os.setsid()

## ignore SIGHUP
signal.signal(signal.SIGHUP, signal.SIG_IGN)

## Second fork()
pid = os.fork()
if pid != 0: sys.exit(0) # First child exits.

## Change working directory to the home directory.
homedir = pwd.getpwuid(os.geteuid())[5]
os.chdir(homedir)

os.umask(0)

for fd in range(0, 1024):
try:
os.close(fd)
except:
pass # fd not open, ignore this exception.

The original C version of this code is from W.R. Stevens' daemon_init() routine in "UNIX Network Programming Volume 1, second edition"

>
> > I am running python from a virtual-env installation from a stock RedHat EL 6.2 installation:
> >
> > (virtual-env)[user@host ~]$ python --version
> > Python 2.6.6
> > (virtual-env)[user@host ~]$ cat /etc/redhat-release
> > Red Hat Enterprise Linux Server release 6.2 (Santiago)
>
> If you use fork(), it drops all file descriptors, and creates new
> ones - may be then loss the __file__...?


I doubt this would be it, or it would stop working immediately, since daemonization is done upon startup of the process. File descriptors are closed immediately upon startup, it doesn't seem to affect the reference to the __file__ string (which is not a file object, but a str object)

>
>
> a.
>
>
> --
> I � UTF-8




On Tuesday, July 24, 2012 2:24:31 PM UTC+2, Ervin Hegedüs wrote:
> hello,
>
> On Tue, Jul 24, 2012 at 04:48:42AM -0700, ivdneut@gmail.com wrote:
> > Hello,
> >
> > I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
> >
> > Exception info: Traceback (most recent call last):
> > File "scheduler.py", line 376, in applyrule
> > result = execrule(rule_code)
> > File "scheduler.py", line 521, in execrule
> > rulepath = os.path.dirname(__file__)+"/"+'/'..join(rule['modules'])+"/"+rule['rulename']
> > NameError: name '__file__' is not defined
> >
> > This section of the code is executed in this process *all the time*,but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the scriptinteractively. This is not the case here.
>
> could you send the relevant part of the code?
>
> I mean: how do you daemonize your process?
>
> > I am running python from a virtual-env installation from a stock RedHat EL 6.2 installation:
> >
> > (virtual-env)[user@host ~]$ python --version
> > Python 2.6.6
> > (virtual-env)[user@host ~]$ cat /etc/redhat-release
> > Red Hat Enterprise Linux Server release 6.2 (Santiago)
>
> If you use fork(), it drops all file descriptors, and creates new
> ones - may be then loss the __file__...?
>
>
> a.
>
>
> --
> I � UTF-8



ivdneut@gmail.com 07-24-2012 12:41 PM

Re: Daemon loses __file__ reference after a while.
 
On Tuesday, July 24, 2012 2:24:31 PM UTC+2, Ervin Hegedüs wrote:
> hello,
>
> On Tue, Jul 24, 2012 at 04:48:42AM -0700, ivdneut@gmail.com wrote:
> > Hello,
> >
> > I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
> >
> > Exception info: Traceback (most recent call last):
> > File "scheduler.py", line 376, in applyrule
> > result = execrule(rule_code)
> > File "scheduler.py", line 521, in execrule
> > rulepath = os.path.dirname(__file__)+"/"+'/'..join(rule['modules'])+"/"+rule['rulename']
> > NameError: name '__file__' is not defined
> >
> > This section of the code is executed in this process *all the time*,but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the scriptinteractively. This is not the case here.
>
> could you send the relevant part of the code?
>
> I mean: how do you daemonize your process?


It's done by a double fork:

## First fork()
pid = os.fork()
if pid != 0: sys.exit(0) # parent exits.

## create new session
os.setsid()

## ignore SIGHUP
signal.signal(signal.SIGHUP, signal.SIG_IGN)

## Second fork()
pid = os.fork()
if pid != 0: sys.exit(0) # First child exits.

## Change working directory to the home directory.
homedir = pwd.getpwuid(os.geteuid())[5]
os.chdir(homedir)

os.umask(0)

for fd in range(0, 1024):
try:
os.close(fd)
except:
pass # fd not open, ignore this exception.

The original C version of this code is from W.R. Stevens' daemon_init() routine in "UNIX Network Programming Volume 1, second edition"

>
> > I am running python from a virtual-env installation from a stock RedHat EL 6.2 installation:
> >
> > (virtual-env)[user@host ~]$ python --version
> > Python 2.6.6
> > (virtual-env)[user@host ~]$ cat /etc/redhat-release
> > Red Hat Enterprise Linux Server release 6.2 (Santiago)
>
> If you use fork(), it drops all file descriptors, and creates new
> ones - may be then loss the __file__...?


I doubt this would be it, or it would stop working immediately, since daemonization is done upon startup of the process. File descriptors are closed immediately upon startup, it doesn't seem to affect the reference to the __file__ string (which is not a file object, but a str object)

>
>
> a.
>
>
> --
> I � UTF-8




On Tuesday, July 24, 2012 2:24:31 PM UTC+2, Ervin Hegedüs wrote:
> hello,
>
> On Tue, Jul 24, 2012 at 04:48:42AM -0700, ivdneut@gmail.com wrote:
> > Hello,
> >
> > I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
> >
> > Exception info: Traceback (most recent call last):
> > File "scheduler.py", line 376, in applyrule
> > result = execrule(rule_code)
> > File "scheduler.py", line 521, in execrule
> > rulepath = os.path.dirname(__file__)+"/"+'/'..join(rule['modules'])+"/"+rule['rulename']
> > NameError: name '__file__' is not defined
> >
> > This section of the code is executed in this process *all the time*,but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the scriptinteractively. This is not the case here.
>
> could you send the relevant part of the code?
>
> I mean: how do you daemonize your process?
>
> > I am running python from a virtual-env installation from a stock RedHat EL 6.2 installation:
> >
> > (virtual-env)[user@host ~]$ python --version
> > Python 2.6.6
> > (virtual-env)[user@host ~]$ cat /etc/redhat-release
> > Red Hat Enterprise Linux Server release 6.2 (Santiago)
>
> If you use fork(), it drops all file descriptors, and creates new
> ones - may be then loss the __file__...?
>
>
> a.
>
>
> --
> I � UTF-8



Dieter Maurer 07-24-2012 02:59 PM

Re: Daemon loses __file__ reference after a while.
 
"ivdneut@gmail.com" <ivdneut@gmail.com> writes:

> I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
>
> Exception info: Traceback (most recent call last):
> File "scheduler.py", line 376, in applyrule
> result = execrule(rule_code)
> File "scheduler.py", line 521, in execrule
> rulepath = os.path.dirname(__file__)+"/"+'/'.join(rule['modules'])+"/"+rule['rulename']
> NameError: name '__file__' is not defined
>
> This section of the code is executed in this process *all the time*, but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the script interactively. This is not the case here.


This is strange indeed.

I have only one vague idea: should something try to terminate the
process, modules would start to lose their variables during shutdown.


Paul Rubin 07-24-2012 07:32 PM

Re: Daemon loses __file__ reference after a while.
 
Dieter Maurer <dieter@handshake.de> writes:
> I have only one vague idea: should something try to terminate the
> process, modules would start to lose their variables during shutdown.


That happens all the time with multi-threaded programs, because the
shutdown is happening concurrently with other threads doing stuff. Are
there threads in this particular program?

Ian Kelly 07-24-2012 08:02 PM

Re: Daemon loses __file__ reference after a while.
 
On Tue, Jul 24, 2012 at 8:59 AM, Dieter Maurer <dieter@handshake.de> wrote:
> "ivdneut@gmail.com" <ivdneut@gmail.com> writes:
>
>> I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
>>
>> Exception info: Traceback (most recent call last):
>> File "scheduler.py", line 376, in applyrule
>> result = execrule(rule_code)
>> File "scheduler.py", line 521, in execrule
>> rulepath = os.path.dirname(__file__)+"/"+'/'.join(rule['modules'])+"/"+rule['rulename']
>> NameError: name '__file__' is not defined
>>
>> This section of the code is executed in this process *all the time*, butsuddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the script interactively. This is not the case here.

>
> This is strange indeed.
>
> I have only one vague idea: should something try to terminate the
> process, modules would start to lose their variables during shutdown.


That's a good theory. Or perhaps something in the code itself is
handling the module's globals() and very occasionally does something
that is incorrect and destructive.

Ian Kelly 07-24-2012 08:08 PM

Re: Daemon loses __file__ reference after a while.
 
On Tue, Jul 24, 2012 at 1:32 PM, Paul Rubin <no.email@nospam.invalid> wrote:
> Dieter Maurer <dieter@handshake.de> writes:
>> I have only one vague idea: should something try to terminate the
>> process, modules would start to lose their variables during shutdown.

>
> That happens all the time with multi-threaded programs, because the
> shutdown is happening concurrently with other threads doing stuff. Are
> there threads in this particular program?


It also comes up in single-threaded programs that use finalizers
(__del__ methods). At the time an object is finalized, many globals
might already be gone.


All times are GMT. The time now is 10:33 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.