Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Python, subprocess, dump, gzip and Cron

Reply
Thread Tools

Python, subprocess, dump, gzip and Cron

 
 
Aidan
Guest
Posts: n/a
 
      06-10-2008
Hi,

I'm having a bit of trouble with a python script I wrote, though I'm not
sure if it's related directly to python, or one of the other software
packages...

The situation is that I'm trying to create a system backup script that
creates an image of the system, filters the output though gzip, and then
uploads the data (via ftp) to a remote site.

The problem is that when I run the script from the command line, it
works as I expect it, but when it is run by cron I only get a 20 byte
file where the compressed image should be... does anyone have any idea
as to why this might be happening? Code follows

<code>

#!/usr/bin/python

from subprocess import PIPE, Popen
from ftplib import FTP

host = 'box'

filename = '%s.img.gz' % host
ftp_host = '192.168.1.250'
ftpuser, ftppass = 'admin', 'admin'
dest_dir = '/share/%s' % host

dump = Popen('dump 0uaf - /',shell=True,stdout=PIPE)
gzip = Popen('gzip',shell=True,stdin=dump.stdout,stdout=P IPE)

ftp = FTP(ftp_host)
ftp.login(ftpuser,ftppass)
ftp.cwd(dest_dir)
ftp.storbinary('STOR %s' % filename,gzip.stdout)
ftp.quit()

print "Image '%s' created" % filename

</code>

I appreciate all feedback. Thanks in advance.
 
Reply With Quote
 
 
 
 
TT
Guest
Posts: n/a
 
      06-10-2008
On Jun 10, 2:37*pm, Aidan <(E-Mail Removed)> wrote:
> Hi,
>
> I'm having a bit of trouble with a python script I wrote, though I'm not
> sure if it's related directly to python, or one of the other software
> packages...
>
> The situation is that I'm trying to create a system backup script that
> creates an image of the system, filters the output though gzip, and then
> uploads the data (via ftp) to a remote site.
>
> The problem is that when I run the script from the command line, it
> works as I expect it, but when it is run by cron I only get a 20 byte
> file where the compressed image should be... *does anyone have any idea
> as to why this might be happening? *Code follows
>
> <code>
>
> #!/usr/bin/python
>
> from subprocess import PIPE, Popen
> from ftplib import FTP
>
> host = 'box'
>
> filename = '%s.img.gz' % host
> ftp_host = '192.168.1.250'
> ftpuser, ftppass = 'admin', 'admin'
> dest_dir = '/share/%s' % host
>
> dump = Popen('dump 0uaf - /',shell=True,stdout=PIPE)
> gzip = Popen('gzip',shell=True,stdin=dump.stdout,stdout=P IPE)
>
> ftp = FTP(ftp_host)
> ftp.login(ftpuser,ftppass)
> ftp.cwd(dest_dir)
> ftp.storbinary('STOR %s' % filename,gzip.stdout)
> ftp.quit()
>
> print "Image '%s' created" % filename
>
> </code>
>
> I appreciate all feedback. *Thanks in advance.


it's possible that the cron doesn't have the environment variables you
have, especially $PATH. So the script failed to find the command it
need to create the image.
 
Reply With Quote
 
 
 
 
Aidan
Guest
Posts: n/a
 
      06-10-2008
TT wrote:
> On Jun 10, 2:37 pm, Aidan <(E-Mail Removed)> wrote:
>> Hi,
>>
>> I'm having a bit of trouble with a python script I wrote, though I'm not
>> sure if it's related directly to python, or one of the other software
>> packages...
>>
>> The situation is that I'm trying to create a system backup script that
>> creates an image of the system, filters the output though gzip, and then
>> uploads the data (via ftp) to a remote site.
>>
>> The problem is that when I run the script from the command line, it
>> works as I expect it, but when it is run by cron I only get a 20 byte
>> file where the compressed image should be... does anyone have any idea
>> as to why this might be happening? Code follows
>>
>> <code>
>>
>> #!/usr/bin/python
>>
>> from subprocess import PIPE, Popen
>> from ftplib import FTP
>>
>> host = 'box'
>>
>> filename = '%s.img.gz' % host
>> ftp_host = '192.168.1.250'
>> ftpuser, ftppass = 'admin', 'admin'
>> dest_dir = '/share/%s' % host
>>
>> dump = Popen('dump 0uaf - /',shell=True,stdout=PIPE)
>> gzip = Popen('gzip',shell=True,stdin=dump.stdout,stdout=P IPE)
>>
>> ftp = FTP(ftp_host)
>> ftp.login(ftpuser,ftppass)
>> ftp.cwd(dest_dir)
>> ftp.storbinary('STOR %s' % filename,gzip.stdout)
>> ftp.quit()
>>
>> print "Image '%s' created" % filename
>>
>> </code>
>>
>> I appreciate all feedback. Thanks in advance.

>
> it's possible that the cron doesn't have the environment variables you
> have, especially $PATH. So the script failed to find the command it
> need to create the image.


*fore head slap*

Of course... adding the full path to both those utilities on the Popen
lines seems to have fixed it.

Thank you very much for your assistance.
 
Reply With Quote
 
Sebastian \lunar\ Wiesner
Guest
Posts: n/a
 
      06-10-2008
Aidan <(E-Mail Removed)> at Dienstag 10 Juni 2008 07:21:

> TT wrote:
>> On Jun 10, 2:37 pm, Aidan <(E-Mail Removed)> wrote:
>>> Hi,
>>>
>>> I'm having a bit of trouble with a python script I wrote, though I'm not
>>> sure if it's related directly to python, or one of the other software
>>> packages...
>>>
>>> The situation is that I'm trying to create a system backup script that
>>> creates an image of the system, filters the output though gzip, and then
>>> uploads the data (via ftp) to a remote site.
>>>
>>> The problem is that when I run the script from the command line, it
>>> works as I expect it, but when it is run by cron I only get a 20 byte
>>> file where the compressed image should be... does anyone have any idea
>>> as to why this might be happening? Code follows
>>>
>>> <code>
>>>
>>> #!/usr/bin/python
>>>
>>> from subprocess import PIPE, Popen
>>> from ftplib import FTP
>>>
>>> host = 'box'
>>>
>>> filename = '%s.img.gz' % host
>>> ftp_host = '192.168.1.250'
>>> ftpuser, ftppass = 'admin', 'admin'
>>> dest_dir = '/share/%s' % host
>>>
>>> dump = Popen('dump 0uaf - /',shell=True,stdout=PIPE)

You should avoid the use of ``shell=True`` here and use a argument list
instead:

dump = Popen(['dump', '0uaf', '-', '/'], stdout=PIPE)

This results in an exception thrown if the executable doesn't exist. This
exception can be caught and handle for instance with the logging module.

>>> gzip = Popen('gzip',shell=True,stdin=dump.stdout,stdout=P IPE)


Same here, but why don't you use the gzip functionality from the standard
library?

--
Freedom is always the freedom of dissenters.
(Rosa Luxemburg)
 
Reply With Quote
 
Aidan
Guest
Posts: n/a
 
      06-10-2008
Sebastian "lunar" Wiesner wrote:
> Aidan <(E-Mail Removed)> at Dienstag 10 Juni 2008 07:21:
>
>> TT wrote:
>>> On Jun 10, 2:37 pm, Aidan <(E-Mail Removed)> wrote:
>>>> Hi,
>>>>
>>>> I'm having a bit of trouble with a python script I wrote, though I'm not
>>>> sure if it's related directly to python, or one of the other software
>>>> packages...
>>>>
>>>> The situation is that I'm trying to create a system backup script that
>>>> creates an image of the system, filters the output though gzip, and then
>>>> uploads the data (via ftp) to a remote site.
>>>>
>>>> The problem is that when I run the script from the command line, it
>>>> works as I expect it, but when it is run by cron I only get a 20 byte
>>>> file where the compressed image should be... does anyone have any idea
>>>> as to why this might be happening? Code follows
>>>>
>>>> <code>
>>>>
>>>> #!/usr/bin/python
>>>>
>>>> from subprocess import PIPE, Popen
>>>> from ftplib import FTP
>>>>
>>>> host = 'box'
>>>>
>>>> filename = '%s.img.gz' % host
>>>> ftp_host = '192.168.1.250'
>>>> ftpuser, ftppass = 'admin', 'admin'
>>>> dest_dir = '/share/%s' % host
>>>>
>>>> dump = Popen('dump 0uaf - /',shell=True,stdout=PIPE)

> You should avoid the use of ``shell=True`` here and use a argument list
> instead:
>
> dump = Popen(['dump', '0uaf', '-', '/'], stdout=PIPE)
>
> This results in an exception thrown if the executable doesn't exist. This
> exception can be caught and handle for instance with the logging module.
>


thanks. That exception certainly would have helped me...

>>>> gzip = Popen('gzip',shell=True,stdin=dump.stdout,stdout=P IPE)

>
> Same here, but why don't you use the gzip functionality from the standard
> library?


is there a way I can create a gzip file-like object which can read the
output from the dump subprocess, and has a read method which outputs the
compressed data, which will not write to disk first? With the above
code python doesn't have to write the system image data to disk at all,
which helps when there is not enough disk space to hold an intermediate
image (at least, that is my understanding of it...).

I had a look at the gzip module, but eventually just fell back to using
the stdin and stdout of a gzip subprocess. I'd be interested to know
how it could be done using the python standard lib though.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Ruby library for cron and at Michael Ulm Ruby 1 04-16-2008 12:39 PM
cron, python and samba restart symbioid Python 1 10-29-2007 08:30 AM
script to read emails and extract attachments from cron JohnRHarlow@gmail.com Python 1 02-06-2006 09:30 PM
Re: Wits end with Python and cron Fredrik Lundh Python 0 09-28-2005 07:08 PM
Python and cron jobs. Fazer Python 8 08-07-2003 09:54 AM



Advertisments