Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: How suitable is Python to write system utilities?

Reply
Thread Tools

Re: How suitable is Python to write system utilities?

 
 
Alice Bevan–McGregor
Guest
Posts: n/a
 
      01-06-2011
On 2011-01-06 01:35:58 -0800, Rohit Coder said:

> Is Python suitable to write low-level system utilities like Defrag,
> Malware Removal Tools and Drivers?


Yes and no.

Python does include libraries (and has available third-party libraries)
to interface with external low-level libraries of every kind, has
Python-native third-party libraries to do things like examine ELF
object files / executables, manipulate raw IP datagrams, etc, and it is
possible to access glib (and other C libraries or even Windows DLLs)
directly from within Python without creating wrapper interfaces.

While you -can- do all of these things, the question becomes do you
really -want to-?

I've implemented server monitoring suites, FUSE filesystem drivers, and
other strange low-level things in Python. This doesn't mean I'm
getting the best "bang for the buck" when it comes to performance or
overhead; I'm trading these things for the ease of prototyping,
debugging, and the ability to utilize other high-level interfaces. My
FUSE driver won't be as performant as it would have been had I written
it in C. My server monitoring suite consumes more RAM than an
equivalent solution in C. When it comes down to it, if you want it
done efficiently, use C.

(As an aside, you -can- create hideous frankenstein monsters by using
compiled CPython modules with glue code between the driver API, e.g.
FUSE, and your CPython driver implementation; but in that case you're
adding the overhead of Python for no gain whatsoever.)

For anything system critical, Python might not be the best choice.
Malware removal tools are themselves the target of malware (e.g. virii
attempting to disable scanners and removal tools), and utilizing Python
adds (IMHO) too many points of failure.

Also, file fragmentation is a non-issue on all modern filesystems
(ext3/4, reiser, ntfs, hfs+, etc.) as they perform live-system
defragmentation to varying degrees. I have never seen a production
server of mine (utilizing reiserfs) go above 11% fragmentation
(non-contiguous extant allocations), and even that resolved itself
within a few hours of active use.

Note that non-contiguous extants are a distinct problem, reducing file
read performance substantially, thus why filesystem drivers generally
handle solving this problem by themselves. The other forms of
fragmentation (free space fragmentation and file scattering /
related-file fragmentation) substantially less so. Certain filesystems
have features designed to avoid the latter (e.g. squashfs for ordering
files on bootable CDs) and the former becomes more of an issue as you
attempt to allocate extremely large contiguous files. (Becoming worse
as free space is exhausted as more and more files of greater size need
to be shuffled around the disk platter in order to free up a contiguous
run of extants.)

Hope this helps,

- Alice.


 
Reply With Quote
 
 
 
 
J.O. Aho
Guest
Posts: n/a
 
      01-06-2011
Alice Bevan–McGregor wrote:
> On 2011-01-06 01:35:58 -0800, Rohit Coder said:
>
>> Is Python suitable to write low-level system utilities like Defrag,
>> Malware Removal Tools and Drivers?

>
> Yes and no.
>
> Also, file fragmentation is a non-issue on all modern filesystems
> (ext3/4, reiser, ntfs, hfs+, etc.) as they perform live-system
> defragmentation to varying degrees.


According to microsoft documentation, the recommendation is to run
defragmentation on ntfs on a regular bases. There seems to come some
improvement on the mft fragmentation, but still it feels long behind the
linux/unix file systems.

Do you have any recent documentation on ntfs that shows it has the capability
to defragmentate itself other than mft?



--

//Aho
 
Reply With Quote
 
 
 
 
Roy Smith
Guest
Posts: n/a
 
      01-06-2011
In article <(E-Mail Removed)>,
"J.O. Aho" <(E-Mail Removed)> wrote:

> According to microsoft documentation, the recommendation is to run
> defragmentation on ntfs on a regular bases. There seems to come some
> improvement on the mft fragmentation, but still it feels long behind the
> linux/unix file systems.
>
> Do you have any recent documentation on ntfs that shows it has the capability
> to defragmentate itself other than mft?


This is the best defragmenter for a windows file system is this:

http://www.ubuntu.com/desktop/get-ubuntu/download
 
Reply With Quote
 
J.O. Aho
Guest
Posts: n/a
 
      01-06-2011
Roy Smith wrote:
> In article <(E-Mail Removed)>,
> "J.O. Aho" <(E-Mail Removed)> wrote:
>
>> According to microsoft documentation, the recommendation is to run
>> defragmentation on ntfs on a regular bases. There seems to come some
>> improvement on the mft fragmentation, but still it feels long behind the
>> linux/unix file systems.
>>
>> Do you have any recent documentation on ntfs that shows it has the capability
>> to defragmentate itself other than mft?

>
> This is the best defragmenter for a windows file system is this:
>
> http://www.ubuntu,com/desktop/get-ubuntu/download


It depends on your taste, I favour to be able to customize quite a lot of my
installation and those rather use a meta-distribution.

As SourceMage ( www.sourcemage.org ) or Gentoo ( www.gentoo.org ).

But your reply don't point at a ms-documentation about auto defragmentation of
a file system.

--

//Aho
 
Reply With Quote
 
Roy Smith
Guest
Posts: n/a
 
      01-06-2011
In article <(E-Mail Removed)>,
"J.O. Aho" <(E-Mail Removed)> wrote:

> Roy Smith wrote:
> > In article <(E-Mail Removed)>,
> > "J.O. Aho" <(E-Mail Removed)> wrote:
> >
> >> According to microsoft documentation, the recommendation is to run
> >> defragmentation on ntfs on a regular bases. There seems to come some
> >> improvement on the mft fragmentation, but still it feels long behind the
> >> linux/unix file systems.
> >>
> >> Do you have any recent documentation on ntfs that shows it has the
> >> capability
> >> to defragmentate itself other than mft?

> >
> > This is the best defragmenter for a windows file system is this:
> >
> > http://www.ubuntu,com/desktop/get-ubuntu/download

>
> It depends on your taste, I favour to be able to customize quite a lot of my
> installation and those rather use a meta-distribution.
>
> As SourceMage ( www.sourcemage.org ) or Gentoo ( www.gentoo.org ).
>
> But your reply don't point at a ms-documentation about auto defragmentation
> of
> a file system.


I think you missed the point
 
Reply With Quote
 
David Boddie
Guest
Posts: n/a
 
      01-06-2011
On Thursday 06 January 2011 12:08, Alice Bevan?McGregor wrote:

> Python does include libraries (and has available third-party libraries)
> to interface with external low-level libraries of every kind, has
> Python-native third-party libraries to do things like examine ELF
> object files / executables, manipulate raw IP datagrams, etc, and it is
> possible to access glib (and other C libraries or even Windows DLLs)
> directly from within Python without creating wrapper interfaces.


Just out of interest, which module/package are you using to examine ELF
files?

David
 
Reply With Quote
 
Alice Bevan–McGregor
Guest
Posts: n/a
 
      01-06-2011
On 2011-01-06 06:38:24 -0800, David Boddie said:

> Just out of interest, which module/package are you using to examine ELF files?


http://pypi.python.org/pypi/elffile

- Alice.


 
Reply With Quote
 
Ahmed, Shakir
Guest
Posts: n/a
 
      01-06-2011
Hi,

I am trying to create a list in a txt file from an ftp server. The
following code is retrieving the list of the files but could not able to
write in a text file. Any help is highly appreciated.

Thanks



****************************
import os
import time
from ftplib import FTP
ftp = FTP("*.org","","") # connect to host, default port
ftp.login()
ftp.cwd("/pub/remotefolder/")
ftp.retrlines('NLST')
******************************
 
Reply With Quote
 
Ahmed, Shakir
Guest
Posts: n/a
 
      01-06-2011


-----Original Message-----
From: python-list-bounces+shahmed=(E-Mail Removed)
[mailtoython-list-bounces+shahmed=(E-Mail Removed)] On Behalf Of
Dan M
Sent: Thursday, January 06, 2011 11:06 AM
To: http://www.velocityreviews.com/forums/(E-Mail Removed)
Subject: Re: list from FTP server to a text file

On Thu, 06 Jan 2011 10:51:42 -0500, Ahmed, Shakir wrote:

> Hi,
>
> I am trying to create a list in a txt file from an ftp server. The
> following code is retrieving the list of the files but could not able

to
> write in a text file. Any help is highly appreciated.
>
> Thanks
>
>
>
> ****************************
> import os
> import time
> from ftplib import FTP
> ftp = FTP("*.org","","") # connect to host, default port ftp.login()
> ftp.cwd("/pub/remotefolder/")
> ftp.retrlines('NLST')
> ******************************


WARNING: I am a newbie! Expect more pythonic ways to do this in other
replies

from ftplib import FTP
ftp = FTP("host", "user", "pass")
ftp.cwd("/pub/myfolder")
files = ftp.nlst(".")
f = open('files.txt', 'w')
for file in files:
f.write('%s\n' % (file,))
f.close()

--

It worked
Thanks,
shk
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
When using System.IO.FileStream, I write 8 bytes, then seek to the start of the file, does the 8 bytes get flushed on seek and the buffer become a readbuffer at that point instead of being a write buffer? DR ASP .Net 2 07-29-2008 09:50 AM
Whats a suitable crypto system for this app? 509/PGP? Kurt Husler NZ Computing 8 07-04-2007 12:00 PM
The system file is not suitable for running MS-DOS and Microsoft Windows applications (on Windows 2000) Alex Vinokur Computer Support 1 10-21-2004 07:43 AM
OnInit(System.EventArgs)': no suitable method found to override Hai Nguyen ASP .Net 1 01-13-2004 07:31 AM



Advertisments