Velocity Reviews

Total Beginner - Extracting Data from a Database Online (Screenshot)

logan.c.graham@gmail.com — Fri, 24 May 2013 17:32:26 GMT

Hey guys,

I'm learning Python and I'm experimenting with different projects -- I like learning by doing. I'm wondering if you can help me here:

http://i.imgur.com/KgvSKWk.jpg

What this is is a publicly-accessible webpage that's a simple database of people who have used the website. Ideally what I'd like to end up with is an excel spreadsheet with data from the columns #fb, # vids, fb sent?, # email tm.

I'd like to use Python to do it -- crawl the page and extract the data in a usable way.

I'd love your input! I'm just a learner.

Re: PEP 378: Format Specifier for Thousands Separator

Andreas Perstinger — Fri, 24 May 2013 17:28:29 GMT

On 24.05.2013 17:25, Carlos Nepomuceno wrote:
> ----------------------------------------
>> Date: Thu, 23 May 2013 19:29:14 -0700
>> Subject: Re: PEP 378: Format Specifier for Thousands Separator
>> From:
>> [some typical dihedral stuff]
>
> I'm sorry but I don't understand your question.

Don't worry. I guess most people on this list don't understand it either.

It looks like dihedral is a bot although nobody knows for sure. Search
for some other posts from him/her/it in the archive and form your own
opinion.

IMHO you can simply ignore him/her/it.

Bye, Andreas

address of local parameter

kerravon — Fri, 24 May 2013 17:12:31 GMT

On 05/24/2013 01:12 PM, kerravon wrote:
> I found that it was necessary to put a "static" below:
>
> static int testGenericBlockDeviceRequest(void)
> {
> int ret;
> int x;
> int y;
> static PB_1560 parm_block;
> char *p=&parm_block;
>
> Why is that? And why didn't the compiler (TC++ 1.01)
> complain about it if it is wrong?

TC++ 1.01 sounds like a C++ compiler. That's OK if you were using it in
a mode where it was working as a C compiler; otherwise, you're likely to
get better answers to your question in comp.lang.c++.

PB_1560 is presumably a typedef. What is a typedef for?

There's two different occurrences of 'static' in that code. Which one
are you referring to?

If the compiler didn't complain, then you reached the conclusion that
'static' was needed based upon some evidence other than a compiler error
message. What was that evidence?

With an appropriate typedef of PB_1560, either or both of the
occurrences of 'static' could be removed from your code, and it could
still be part of a perfectly reasonable program, which is probably why
your compiler didn't complain. Of course, removing either one would
change the meaning of the program, and that difference in meaning could
cause problems, but that depends upon what the rest of your program does
with testGenericBlockDeviceRequest(). What was the problem you had,
which you fixed by inserting 'static'?

RE: PEP 378: Format Specifier for Thousands Separator

Carlos Nepomuceno — Fri, 24 May 2013 15:25:45 GMT

----------------------------------------
> Date: Thu, 23 May 2013 19:29:14 -0700
> Subject: Re: PEP 378: Format Specifier for Thousands Separator
> From:
[...]
> Could a separate instance like the I/O device of a subprocess
> to be easily available in Python?
>
> The next question would be whether the flow of several I/O data streams could be easily piped and manipulated in the high level
> programming designs without consuming too much resources.

I'm sorry but I don't understand your question.

Do you mean returning the output of another process to the caller? Like:

import subprocess
output = subprocess.check_output(['ping', 'google.com'])

Piping processes works with 'shell = True' but not otherwise.

Luca Cerone — Fri, 24 May 2013 14:04:36 GMT

Hi everybody,
I am new to the group (and relatively new to Python)
so I am sorry if this issues has been discussed (although searching for topics in the group I couldn't find a solution to my problem).

I am using Python 2.7.3 to analyse the output of two 3rd parties programs that can be launched in a linux shell as:

program1 | program2

To do this I have written a function that pipes program1 and program2 (using subprocess.Popen) and the stdout of the subprocess, and a function that parses the output:

A basic example:

from subprocess import Popen, STDOUT, PIPE
def run():
p1 = Popen(['program1'], stdout = PIPE, stderr = STDOUT)
p2 = Popen(['program2'], stdin = p1.stdout, stdout = PIPE, stderr = STDOUT)
p1.stdout.close()
return p2.stdout

def parse(out):
for row in out:
print row
#do something else with each line
out.close()
return parsed_output

# main block here

pout = run()

parsed = parse(pout)

#--- END OF PROGRAM ----#

I want to parse the output of 'program1 | program2' line by line because the output is very large.

When running the code above, occasionally some error occurs (IOERROR: [Errno 0]). However this error doesn't occur if I code the run() function as:

def run():
p = Popen('program1 | program2', shell = True, stderr = STDOUT, stdout = PIPE)
return p.stdout

I really can't understand why the first version causes errors, while the second one doesn't.

Can you please help me understanding what's the difference between the two cases?

Thanks a lot in advance for the help,
Cheers, Luca

Re: NetBSD Crypto Bug: sizeof v. sizeof()

Ian Pilcher — Fri, 24 May 2013 14:00:07 GMT

On 05/23/2013 08:20 PM, wrote:
> This bad code[1] compromised[2] NetBSD's CPRNG implementation:
>
> rnd_extract_data(key + r, sizeof(key - r),
> RND_EXTRACT_ANY);
>
> One might argue this bug stemmed from using the sizeof() idiom instead of
> stylistically treating sizeof as a regular operator.

That seems like a construct for a which a compiler might want to issue a
warning.

--
================================================== ======================
Ian Pilcher
Sometimes there's nothing left to do but crash and burn...or die trying.
================================================== ======================

Re: Read txt file, add to iptables not working on new host

Chris Angelico — Fri, 24 May 2013 13:54:05 GMT

On Sat, May 25, 2013 at 2:32 AM, JackM <> wrote:
> So Chris, does this version look better? Changed to inFile to with.
>

Heh, I didn't know you knew about with :) Since you know how to use
it, you probably also know why it's useful. Anyway, the main thing is
to see the exact command that's being executed, which you then should
be able to try at a shell prompt.

ChrisA

Photo ops at Acadia National Park and environs

Bill — Fri, 24 May 2013 13:42:02 GMT

Planning a week-long visit to Acadia. Looking for interesting venues to shoot and, of course, the iconic views. I'd be grateful for suggestions.

RE: Read txt file, add to iptables not working on new host

Carlos Nepomuceno — Fri, 24 May 2013 13:15:45 GMT

No, there's no need to change your python script, although it can be improved because as it is it may flush (delete all) iptables rules and let you vulnerable and don't create the new rules.

All you need to do is enter the commands in the shell and send it's output.The 'iptables' have changed.

----------------------------------------
> From:
> Subject: Re: Read txt file, add to iptables not working on new host
> Date: Fri, 24 May 2013 09:08:26 -0400
> To: python-
>
> Thanks for answering. Do you mean something like this?
>
> outPut = os.popen('uname -a' '/sbin/iptables -V INPUT -s' + ' ' + IP + '
> ' + '-j REJECT' )
>
> Sorry but like I said, I have no experience with any of this.
>
>
>
> On 5/23/2013 11:10 PM, Carlos Nepomuceno wrote:
>> Send the output of the following commands:
>>
>> uname -a
>> /sbin/iptables -V
>>
>>
>> ----------------------------------------
>>> From:
>>> Subject: Read txt file, add to iptables not working on new host
>>> Date: Thu, 23 May 2013 22:44:38 -0400
>>> To: python-
>>>
>>> First, let me say that I have no knowledge of or experience with Python
>>> or Linux/Unix. I have a script which was written by a host tech person
>>> that ran via cron on my old server. It was designed to read IP addresses
>>> from a text file and add them to be blocked on iptables. That way, we
>>> could add or remove IPs without involving tech support daily. It worked
>>> great.
>>>
>>> Then we changed hosts and this script is now throwing errors on the new
>>> server. This host runs Python 2.6.6. This is the script:
>>>
>>> #!/usr/bin/python
>>> import os,time
>>>
>>> ##Input, Output, and TimeStamp
>>> inFile = open('/var/www/html/mydomain.com/banlist.txt','r')
>>> logFile = open('/var/log/banList.log','w')
>>> stamp = time.asctime(time.localtime())
>>>
>>>
>>> ##Daily Flush of blockList rules before re-applying Blocks
>>> os.popen('/sbin/iptables -F INPUT')
>>> logFile.write(stamp), logFile.write('\n'), logFile.write('Flushing
>>> Rules..\n')
>>>
>>> ##Loop to read in file and Apply rules to IPtables
>>> for line in inFile.readlines():
>>> tmp = line.split(';')
>>> IP = tmp[0]
>>> outPut = os.popen( '/sbin/iptables -A INPUT -s' + ' ' + IP + ' ' + '-j
>>> REJECT' )
>>> logFile.write(IP), logFile.write(' - Has been blocked '),
>>> logFile.write(stamp),logFile.write
>>>
>>>
>>> The errors we're getting are like these:
>>>
>>> Bad argument `174.37.65.204'
>>> Try `iptables -h' or 'iptables --help' for more information.
>>> Bad argument `94.159.162.182'
>>> Try `iptables -h' or 'iptables --help' for more information.
>>> Bad argument `95.134.132.98'
>>> Try `iptables -h' or 'iptables --help' for more information.
>>> etc.
>>>
>>> Entries from the banlist.txt are like these:
>>>
>>> 200.193.54.138; February 9, 2013, 7:42 am

>>> 87.120.57.4; February 9, 2013, 7:42 am

>>> 82.206.129.160; February 9, 2013, 7:43 am

>>> etc.
>>>
>>> I know the error points to a bad iptables command.
>>> Can someone tell me what change(s) I need to make to this script to get
>>> it working again? Thanks.
>>>
>>>
>>>
>>> --
>>> My email address on the header is a non-monitored spam catching account..
>>> I can be reached via http://www.wvnh.net/contact.htm
>>>
>>> --
>
>
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list

Re: Utility to locate errors in regular expressions

Devin Jeanpierre — Fri, 24 May 2013 13:13:41 GMT

In article ,
Devin Jeanpierre <> wrote:

> On Fri, May 24, 2013 at 8:58 AM, Malte Forkel <> wrote:
> > As a first step, I am looking for a parser for Python regular
> > expressions, or a Python regex grammar to create a parser from.
>
> the sre_parse module is undocumented, but very usable.
>
> > But may be my idea is flawed? Or a similar (or better) tools already
> > exists? Any advice will be highly appreciated!
>
> I think your task is made problematic by the possibility that no
> single part of the regexp causes a match failure. What causes failure
> depends on what branches are chosen with the |, *, +, ?, etc.
> operators -- it might be a different character/subexpression for each
> branch. And then there's exponentially many possible branches.

That's certainly true. The full power of regex makes stuff like this
very hard to do in the general case. That being said, people tend to
write regexen which match hunks of text from left to right.

So, in theory, it's probably an intractable problem. But, in practice,
such a tool would actually be useful in a large set of real-life cases.