Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > help re recording/replaying (i.e. automating) HTTP interactions to aweb-site???

Reply
Thread Tools

help re recording/replaying (i.e. automating) HTTP interactions to aweb-site???

 
 
Greg Hauptmann
Guest
Posts: n/a
 
      04-26-2008
Hi,

Actually can anyone recommend a good technique / software / plugin
that would assist if I wanted to effectively (a) record my interaction
with my bank at the HTTP level, then (b) use this to automate that
behavior in my RoR application to automate pulling down daily account
details?

The best I can think of at the moment is: (a) Firefox Live HTTP
Headers plugin then (b) manually write Ruby code that sends these out
and waits for the response & check it before proceeding to the next
http request. I'm thinking someone probably has a better way, or
plugin, to handle at least part (b)?

Tks

 
Reply With Quote
 
 
 
 
Peter Szinek
Guest
Posts: n/a
 
      04-26-2008
[Note: parts of this message were removed to make it a legal post.]

Greg,

Have you looked into (Fire)Watir?

I am just releasing a new version of scRUBYt! (a web scraping
framework) where it will be possible to use FireWatir as the agent for
navigation/scraping, so you can write a simple but powerful DSL (stuff
like 'click_link', 'fill_textfield' etc) which is executed through
Firefox and is very well suited for scenarios you just described. Drop
me a line if you are interested.

But of course plain (Fire)Watir would do, too.

Cheers,
Peter
___
http://www.rubyrailways.com
http://scrubyt.org

On Apr 26, 2008, at 10:47 AM, Greg Hauptmann wrote:

> Hi,
>
> Actually can anyone recommend a good technique / software / plugin
> that would assist if I wanted to effectively (a) record my interaction
> with my bank at the HTTP level, then (b) use this to automate that
> behavior in my RoR application to automate pulling down daily account
> details?
>
> The best I can think of at the moment is: (a) Firefox Live HTTP
> Headers plugin then (b) manually write Ruby code that sends these out
> and waits for the response & check it before proceeding to the next
> http request. I'm thinking someone probably has a better way, or
> plugin, to handle at least part (b)?
>
> Tks
>



 
Reply With Quote
 
 
 
 
Greg Hauptmann
Guest
Posts: n/a
 
      04-26-2008
sounds good, it support:
=95 https?
=95 cookies?
=95 building in some intelligence? (say when the link for step N will
change over time but you can write an algorithm for it)

thanks

On 4/26/08, Peter Szinek <(E-Mail Removed)> wrote:
> Greg,
>
> Have you looked into (Fire)Watir?
>
> I am just releasing a new version of scRUBYt! (a web scraping
> framework) where it will be possible to use FireWatir as the agent for
> navigation/scraping, so you can write a simple but powerful DSL (stuff
> like 'click_link', 'fill_textfield' etc) which is executed through
> Firefox and is very well suited for scenarios you just described. Drop
> me a line if you are interested.
>
> But of course plain (Fire)Watir would do, too.
>
> Cheers,
> Peter
> ___
> http://www.rubyrailways.com
> http://scrubyt.org
>
> On Apr 26, 2008, at 10:47 AM, Greg Hauptmann wrote:
>
> > Hi,
> >
> > Actually can anyone recommend a good technique / software / plugin
> > that would assist if I wanted to effectively (a) record my interaction
> > with my bank at the HTTP level, then (b) use this to automate that
> > behavior in my RoR application to automate pulling down daily account
> > details?
> >
> > The best I can think of at the moment is: (a) Firefox Live HTTP
> > Headers plugin then (b) manually write Ruby code that sends these out
> > and waits for the response & check it before proceeding to the next
> > http request. I'm thinking someone probably has a better way, or
> > plugin, to handle at least part (b)?
> >
> > Tks
> >

>
>


 
Reply With Quote
 
Greg Hauptmann
Guest
Posts: n/a
 
      04-26-2008
PS. 4th question Peter I forgot:
=95 does it support downloading a file (eg csv file, account transactions)

On 4/27/08, Greg Hauptmann <(E-Mail Removed)> wrote:
> sounds good, it support:
> =95 https?
> =95 cookies?
> =95 building in some intelligence? (say when the link for step N will
> change over time but you can write an algorithm for it)
>
> thanks
>
> On 4/26/08, Peter Szinek <(E-Mail Removed)> wrote:
> > Greg,
> >
> > Have you looked into (Fire)Watir?
> >
> > I am just releasing a new version of scRUBYt! (a web scraping
> > framework) where it will be possible to use FireWatir as the agent for
> > navigation/scraping, so you can write a simple but powerful DSL (stuff
> > like 'click_link', 'fill_textfield' etc) which is executed through
> > Firefox and is very well suited for scenarios you just described. Drop
> > me a line if you are interested.
> >
> > But of course plain (Fire)Watir would do, too.
> >
> > Cheers,
> > Peter
> > ___
> > http://www.rubyrailways.com
> > http://scrubyt.org
> >
> > On Apr 26, 2008, at 10:47 AM, Greg Hauptmann wrote:
> >
> > > Hi,
> > >
> > > Actually can anyone recommend a good technique / software / plugin
> > > that would assist if I wanted to effectively (a) record my interactio=

n
> > > with my bank at the HTTP level, then (b) use this to automate that
> > > behavior in my RoR application to automate pulling down daily account
> > > details?
> > >
> > > The best I can think of at the moment is: (a) Firefox Live HTTP
> > > Headers plugin then (b) manually write Ruby code that sends these out
> > > and waits for the response & check it before proceeding to the next
> > > http request. I'm thinking someone probably has a better way, or
> > > plugin, to handle at least part (b)?
> > >
> > > Tks
> > >

> >
> >

>


 
Reply With Quote
 
Phillip Gawlowski
Guest
Posts: n/a
 
      04-26-2008
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greg Hauptmann wrote:
| PS. 4th question Peter I forgot:
| does it support downloading a file (eg csv file, account transactions)

http://wtr.rubyforge.org/

Find it out?

- --
Phillip Gawlowski
Twitter: twitter.com/cynicalryan

A born loser:
~ Somebody who calls the number that's scrawled in lipstick on the phone
~ booth wall-- and his wife answers.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkgTn5wACgkQbtAgaoJTgL+jvgCePwARmYTIE1 hktGz6yVD0JeWk
rHMAnRt+JpgafQAJivHFyXvag8Tt2duT
=smt6
-----END PGP SIGNATURE-----

 
Reply With Quote
 
Greg Hauptmann
Guest
Posts: n/a
 
      04-27-2008
I see Watar requircs/drives a browser...i'm after something browser
independent...any other library/plugin suggestions?

On 4/27/08, Phillip Gawlowski <(E-Mail Removed)> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Greg Hauptmann wrote:
> | PS. 4th question Peter I forgot:
> | =95 does it support downloading a file (eg csv file, account transactio=

ns)
>
> http://wtr.rubyforge.org/
>
> Find it out?
>
> - --
> Phillip Gawlowski
> Twitter: twitter.com/cynicalryan
>
> A born loser:
> ~ Somebody who calls the number that's scrawled in lipstick on the phone
> ~ booth wall-- and his wife answers.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkgTn5wACgkQbtAgaoJTgL+jvgCePwARmYTIE1 hktGz6yVD0JeWk
> rHMAnRt+JpgafQAJivHFyXvag8Tt2duT
> =3Dsmt6
> -----END PGP SIGNATURE-----
>
>


 
Reply With Quote
 
Phillip Gawlowski
Guest
Posts: n/a
 
      04-27-2008
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

A: Because it makes it hard to follow the discussion.
Q: Why is top posting bad?

Greg Hauptmann wrote:
| I see Watar requircs/drives a browser...i'm after something browser
| independent...any other library/plugin suggestions?

WWW::Mechanize is quite popular, from what I've seen so far.
- --
Phillip Gawlowski
Twitter: twitter.com/cynicalryan

~ - You know you've been hacking too long when...
...you discover that you're balancing your checkbook in octal.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkgT8P4ACgkQbtAgaoJTgL+AogCggpoBDGb4G+ FJwdFWR5y0HCUd
OcEAoJAt9GtYI6/yawF2INPbm5mt8lum
=vQTZ
-----END PGP SIGNATURE-----

 
Reply With Quote
 
7stud --
Guest
Posts: n/a
 
      04-27-2008
Greg Hauptmann wrote:
> Hi,
>
> Actually can anyone recommend a good technique / software / plugin
> that would assist if I wanted to effectively (a) record my interaction
> with my bank at the HTTP level, then (b) use this to automate that
> behavior in my RoR application to automate pulling down daily account
> details?
>
> The best I can think of at the moment is: (a) Firefox Live HTTP
> Headers plugin then (b) manually write Ruby code that sends these out
> and waits for the response & check it before proceeding to the next
> http request. I'm thinking someone probably has a better way, or
> plugin, to handle at least part (b)?
>


I'm not sure what the Firefox Live HTTP headers plugin will do for you.
If you write a ruby program to send out requests to a url, then you know
what headers you are sending in your request, and when you get the
response, you can read the headers in the response.


--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Peter Szinek
Guest
Posts: n/a
 
      04-28-2008
> PS. 4th question Peter I forgot:
> =95 does it support downloading a file (eg csv file, account =20
> transactions)
>
> On 4/27/08, Greg Hauptmann <(E-Mail Removed)> wrote:
>> sounds good, it support:
>> =95 https?
>> =95 cookies?
>> =95 building in some intelligence? (say when the link for step N will
>> change over time but you can write an algorithm for it)
>>


Yes, scRUBYt! supports all these things... In the current =20
implementation WWW::Mechanize is used as the agent, but it doesn't =20
support JavaScript and (more often than not) e-banking sites have =20
some JS... so that's why I suggested the FireWatir based solution.

A browser-agnostic solution doesn't exist (Mechanize is a browser too) =20=

- the nature of the task requires a browser. Call it as you like, but =20=

if something is able to GET/POST requests, store cookies, use https, =20
sessions, .... then it is a browser in my vocabulary.

Besides FireWatir is platform-independent (unlike Watir which is win32 =20=

only).

Cheers,
Peter=

 
Reply With Quote
 
Greg Hauptmann
Guest
Posts: n/a
 
      04-28-2008
thanks Peter - I was starting to look at Mechanize but will focus in
at scRUBYt...
2008/4/28 Peter Szinek <(E-Mail Removed)>:

> PS. 4th question Peter I forgot:
> > =95 does it support downloading a file (eg csv file, account transactio=

ns)
> >
> > On 4/27/08, Greg Hauptmann <(E-Mail Removed)> wrote:
> >
> > > sounds good, it support:
> > > =95 https?
> > > =95 cookies?
> > > =95 building in some intelligence? (say when the link for step N will
> > > change over time but you can write an algorithm for it)
> > >
> > >

> Yes, scRUBYt! supports all these things... In the current implementation
> WWW::Mechanize is used as the agent, but it doesn't support JavaScript an=

d
> (more often than not) e-banking sites have some JS... so that's why I
> suggested the FireWatir based solution.
>
> A browser-agnostic solution doesn't exist (Mechanize is a browser too) -
> the nature of the task requires a browser. Call it as you like, but if
> something is able to GET/POST requests, store cookies, use https, session=

s,
> .... then it is a browser in my vocabulary.
>
> Besides FireWatir is platform-independent (unlike Watir which is win32
> only).
>
> Cheers,
> Peter
>


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Refresh my memory:Interactions between separate IE instances? B. Chernick ASP .Net 3 01-25-2008 01:53 PM
Filemaker interactions Ian Witham Python 3 08-25-2007 08:35 PM
Database interactions don't fire from Application_Start in Global.asax? Christian Blackburn ASP .Net 7 06-16-2006 07:40 PM
Questions re. J2EE and Tomcat interactions/relations mot12345@alexandria.ucsb.edu Java 1 07-05-2005 04:49 PM
ASP.NET/IIS and MS Windows Anti-spyware Beta interactions ~~~ .NET Ed ~~~ ASP .Net 1 03-21-2005 01:09 PM



Advertisments