Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Speeding up URLConnection

Reply
Thread Tools

Speeding up URLConnection

 
 
mark13.pl@gmail.com
Guest
Posts: n/a
 
      08-30-2006
Hello,

I want to save as a string html file. My code looks like that:

URL url = new URL(fileName);
URLConnection conn = url.openConnection();
conn.setRequestProperty("Cookie", myCookieCode);
conn.connect();
BufferedReader dis = new BufferedReader(new
InputStreamReader(conn.getInputStream()));
String inputLine = null;
for(; {
inputLine = dis.readLine();
htmlCode += inputLine;
if (inputLine == null) break;
}

It works perfectly (in htmlCode I have the whole page as I wanted) but
it has very big disadvantage - it is VERY slow. In both my browsers
(Firefox, IE6.0, cache cleaned) it takes about 2seconds to load it
while in java: about 14seconds. Do you know where is a problem and how
can I speed it up??

Regards, mark

 
Reply With Quote
 
 
 
 
Oliver Wong
Guest
Posts: n/a
 
      08-30-2006

<(E-Mail Removed)> wrote in message
news:(E-Mail Removed) ps.com...
> Hello,
>
> I want to save as a string html file. My code looks like that:
>
> URL url = new URL(fileName);
> URLConnection conn = url.openConnection();
> conn.setRequestProperty("Cookie", myCookieCode);
> conn.connect();
> BufferedReader dis = new BufferedReader(new
> InputStreamReader(conn.getInputStream()));
> String inputLine = null;
> for(; {
> inputLine = dis.readLine();
> htmlCode += inputLine;
> if (inputLine == null) break;
> }
>
> It works perfectly (in htmlCode I have the whole page as I wanted) but
> it has very big disadvantage - it is VERY slow. In both my browsers
> (Firefox, IE6.0, cache cleaned) it takes about 2seconds to load it
> while in java: about 14seconds. Do you know where is a problem and how
> can I speed it up??


If you have a profiler, you should use it to measure where your code is
spending all your time. However, from a quick glance, my guess is that the
slowestp art is string concatenation. Use a StringBuilder instead:

URL url = new URL(fileName);
URLConnection conn = url.openConnection();
conn.setRequestProperty("Cookie", myCookieCode);
conn.connect();
BufferedReader dis = new BufferedReader(new
InputStreamReader(conn.getInputStream()));
StringBuilder tempHtmlCode = new StringBuilder();
do {
String inputLine = dis.readLine();
tempHtmlCode.append(inputLine);
} while (inputLine != null)
htmlCode = tempHtmlCode.toString();

- Oliver

 
Reply With Quote
 
 
 
 
Chris Uppal
Guest
Posts: n/a
 
      08-30-2006
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

> BufferedReader dis = new BufferedReader(new
> InputStreamReader(conn.getInputStream()));


In general the buffering should be immediately around the lowest-level stream
(the one that does real IO). There may be circumstances where you need
buffering at outer levels too, but that's not so common.

Reader reader = new InputStreamReader(
new BufferedInputStream(
conn.getInputStream()));

BTW, /don't/ forget about the character encoding -- in this context it is very
unlikely to be an issue you can safely ignore.

-- chris


 
Reply With Quote
 
Babu Kalakrishnan
Guest
Posts: n/a
 
      08-30-2006

(E-Mail Removed) wrote:
> Hello,
>
> I want to save as a string html file. My code looks like that:
>
> URL url = new URL(fileName);
> URLConnection conn = url.openConnection();
> conn.setRequestProperty("Cookie", myCookieCode);
> conn.connect();
> BufferedReader dis = new BufferedReader(new
> InputStreamReader(conn.getInputStream()));
> String inputLine = null;
> for(; {
> inputLine = dis.readLine();
> htmlCode += inputLine;
> if (inputLine == null) break;
> }
>
> It works perfectly (in htmlCode I have the whole page as I wanted) but
> it has very big disadvantage - it is VERY slow. In both my browsers
> (Firefox, IE6.0, cache cleaned) it takes about 2seconds to load it
> while in java: about 14seconds. Do you know where is a problem and how
> can I speed it up??
>


While I agree with Oliver that your String concatenation code is pretty
inefficient, I wouldn't think it could cause that kinds of delays
(unless the number of lines was very very large).

One probable cause ( a guess really) is that the HttpUrlConnection
class tries to keep the connection open in anticipation of futher
traffic to the same URL, and so the reader returning null (indicating
the end of output) is delayed till the connection is eventually
dropped. Browsers on the other hand would keep the connection open, but
would look at headers to see the length of the expected response, and
terminate the read when the expecteed number of bytes were received.

I suggest that you try out another client (like say jakarta's
HttpClient) , and see if it helps. Sun's implementation had quite afew
issues like this in the past - and was really never meant to be
full-fledged HttpClient - but rather a simplistic mechanism for loading
Jar files from a remote source.

BK

 
Reply With Quote
 
mark13.pl@gmail.com
Guest
Posts: n/a
 
      08-30-2006
Hello,

Thank you for your answers. I have changed it to StringBuilder and now
everything speeds up a few times (it is perfectly acceptable now so I
will stop here without making additional experiments with other http
readers) . Thank you very much!

Regards, mark

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
speeding up URLConnection reading mark Java 10 11-13-2006 08:26 AM
OutputStream from a URLConnection produces an OutOfMemory OutputStream from a URLConnection produces an OutOfMemory WinstonSmith_101@hotmail.com Java 2 10-25-2006 04:45 PM
Problems encoding in utf8 with urlconnection response Bronwyn Howard Java 0 10-01-2003 06:10 AM
URLConnection to WebService gets FileNotFoundException Brock Jones Java 0 09-04-2003 12:10 AM
URLConnection and proxy Kristoffel Java 0 07-27-2003 08:27 PM



Advertisments