«

Hi,

We are arguing about the elegance/irregularity of the possibility to send
*gigs and gigs of data
*during many days
*over a single https response.
*[with tomcat5 & java 1.4.2]

Assuming the client can handle it and that the tcp/ssl layer are reliable
and won't disconnect, would you say it is:

question 1
a-against the http spec (what section______)
b-supported/covered by http spec (chunked encoding)
c-other________

question 2:
a-risky (reason:_______)
b-unusual (reason:_______)
c-safe (reason:_______)
d-common (example:_____)
e-other_________


Thanks to all in advance.

NOBODY wrote:
> Hi,
>
> We are arguing about the elegance/irregularity of the possibility to send
> *gigs and gigs of data
> *during many days
> *over a single https response.
> *[with tomcat5 & java 1.4.2]
>
> Assuming the client can handle it and that the tcp/ssl layer are reliable
> and won't disconnect, would you say it is:
>
> question 1
> a-against the http spec (what section______)
> b-supported/covered by http spec (chunked encoding)

I'd say it would be supported using chunked encoding.

> c-other________
>
> question 2:
> a-risky (reason:_______)

What happens if there IS a disconnect for some reason? Can you resume
operations where you left off?

> b-unusual (reason:_______)

Not unusual, per se. Internet routers, for example, create long-lived
TCP connections to exchange routing information. The volume may be
greater, but that's no matter.

> c-safe (reason:_______)

Dunno about safe. See question about disconnects.

> d-common (example:_____)
> e-other_________
>
>
> Thanks to all in advance.

Rogan Dawes wrote:
>> We are arguing about the elegance/irregularity of the possibility to
>> send *gigs and gigs of data
>> *during many days
>> *over a single https response.
>> *[with tomcat5 & java 1.4.2]
>>
>> Assuming the client can handle it and that the tcp/ssl layer are
>> reliable and won't disconnect, would you say it is:
>>
>> question 1
>> a-against the http spec (what section______)
>> b-supported/covered by http spec (chunked encoding)
>
>
> I'd say it would be supported using chunked encoding.

I agree. The spec does not place any limit that I know of on the total
size of a message delivered via the chunked transfer coding. It says
little or nothing about the permissible time to deliver a complete response.

>
>> c-other________
>>
>> question 2:
>> a-risky (reason:_______)
>
>
> What happens if there IS a disconnect for some reason? Can you resume
> operations where you left off?

Spot on. A service interruption probably means that you're hosed,
because as far as I can tell, there is no way to resume an HTTP transfer
in the middle. You could build a resumable protocol on top of HTTP, but
then you wouldn't be delivering the whole logical response in a single
HTTP response.

>> b-unusual (reason:_______)
>
>
> Not unusual, per se. Internet routers, for example, create long-lived
> TCP connections to exchange routing information. The volume may be
> greater, but that's no matter.

I disagree. Routers may establish long-lived connections amongst
themselves, but that's not the same thing as a continuous data stream
over a comparable period. At least it doesn't seem that way to me. The
proposed behavior is in any case assuredly unusual in the HTTP space.

--
John Bollinger

John C. Bollinger wrote:
> Rogan Dawes wrote:
>
>>> We are arguing about the elegance/irregularity of the possibility to
>>> send *gigs and gigs of data
>>> *during many days
>>> *over a single https response.
>>> *[with tomcat5 & java 1.4.2]
>>>


>>> question 2:
>>> a-risky (reason:_______)
>>
>>
>>
>> What happens if there IS a disconnect for some reason? Can you resume
>> operations where you left off?
>
>
> Spot on. A service interruption probably means that you're hosed,
> because as far as I can tell, there is no way to resume an HTTP transfer
> in the middle. You could build a resumable protocol on top of HTTP, but
> then you wouldn't be delivering the whole logical response in a single
> HTTP response.

Well, HTTP DOES have support for "partial content", for example, see the
"206 Partial Content" response status, and the "request ranges" support
defined in RFC2616 (I forget the exact header

>
>>> b-unusual (reason:_______)
>>
>>
>>
>> Not unusual, per se. Internet routers, for example, create long-lived
>> TCP connections to exchange routing information. The volume may be
>> greater, but that's no matter.
>
>
> I disagree. Routers may establish long-lived connections amongst
> themselves, but that's not the same thing as a continuous data stream
> over a comparable period. At least it doesn't seem that way to me. The
> proposed behavior is in any case assuredly unusual in the HTTP space.
>

Essentially, I think that the OP is wanting to do something that
requires lots of data to be transferred, and wants to do it without
getting the firewall administrators involved. Hence the desire to do
this on top of HTTP . . .

Not necessarily a good idea, IMO.

Rogan

NOBODY wrote:

> We are arguing about the elegance/irregularity of the possibility to send
> *gigs and gigs of data
> *during many days
> *over a single https response.
> *[with tomcat5 & java 1.4.2]

I would worry about this. Specifically, I'd worry about 32-bit counters.

At the TCP level there /shouldn't/ be a problem. TCP does have a 32-bit
sequence number in it, but that is defined to wrap around. What /might/ be a
worry is any other code that is in the transmission path, anti-virus software,
anti-intrusion software, statefull firewalls, etc. The point is that although
these things /should/ handle that much data, there's always a chance that
they'll contain 32-bit counters that will cause problems if they wrap. I,
personally, wouldn't worry /too/ much about this since I would expect the
makers/writers of such tools to test long flows.

I have no idea how a multi-gigabyte flow would interact with the SSL protocol.
Nor what the security/crypto implications are (would you be effectively
supplying an attacker with a very large body of data all encrypted with the
same key ? I believe SSL has provision for negotiating a new key during a
session, but I don't know when/how/whether that happens).

At the HTTP level, there /shouldn't/ be a problem either (assuming chunked
transmission). This time, though, I wouldn't feel nearly as confident that
there would be no problems with 32-bit wraparound. Neither HTTP clients nor
HTTP servers are designed /specifically/ for that kind of load, and I wouldn't
be surprised if such conditions had received little or no testing (or, perhaps,
even thought). The same considerations apply to any other s/w that is working
at the HTTP level and which is involved in the execution and/or transmission
path.

-- chris

John C. Bollinger wrote:

> Spot on. A service interruption probably means that you're hosed,
> because as far as I can tell, there is no way to resume an HTTP transfer
> in the middle.

I thought there was some sort of request-range: header in HTTP 1.1 ? I admit
that I'm too lazy to go look it up, so that may be complete *******s...

But even if it did exist, the question would arise: will the actual
implementations work with ranges that can't be expressed in 32-bits ?

-- chris

> Essentially, I think that the OP is wanting to do something that
> requires lots of data to be transferred, and wants to do it without
> getting the firewall administrators involved. Hence the desire to do
> this on top of HTTP . . .
>
> Not necessarily a good idea, IMO.
>
> Rogan


Hehehe...
As a coder, I would go straight to TCP.
But yeah... I'm JACWABB (just a coder with a boss behind...)

It's more like streaming data to simple clients, like applets, perl, wget
to console, etc... Any easy-to-code data consumer.

You're thinking: "but why don't you make many shorter connections?"
Well, for now, we would really like not to 'skip a beat' because the
streamed data in question is a flow of events. Having a 'mark' in the
protocol (to resume at requested position) would suddenly mean extra
bufferring and connection subscription/management code and ...
aaaahhhhAAHH...(sigh...) while a non-expected loss of connection is not
going to require lossless recovery. We would just tell the user, and he
would restart.

"Chris Uppal" <> wrote in
news:4374cb3d$0$38037$:

> NOBODY wrote:
>
>> We are arguing about the elegance/irregularity of the possibility to
>> send *gigs and gigs of data
>> *during many days
>> *over a single https response.
>> *[with tomcat5 & java 1.4.2]
>
> I would worry about this. Specifically, I'd worry about 32-bit
> counters.
>
> At the TCP level there /shouldn't/ be a problem. TCP does have a
> 32-bit sequence number in it, but that is defined to wrap around.
> What /might/ be a worry is any other code that is in the transmission
> path, anti-virus software, anti-intrusion software, statefull
> firewalls, etc. The point is that although these things /should/
> handle that much data, there's always a chance that they'll contain
> 32-bit counters that will cause problems if they wrap. I, personally,
> wouldn't worry /too/ much about this since I would expect the
> makers/writers of such tools to test long flows.
>
> I have no idea how a multi-gigabyte flow would interact with the SSL
> protocol. Nor what the security/crypto implications are (would you be
> effectively supplying an attacker with a very large body of data all
> encrypted with the same key ? I believe SSL has provision for
> negotiating a new key during a session, but I don't know
> when/how/whether that happens).
>
> At the HTTP level, there /shouldn't/ be a problem either (assuming
> chunked transmission). This time, though, I wouldn't feel nearly as
> confident that there would be no problems with 32-bit wraparound.
> Neither HTTP clients nor HTTP servers are designed /specifically/ for
> that kind of load, and I wouldn't be surprised if such conditions had
> received little or no testing (or, perhaps, even thought). The same
> considerations apply to any other s/w that is working at the HTTP
> level and which is involved in the execution and/or transmission path.
>
> -- chris

Thanks for the details.

I keep my eye on tomcat: will the httpsession stay alive during that
time, and if not, will the socket be forcibly closed... I think not, but
eh, dunno. For sure, the CLF web log will look funny...!
ex: 200 ok, 293'465'023 milliseconds, 19'535'487'623 bytes...!

Chris Uppal wrote:
> John C. Bollinger wrote:
>
>
>>Spot on. A service interruption probably means that you're hosed,
>>because as far as I can tell, there is no way to resume an HTTP transfer
>>in the middle.
>
>
> I thought there was some sort of request-range: header in HTTP 1.1 ? I admit
> that I'm too lazy to go look it up, so that may be complete *******s...

Ah, um, yes; good point. Time to go back and reread the spec again.
Request ranges might not really help, however, if the response is
dynamically generated. That is especially true if it is a real-time
stream. (E.g. data from a scientific instrument.)


--
John Bollinger

NOBODY wrote:

>. For sure, the CLF web log will look funny...!
> ex: 200 ok, 293'465'023 milliseconds, 19'535'487'623 bytes...!

And if it wraps to negative numbers, then your logfile analyser will have
kittens

-- chris