Poor performance with large datasets
I have noticed that performance is very poor when a web service
returns a large dataset.
Further testing has shown that the XML version of the dataset is
approximately 1.3GB in size - about 10x the binary representation of
the data. It is the transfer of this data over the network that is
causing the performance problem.
We are pulling a large result set into an Excel application for pivot
Connecting directly to the database using ADO takes only about 30
seconds to finish retrieving all the data - presumably because the
data is coming over efficiently in binary format.
We also looked into using a 3rd party product to compress the XML
dataset before returning it from the web service, but that too is
How is Microsoft recommending that applications deal with large
datasets? How are other people dealing with this issue (if at all)? I
realize that .NET 2.0 is supposed to support binary data over web
services, but this needs to be a 1.1 solution.
Re: Poor performance with large datasets
> We also looked into using a 3rd party product to compress the XML
> dataset before returning it from the web service, but that too is
> prohibitively slow.
check www.remobjects.com .. .NET SDK 3 is still in beta but you can get
beta access if you ask nicely :)
their binary format supports compression + encryption and is really fast
I've used RO SDK for over a year (with delphi) and have no complaints .. :)
current RO SDK 2 is only a client version and only supports their binary
contact them and I'm sure you can arrange something :)
Re: Poor performance with large datasets
Sorry, this is not an answer.. but
I too am facing the same problem.. Having tried various techniques, asp.net can't handle huge Datasets..It simply recycles... Now how Can I make my application work? I can't tell my people to buy .Net 2.0...
My DataSets will generally contain 100,000 records with relations... Its killing ...
I tried storing data in Session, Cache, other "Compact" ing techniques.. No use!!
Can any expert help pleaseeeee......
Sent via Fuzzy Software @ http://www.fuzzysoftware.com/
Comprehensive, categorised, searchable collection of links to ASP & ASP.NET resources...
RE: Poor performance with large datasets
I think the crux of the issue comes down to why are you deciding to
implement this large-data-set requirement on a Web Service stack? When I
think thru all of the distributed computing (WS is distributed computing)
wisdom that I've encountered over the years, one of the repeating themes is
"the farther away you are, the smaller your data transmission requirements
should be". In a web service world, requests are likely to be more
serviceable if you keep the payloads of the requests and responses small.
If you find yourself needing to send large binary payloads, perhaps the
answer is to send large binary payloads. You can use remoting, save to
file and use things like FTP, or open direct streams for these kind of
"batch transfers". With web services, if you are using the WS interface to
transfer what amounts to reference copies of data, some might suggest that
a different approach to managing the publication of large reference data
sets is called for.
Compression to binary will certainly reduce the number of packets required,
and thus effect your transfer speed by a factor of the compression ratio.
But compression adds several complicating factors. An easy way to get mass
compression is to use some of the hardware solutions available at both ends
of predictable high latency connections (I say high latency because of the
end to end response time). In a controlled corporate network setting, or
high volume commercial web infrastructures, this is economically feasible
and eliminates the processing intensive and coding intensive overhead of
both sides having to know some new compression protocol.
Products from Netscalar, Cisco and F5 all can do what you want without
adding any coding overhead to your network. Another option is faster
network connections as well (gigabit ethernet, etc). I know these all may
seem daunting to the project manager/budget minded, but they are a real
part of the overall solution you should consider before you add in a
complicated software fix. One real advantage is to the budget - these are
typically capital budget items and not R&D expenses, so that added cost can
be amoritized more rapidly.
Hardware solutions to this issue also has the added benefit of degrading
gracefully. Requests from clients that are outside of your corporate
network (and thus who cannot make the efficient direct binary calls) can
still come in thru an HTTP security gateway with a normal web service
request. Yes, they will see large files, and the requisite effect of
aggregate bandwidth and latency, but it'll work without having to code a
special version of the application.
I hope this adds some aspects to your trade-off matrix,
>From: firstname.lastname@example.org (Luther Miller)
>Newsgroups: microsoft.public.dotnet.framework.aspnet.webservic es
>Subject: Poor performance with large datasets
>Date: 1 Oct 2004 09:50:01 -0700
>Content-Type: text/plain; charset=ISO-8859-1
>X-Trace: posting.google.com 1096649401 13991 127.0.0.1 (1 Oct 2004
>NNTP-Posting-Date: Fri, 1 Oct 2004 16:50:01 +0000 (UTC)
>X-Tomcat-NG: microsoft.public.dotnet.framework.aspnet.webservic es
>I have noticed that performance is very poor when a web service
>returns a large dataset.
>Further testing has shown that the XML version of the dataset is
>approximately 1.3GB in size - about 10x the binary representation of
>the data. It is the transfer of this data over the network that is
>causing the performance problem.
>We are pulling a large result set into an Excel application for pivot
>Connecting directly to the database using ADO takes only about 30
>seconds to finish retrieving all the data - presumably because the
>data is coming over efficiently in binary format.
>We also looked into using a 3rd party product to compress the XML
>dataset before returning it from the web service, but that too is
>How is Microsoft recommending that applications deal with large
>datasets? How are other people dealing with this issue (if at all)? I
>realize that .NET 2.0 is supposed to support binary data over web
>services, but this needs to be a 1.1 solution.
|All times are GMT. The time now is 11:14 AM.|
Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.