Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Java heap and big database queries

Reply
Thread Tools

Java heap and big database queries

 
 
Mark Thornton
Guest
Posts: n/a
 
      02-16-2007
Mike Schilling wrote:
> "Mark Thornton" <(E-Mail Removed)> wrote in message
> news:MJeBh.11577$(E-Mail Removed)...
>>
>>Transmitting rows a batch at a time (say 100) avoids that problem and also
>>has only a modest heap impact.

>
>
> Yes. That MySQL doesn't batch rows is a sign of its primitiveness.
>


Indeed. However this behaviour is common to quite a lot of JDBC
driver/database combinations. Apparently it suits common enterprise
database activities. It is rather less suitable to
scientific/mathematical use of databases. In these cases you would have
to be barking mad to perform the computation in SQL, but the alternative
involves retrieving a very substantial amount of data. There are also
other issues with this kind of use, notably in locking / transactions.

SQL Server with a suitable driver will stream results. For smallish
datasets you can use SQL Server Express for free (assuming you are
running on Windows).

FireBird is another option worth considering.

Mark Thornton
 
Reply With Quote
 
 
 
 
Patricia Shanahan
Guest
Posts: n/a
 
      02-17-2007
Mark Thornton wrote:
> Mike Schilling wrote:
>> "Mark Thornton" <(E-Mail Removed)> wrote in message
>> news:MJeBh.11577$(E-Mail Removed)...
>>>
>>> Transmitting rows a batch at a time (say 100) avoids that problem and
>>> also has only a modest heap impact.

>>
>>
>> Yes. That MySQL doesn't batch rows is a sign of its primitiveness.

>
> Indeed. However this behaviour is common to quite a lot of JDBC
> driver/database combinations. Apparently it suits common enterprise
> database activities. It is rather less suitable to
> scientific/mathematical use of databases. In these cases you would have
> to be barking mad to perform the computation in SQL, but the alternative
> involves retrieving a very substantial amount of data. There are also
> other issues with this kind of use, notably in locking / transactions.


Ultimately, I will be using this data e.g. as training data in my Ph.D.
research project, so I do feel more comfortable knowing how to get the
data to my Java program.

>
> SQL Server with a suitable driver will stream results. For smallish
> datasets you can use SQL Server Express for free (assuming you are
> running on Windows).
>
> FireBird is another option worth considering.
>
> Mark Thornton


SQL Server Express is limited to 4 GB total database size, and one of my
tables is about 10 GB, so it is not suitable for my particular situation.

Meanwhile, I've moved my database to PostgreSQL. I had some initial
performance problems with a large query, but I was able to cure it by
increasing the sort buffer size.

It streams data to my Java program, so I no longer run out of memory.

Thanks to everyone for all the advice.

Patricia
 
Reply With Quote
 
 
 
 
=?ISO-8859-1?Q?Arne_Vajh=F8j?=
Guest
Posts: n/a
 
      02-19-2007
David Segall wrote:
> Arne, I want you to like it! If I can achieve this by adding some
> brief, relevant information to the web page I would be pleased to do
> so and would appreciate it if you would post the information here or
> email it to david at segall dot net. I am reluctant to include any
> detailed licensing information because each of the products listed
> comes with several pages of legal restrictions that I do not wish to
> maintain.
>
> I added the licensing qualifications to the MySQL entry in my list
> because I was concerned by a previous post from Chris Smith that
> contained the information that he has posted in this thread. Because
> MySQL is so popular and many people, encouraged by MySQL AB, consider
> it to be freely distributable I chose to qualify the entry in my list
> rather than simply delete it.


Maybe something like:

You should be aware that the MySQL ODBC, JDBC and ADO.NET drivers
is only freely available under GPL license (not LGPL license) with
a FLOSS exception. The GPL license states that code "linked" to
GPL code must be GPL code itself. If your code is GPL or other
open source, then you do not have any problems. If your code
is closed source, then you should seek legal advice before
using those drivers.

Arne
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
GIDS 2009 .Net:: Save Big, Win Big, Learn Big: Act Before Dec 29 2008 Shaguf ASP .Net 0 12-26-2008 09:29 AM
GIDS 2009 .Net:: Save Big, Win Big, Learn Big: Act Before Dec 29 2008 Shaguf ASP .Net Web Controls 0 12-26-2008 06:11 AM
GIDS 2009 Java:: Save Big, Win Big, Learn Big: Act Before Dec 29 2008 Shaguf Python 0 12-24-2008 07:35 AM
GIDS 2009 Java:: Save Big, Win Big, Learn Big: Act Before Dec 29 2008 Shaguf Ruby 0 12-24-2008 05:07 AM
xslt queries in xml to SQL queries Ian Roddis Python 3 02-26-2006 06:49 PM



Advertisments