PLEASE HELP ABOUT HASH!
 

I have the vast data in mysql about (1.5G)
I got some data from machine and want to search some field
that does not exist in mysql. i just want to insert these data that
does not exist in mysql before.

I use hash to handle these data.

First i load data into hash from mysql and search data if it is exist.
but i use the vast memory at the same time

how to solve this program??

if i insert all data by setup db field data is unique. this will spend a lot
of time.


give me some suggest!!
ps:the data is more than 500M one time insert process

Re: PLEASE HELP ABOUT HASH!
 

On Tue, 02 Mar 2004 10:57:54 +0800, news.hinet.net wrote:

> I have the vast data in mysql about (1.5G)
> I got some data from machine and want to search some field
> that does not exist in mysql. i just want to insert these data that
> does not exist in mysql before.
>
> I use hash to handle these data.
>
> First i load data into hash from mysql and search data if it is exist.
> but i use the vast memory at the same time
>
> how to solve this program??
>
> if i insert all data by setup db field data is unique. this will spend a lot
> of time.
>
>
> give me some suggest!!
> ps:the data is more than 500M one time insert process

This is all very interesting, but where's the code you use at present?
That *may* shed some light on this most wonderful issue.

Right now, the only thing I could suggest is write to a file instead of a
hash and then parse the file - one line at a time.

OTOH, maybe you need to use SQL more efficently.

Don't know without the code.

--
Jim

Copyright notice: all code written by the author in this post is
released under the GPL. http://www.gnu.org/licenses/gpl.txt
for more information.

a fortune quote ...
Pascal, n.: A programming language named after a man who would
<turn over in his grave if he knew about it.

Re: PLEASE HELP ABOUT HASH!
 

"news.hinet.net" <luke@program.com.tw> wrote:
> I have the vast data in mysql about (1.5G)
> I got some data from machine and want to search some field
> that does not exist in mysql.

I think you mean that the field exists, and you wish to identify values for
that field which do not exist.

> i just want to insert these data that
> does not exist in mysql before.

Is the "insert ignore..." MySQL-specific extension to sql what you want?

> I use hash to handle these data.
>
> First i load data into hash from mysql and search data if it is exist.
> but i use the vast memory at the same time
>
> how to solve this program??

Huge hashes take huge amounts of memory. There is no ready solution to
that problem.

>
> if i insert all data by setup db field data is unique. this will spend a
> lot of time.

Please explain in code what you mean by this, and specify in numbers the
amount of time it would take (and in what an acceptable time would be).

> give me some suggest!!
> ps:the data is more than 500M one time insert process

It is unlikely that the programming, debugging, and run-time of a
sophisticated one-time only solution will be less than the run-time of the
naive "insert ignore..." solution.



Xho

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB