Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > PLEASE HELP ABOUT HASH!

Reply
Thread Tools

PLEASE HELP ABOUT HASH!

 
 
news.hinet.net
Guest
Posts: n/a
 
      03-02-2004
I have the vast data in mysql about (1.5G)
I got some data from machine and want to search some field
that does not exist in mysql. i just want to insert these data that
does not exist in mysql before.

I use hash to handle these data.

First i load data into hash from mysql and search data if it is exist.
but i use the vast memory at the same time

how to solve this program??

if i insert all data by setup db field data is unique. this will spend a lot
of time.


give me some suggest!!
ps:the data is more than 500M one time insert process



 
Reply With Quote
 
 
 
 
James Willmore
Guest
Posts: n/a
 
      03-02-2004
On Tue, 02 Mar 2004 10:57:54 +0800, news.hinet.net wrote:

> I have the vast data in mysql about (1.5G)
> I got some data from machine and want to search some field
> that does not exist in mysql. i just want to insert these data that
> does not exist in mysql before.
>
> I use hash to handle these data.
>
> First i load data into hash from mysql and search data if it is exist.
> but i use the vast memory at the same time
>
> how to solve this program??
>
> if i insert all data by setup db field data is unique. this will spend a lot
> of time.
>
>
> give me some suggest!!
> ps:the data is more than 500M one time insert process


This is all very interesting, but where's the code you use at present?
That *may* shed some light on this most wonderful issue.

Right now, the only thing I could suggest is write to a file instead of a
hash and then parse the file - one line at a time.

OTOH, maybe you need to use SQL more efficently.

Don't know without the code.

--
Jim

Copyright notice: all code written by the author in this post is
released under the GPL. http://www.gnu.org/licenses/gpl.txt
for more information.

a fortune quote ...
Pascal, n.: A programming language named after a man who would
<turn over in his grave if he knew about it.

 
Reply With Quote
 
 
 
 
ctcgag@hotmail.com
Guest
Posts: n/a
 
      03-02-2004
"news.hinet.net" <(E-Mail Removed)> wrote:
> I have the vast data in mysql about (1.5G)
> I got some data from machine and want to search some field
> that does not exist in mysql.


I think you mean that the field exists, and you wish to identify values for
that field which do not exist.

> i just want to insert these data that
> does not exist in mysql before.


Is the "insert ignore..." MySQL-specific extension to sql what you want?

> I use hash to handle these data.
>
> First i load data into hash from mysql and search data if it is exist.
> but i use the vast memory at the same time
>
> how to solve this program??


Huge hashes take huge amounts of memory. There is no ready solution to
that problem.

>
> if i insert all data by setup db field data is unique. this will spend a
> lot of time.


Please explain in code what you mean by this, and specify in numbers the
amount of time it would take (and in what an acceptable time would be).

> give me some suggest!!
> ps:the data is more than 500M one time insert process


It is unlikely that the programming, debugging, and run-time of a
sophisticated one-time only solution will be less than the run-time of the
naive "insert ignore..." solution.



Xho

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Please please please help this guy with his open source java app casioculture@gmail.com Java 4 05-05-2005 08:24 AM
Console profile for Windows app in VC++ - PLEASE PLEASE PLEASE HELP! MuZZy C++ 7 01-07-2005 08:40 PM
Computer problems please please please help Nick Computer Support 0 06-04-2004 08:49 PM
HELP! HELP! PLEASE, PLEASE, PLEASE tpg comcntr Computer Support 11 02-15-2004 06:22 PM
please help... ...me learn C++ please please please :) KK C++ 2 10-14-2003 02:08 PM



Advertisments