Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > java based supercomputer

Reply
Thread Tools

java based supercomputer

 
 
sav
Guest
Posts: n/a
 
      02-14-2007
I am trying an ambitous project of trying to adapt a piece of code
that i came accross on an IBM website. It uses psuedo remote threads
and is a way of utilising RMI, multithreading and will develop a java
based supercomuting technique. I wish to use a similar method for
checking the correlation beteween an array of data and another array
comprised of combining 8 other arrays of data, so by using a
straightforward for next loop will take forever but by utilising this
java psuedo remote threads will take a considerrably less time.
Howeve, i am relatively new to java but i learn fast. I need someont
to help me achieve this, i am willing to pay. I hacve the code
available for review

 
Reply With Quote
 
 
 
 
Daniel Pitts
Guest
Posts: n/a
 
      02-14-2007
On Feb 14, 1:56 pm, "sav" <(E-Mail Removed)> wrote:
> I am trying an ambitous project of trying to adapt a piece of code
> that i came accross on an IBM website. It uses psuedo remote threads
> and is a way of utilising RMI, multithreading and will develop a java
> based supercomuting technique. I wish to use a similar method for
> checking the correlation beteween an array of data and another array
> comprised of combining 8 other arrays of data, so by using a
> straightforward for next loop will take forever but by utilising this
> java psuedo remote threads will take a considerrably less time.
> Howeve, i am relatively new to java but i learn fast. I need someont
> to help me achieve this, i am willing to pay. I hacve the code
> available for review


This first question, which isn't even Java specific, is:
Does your algorithm lend itself well to paralellization? You could
find out by running it with normal Thread's on a multiprocessor
system. If this improves performance, then you can proceed to the next
questions.

The second question, still not Java specific:
When you split the computation between N threads, how much
communication is there between those threads? If the communication
between threads has higher latency or limited bandwidth, how does that
effect your algorithms throughput?

Java specific question: Have you profiled your existing "slow" program
to see what is taking the most time? It might not be what you
expect.

Also, have you considered finding a less naive algorithm? Without
more details about your goal I can only speculate, but it might be
that your approach is an O(N^3) algorithm, when in fact there may
exist an O(n*logn) algorithm.

So, before trying to cross machine boundaries, or even thread
boundaries, it might be worth looking into other algorithms that speed
up your process.

 
Reply With Quote
 
 
 
 
sav
Guest
Posts: n/a
 
      02-15-2007
On 14 Feb, 22:24, "Daniel Pitts" <(E-Mail Removed)> wrote:
> On Feb 14, 1:56 pm, "sav" <(E-Mail Removed)> wrote:
>
> > I am trying an ambitous project of trying to adapt a piece of code
> > that i came accross on an IBM website. It uses psuedo remote threads
> > and is a way of utilising RMI, multithreading and will develop a java
> > based supercomuting technique. I wish to use a similar method for
> > checking the correlation beteween an array of data and another array
> > comprised of combining 8 other arrays of data, so by using a
> > straightforward for next loop will take forever but by utilising this
> > java psuedo remote threads will take a considerrably less time.
> > Howeve, i am relatively new to java but i learn fast. I need someont
> > to help me achieve this, i am willing to pay. I hacve the code
> > available for review

>
> This first question, which isn't even Java specific, is:
> Does your algorithm lend itself well to paralellization? You could
> find out by running it with normal Thread's on a multiprocessor
> system. If this improves performance, then you can proceed to the next
> questions.
>
> The second question, still not Java specific:
> When you split the computation between N threads, how much
> communication is there between those threads? If the communication
> between threads has higher latency or limited bandwidth, how does that
> effect your algorithms throughput?
>
> Java specific question: Have you profiled your existing "slow" program
> to see what is taking the most time? It might not be what you
> expect.
>
> Also, have you considered finding a less naive algorithm? Without
> more details about your goal I can only speculate, but it might be
> that your approach is an O(N^3) algorithm, when in fact there may
> exist an O(n*logn) algorithm.
>
> So, before trying to cross machine boundaries, or even thread
> boundaries, it might be worth looking into other algorithms that speed
> up your process.


Hi i did send you a full email to your email address did you get it?

 
Reply With Quote
 
Daniel Pitts
Guest
Posts: n/a
 
      02-15-2007
On Feb 15, 11:16 am, "sav" <(E-Mail Removed)> wrote:
> On 14 Feb, 22:24, "Daniel Pitts" <(E-Mail Removed)> wrote:
>
>
>
> > On Feb 14, 1:56 pm, "sav" <(E-Mail Removed)> wrote:

>
> > > I am trying an ambitous project of trying to adapt a piece of code
> > > that i came accross on an IBM website. It uses psuedo remote threads
> > > and is a way of utilising RMI, multithreading and will develop a java
> > > based supercomuting technique. I wish to use a similar method for
> > > checking the correlation beteween an array of data and another array
> > > comprised of combining 8 other arrays of data, so by using a
> > > straightforward for next loop will take forever but by utilising this
> > > java psuedo remote threads will take a considerrably less time.
> > > Howeve, i am relatively new to java but i learn fast. I need someont
> > > to help me achieve this, i am willing to pay. I hacve the code
> > > available for review

>
> > This first question, which isn't even Java specific, is:
> > Does your algorithm lend itself well to paralellization? You could
> > find out by running it with normal Thread's on a multiprocessor
> > system. If this improves performance, then you can proceed to the next
> > questions.

>
> > The second question, still not Java specific:
> > When you split the computation between N threads, how much
> > communication is there between those threads? If the communication
> > between threads has higher latency or limited bandwidth, how does that
> > effect your algorithms throughput?

>
> > Java specific question: Have you profiled your existing "slow" program
> > to see what is taking the most time? It might not be what you
> > expect.

>
> > Also, have you considered finding a less naive algorithm? Without
> > more details about your goal I can only speculate, but it might be
> > that your approach is an O(N^3) algorithm, when in fact there may
> > exist an O(n*logn) algorithm.

>
> > So, before trying to cross machine boundaries, or even thread
> > boundaries, it might be worth looking into other algorithms that speed
> > up your process.

>
> Hi i did send you a full email to your email address did you get it?


No. Three reasons. The first is that my home computer has been
disabled for a few weeks.
The second reason is that the given e-mail address gets so much spam
that I usually just delete it with out looking.
The third reason is that I didn't invite you to e-mail me, please keep
the discussion here unless invited to otherwise.

As far as I know, no one here gets paid to answer questions on this
newsgroup. On the other hand, many of us DO get paid to answer
questions one-on-one. If you want free advice, post here; if you want
to pay for solutions, find a contractor.

 
Reply With Quote
 
sav
Guest
Posts: n/a
 
      02-15-2007
On 15 Feb, 20:39, "Daniel Pitts" <(E-Mail Removed)> wrote:
> On Feb 15, 11:16 am, "sav" <(E-Mail Removed)> wrote:
>
>
>
> > On 14 Feb, 22:24, "Daniel Pitts" <(E-Mail Removed)> wrote:

>
> > > On Feb 14, 1:56 pm, "sav" <(E-Mail Removed)> wrote:

>
> > > > I am trying an ambitous project of trying to adapt a piece of code
> > > > that i came accross on an IBM website. It uses psuedo remote threads
> > > > and is a way of utilising RMI, multithreading and will develop a java
> > > > based supercomuting technique. I wish to use a similar method for
> > > > checking the correlation beteween an array of data and another array
> > > > comprised of combining 8 other arrays of data, so by using a
> > > > straightforward for next loop will take forever but by utilising this
> > > > java psuedo remote threads will take a considerrably less time.
> > > > Howeve, i am relatively new to java but i learn fast. I need someont
> > > > to help me achieve this, i am willing to pay. I hacve the code
> > > > available for review

>
> > > This first question, which isn't even Java specific, is:
> > > Does your algorithm lend itself well to paralellization? You could
> > > find out by running it with normal Thread's on a multiprocessor
> > > system. If this improves performance, then you can proceed to the next
> > > questions.

>
> > > The second question, still not Java specific:
> > > When you split the computation between N threads, how much
> > > communication is there between those threads? If the communication
> > > between threads has higher latency or limited bandwidth, how does that
> > > effect your algorithms throughput?

>
> > > Java specific question: Have you profiled your existing "slow" program
> > > to see what is taking the most time? It might not be what you
> > > expect.

>
> > > Also, have you considered finding a less naive algorithm? Without
> > > more details about your goal I can only speculate, but it might be
> > > that your approach is an O(N^3) algorithm, when in fact there may
> > > exist an O(n*logn) algorithm.

>
> > > So, before trying to cross machine boundaries, or even thread
> > > boundaries, it might be worth looking into other algorithms that speed
> > > up your process.

>
> > Hi i did send you a full email to your email address did you get it?

>
> No. Three reasons. The first is that my home computer has been
> disabled for a few weeks.
> The second reason is that the given e-mail address gets so much spam
> that I usually just delete it with out looking.
> The third reason is that I didn't invite you to e-mail me, please keep
> the discussion here unless invited to otherwise.
>
> As far as I know, no one here gets paid to answer questions on this
> newsgroup. On the other hand, many of us DO get paid to answer
> questions one-on-one. If you want free advice, post here; if you want
> to pay for solutions, find a contractor.


ok cool, weel first to answer your questions, i am pretty sure that
the algorithm will be quite easy to parralelise if that is the correct
term, the only bottleneck i can see is checking the correlation value
against the highest rank, ie the algorithm basically is a for next
loop BUT with billions of calcs to perform, the bottlenck will
obviously be when the calcs need to check for a higher correlation
against the current highest rank, however with the java program i have
seen it seesm to perform many of the loop functions simoultaneously,
ie i have 8 arrays of of which perform 10^3 calculations so many of
these calcs can be carried out concurrently.

The current system runs in VBA not in java so i havent run in java
yet.

Look forward to your reply


 
Reply With Quote
 
Daniel Pitts
Guest
Posts: n/a
 
      02-16-2007
On Feb 15, 12:54 pm, "sav" <(E-Mail Removed)> wrote:
> On 15 Feb, 20:39, "Daniel Pitts" <(E-Mail Removed)> wrote:
>
>
>
> > On Feb 15, 11:16 am, "sav" <(E-Mail Removed)> wrote:

>
> > > On 14 Feb, 22:24, "Daniel Pitts" <(E-Mail Removed)> wrote:

>
> > > > On Feb 14, 1:56 pm, "sav" <(E-Mail Removed)> wrote:

>
> > > > > I am trying an ambitous project of trying to adapt a piece of code
> > > > > that i came accross on an IBM website. It uses psuedo remote threads
> > > > > and is a way of utilising RMI, multithreading and will develop a java
> > > > > based supercomuting technique. I wish to use a similar method for
> > > > > checking the correlation beteween an array of data and another array
> > > > > comprised of combining 8 other arrays of data, so by using a
> > > > > straightforward for next loop will take forever but by utilising this
> > > > > java psuedo remote threads will take a considerrably less time.
> > > > > Howeve, i am relatively new to java but i learn fast. I need someont
> > > > > to help me achieve this, i am willing to pay. I hacve the code
> > > > > available for review

>
> > > > This first question, which isn't even Java specific, is:
> > > > Does your algorithm lend itself well to paralellization? You could
> > > > find out by running it with normal Thread's on a multiprocessor
> > > > system. If this improves performance, then you can proceed to the next
> > > > questions.

>
> > > > The second question, still not Java specific:
> > > > When you split the computation between N threads, how much
> > > > communication is there between those threads? If the communication
> > > > between threads has higher latency or limited bandwidth, how does that
> > > > effect your algorithms throughput?

>
> > > > Java specific question: Have you profiled your existing "slow" program
> > > > to see what is taking the most time? It might not be what you
> > > > expect.

>
> > > > Also, have you considered finding a less naive algorithm? Without
> > > > more details about your goal I can only speculate, but it might be
> > > > that your approach is an O(N^3) algorithm, when in fact there may
> > > > exist an O(n*logn) algorithm.

>
> > > > So, before trying to cross machine boundaries, or even thread
> > > > boundaries, it might be worth looking into other algorithms that speed
> > > > up your process.

>
> > > Hi i did send you a full email to your email address did you get it?

>
> > No. Three reasons. The first is that my home computer has been
> > disabled for a few weeks.
> > The second reason is that the given e-mail address gets so much spam
> > that I usually just delete it with out looking.
> > The third reason is that I didn't invite you to e-mail me, please keep
> > the discussion here unless invited to otherwise.

>
> > As far as I know, no one here gets paid to answer questions on this
> > newsgroup. On the other hand, many of us DO get paid to answer
> > questions one-on-one. If you want free advice, post here; if you want
> > to pay for solutions, find a contractor.

>
> ok cool, weel first to answer your questions, i am pretty sure that
> the algorithm will be quite easy to parralelise if that is the correct
> term, the only bottleneck i can see is checking the correlation value
> against the highest rank, ie the algorithm basically is a for next
> loop BUT with billions of calcs to perform, the bottlenck will
> obviously be when the calcs need to check for a higher correlation
> against the current highest rank, however with the java program i have
> seen it seesm to perform many of the loop functions simoultaneously,
> ie i have 8 arrays of of which perform 10^3 calculations so many of
> these calcs can be carried out concurrently.
>
> The current system runs in VBA not in java so i havent run in java
> yet.
>
> Look forward to your reply


Okay, so if I understand correctly:

You have N number of arrays to test;
Testing array A doesn't interact with the testing of array B.
Testing any array results in an easy-to-compare score.
So, you could create up to N threads, each calculates and records the
score.
All of the scores can then be easily compared to one another.

Since there are only 8 arrays (hence 8 scores), the sorting isn't too
hard to do.

How do you score array A? What do you mean by correlation? longest-
common-subsequence? number of similar/same elements? Something else?

 
Reply With Quote
 
sav
Guest
Posts: n/a
 
      02-16-2007
On 16 Feb, 00:17, "Daniel Pitts" <(E-Mail Removed)> wrote:
> On Feb 15, 12:54 pm, "sav" <(E-Mail Removed)> wrote:
>
>
>
> > On 15 Feb, 20:39, "Daniel Pitts" <(E-Mail Removed)> wrote:

>
> > > On Feb 15, 11:16 am, "sav" <(E-Mail Removed)> wrote:

>
> > > > On 14 Feb, 22:24, "Daniel Pitts" <(E-Mail Removed)> wrote:

>
> > > > > On Feb 14, 1:56 pm, "sav" <(E-Mail Removed)> wrote:

>
> > > > > > I am trying an ambitous project of trying to adapt a piece of code
> > > > > > that i came accross on an IBM website. It uses psuedo remote threads
> > > > > > and is a way of utilising RMI, multithreading and will develop a java
> > > > > > based supercomuting technique. I wish to use a similar method for
> > > > > > checking the correlation beteween an array of data and another array
> > > > > > comprised of combining 8 other arrays of data, so by using a
> > > > > > straightforward for next loop will take forever but by utilising this
> > > > > > java psuedo remote threads will take a considerrably less time.
> > > > > > Howeve, i am relatively new to java but i learn fast. I need someont
> > > > > > to help me achieve this, i am willing to pay. I hacve the code
> > > > > > available for review

>
> > > > > This first question, which isn't even Java specific, is:
> > > > > Does your algorithm lend itself well to paralellization? You could
> > > > > find out by running it with normal Thread's on a multiprocessor
> > > > > system. If this improves performance, then you can proceed to the next
> > > > > questions.

>
> > > > > The second question, still not Java specific:
> > > > > When you split the computation between N threads, how much
> > > > > communication is there between those threads? If the communication
> > > > > between threads has higher latency or limited bandwidth, how does that
> > > > > effect your algorithms throughput?

>
> > > > > Java specific question: Have you profiled your existing "slow" program
> > > > > to see what is taking the most time? It might not be what you
> > > > > expect.

>
> > > > > Also, have you considered finding a less naive algorithm? Without
> > > > > more details about your goal I can only speculate, but it might be
> > > > > that your approach is an O(N^3) algorithm, when in fact there may
> > > > > exist an O(n*logn) algorithm.

>
> > > > > So, before trying to cross machine boundaries, or even thread
> > > > > boundaries, it might be worth looking into other algorithms that speed
> > > > > up your process.

>
> > > > Hi i did send you a full email to your email address did you get it?

>
> > > No. Three reasons. The first is that my home computer has been
> > > disabled for a few weeks.
> > > The second reason is that the given e-mail address gets so much spam
> > > that I usually just delete it with out looking.
> > > The third reason is that I didn't invite you to e-mail me, please keep
> > > the discussion here unless invited to otherwise.

>
> > > As far as I know, no one here gets paid to answer questions on this
> > > newsgroup. On the other hand, many of us DO get paid to answer
> > > questions one-on-one. If you want free advice, post here; if you want
> > > to pay for solutions, find a contractor.

>
> > ok cool, weel first to answer your questions, i am pretty sure that
> > the algorithm will be quite easy to parralelise if that is the correct
> > term, the only bottleneck i can see is checking the correlation value
> > against the highest rank, ie the algorithm basically is a for next
> > loop BUT with billions of calcs to perform, the bottlenck will
> > obviously be when the calcs need to check for a higher correlation
> > against the current highest rank, however with the java program i have
> > seen it seesm to perform many of the loop functions simoultaneously,
> > ie i have 8 arrays of of which perform 10^3 calculations so many of
> > these calcs can be carried out concurrently.

>
> > The current system runs in VBA not in java so i havent run in java
> > yet.

>
> > Look forward to your reply

>
> Okay, so if I understand correctly:
>
> You have N number of arrays to test;
> Testing array A doesn't interact with the testing of array B.
> Testing any array results in an easy-to-compare score.
> So, you could create up to N threads, each calculates and records the
> score.
> All of the scores can then be easily compared to one another.
>
> Since there are only 8 arrays (hence 8 scores), the sorting isn't too
> hard to do.
>
> How do you score array A? What do you mean by correlation? longest-
> common-subsequence? number of similar/same elements? Something else?


Nok you are quite close its a little more compliacted than that but
you have the basic idea, Array A values dont change they are fixed
upon loaded from en excel file, i have written the code for that,
array b is made up of values based upon the values from 8 other arrays
each array is a calculation of (10^3)*N calcs, ie array 1 does a clac
and is then added to the values of the other 8 arrays, the addition or
the product of these arrays makes up Array B, array A and array B are
then checked for correlation, we can then check to se if the rank
value breaks the current highest rank. Its basically a very simple
algorithm, i have already have a working version in VBA and it works
perfectly but the amount of time it would take to complete is is the
tens of years so remote psuedo threads seems the way forward i have
two models i have downloaded from sourceforge but i dont fully
understand how they work yet, hence the need for someone to help me,
and as i said i am willing to pay.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
supercomputer in java? sav Java 4 02-16-2007 10:53 PM
Supercomputer and encryption and compression @ rate of 96% c3poptuse@yahoo.com C++ 1 04-16-2005 05:11 AM
January 29 (web services security) panel discussion at San Diego Supercomputer Center Ken North XML 0 01-20-2004 05:51 AM
SuperComputer Top 500 List Gunny Bunny Computer Information 0 11-17-2003 01:46 PM
Supercomputer! Danger Mouse Computer Support 2 08-20-2003 08:46 AM



Advertisments