Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   Memory error with quadratic interpolation (http://www.velocityreviews.com/forums/t956819-memory-error-with-quadratic-interpolation.html)

Isaac Won 01-23-2013 04:06 AM

Memory error with quadratic interpolation
 
Hi all,

I have tried to use different interpolation methods with Scipy. My code seems just fine with linear interpolation, but shows memory error with quadratic. I am a novice for python. I will appreciate any help.

#code
f = open(filin, "r")
for columns in ( raw.strip().split() for raw in f ):
a.append(columns[5])
x = np.array(a, float)


not_nan = np.logical_not(np.isnan(x))
indices = np.arange(len(x))
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

p = interp(indices)

------------------------------------------------------------------------
The number of data is 31747.

Thank you,

Isaac

Ulrich Eckhardt 01-23-2013 08:55 AM

Re: Memory error with quadratic interpolation
 
Am 23.01.2013 05:06, schrieb Isaac Won:
> I have tried to use different interpolation methods with Scipy. My
> code seems just fine with linear interpolation, but shows memory
> error with quadratic. I am a novice for python. I will appreciate any
> help.
>
> #code
> f = open(filin, "r")


Check out the "with open(...) as f" syntax.


> for columns in ( raw.strip().split() for raw in f ):


For the record, this first builds a sequence and then iterates over that
sequence. This is not very memory-efficient, try this instead:

for line in f:
columns = line.strip().split()


Concerning the rest of your problems, there is lots of code and the
datafile missing. However, there is also too much of it, try replacing
the file with generated data and remove everything from the code that is
not absolutely necessary.

Good luck!

Uli



Oscar Benjamin 01-23-2013 10:08 AM

Re: Memory error with quadratic interpolation
 
On 23 January 2013 08:55, Ulrich Eckhardt
<ulrich.eckhardt@dominolaser.com> wrote:
> Am 23.01.2013 05:06, schrieb Isaac Won:
>
>> I have tried to use different interpolation methods with Scipy. My
>> code seems just fine with linear interpolation, but shows memory
>> error with quadratic. I am a novice for python. I will appreciate any
>> help.

>

[SNIP]
>
>
> Concerning the rest of your problems, there is lots of code and the datafile
> missing. However, there is also too much of it, try replacing the file with
> generated data and remove everything from the code that is not absolutely
> necessary.


Also please copy paste the actual error message rather than paraphrasing it.


Oscar

Isaac Won 01-23-2013 02:24 PM

Re: Memory error with quadratic interpolation
 
On Tuesday, January 22, 2013 10:06:41 PM UTC-6, Isaac Won wrote:
> Hi all,
>
>
>
> I have tried to use different interpolation methods with Scipy. My code seems just fine with linear interpolation, but shows memory error with quadratic. I am a novice for python. I will appreciate any help.
>
>
>
> #code
>
> f = open(filin, "r")
>
> for columns in ( raw.strip().split() for raw in f ):
>
> a.append(columns[5])
>
> x = np.array(a, float)
>
>
>
>
>
> not_nan = np.logical_not(np.isnan(x))
>
> indices = np.arange(len(x))
>
> interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
>
>
>
> p = interp(indices)
>
>
>
> ------------------------------------------------------------------------
>
> The number of data is 31747.
>
>
>
> Thank you,
>
>
>
> Isaac


I really appreciate to both Ulich and Oscar.

To Oscar
My actual error message is:
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError
--------------------------------------------------------------------------
Thank you,

Hoonill

Isaac Won 01-23-2013 02:26 PM

Re: Memory error with quadratic interpolation
 
On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 08:55, Ulrich Eckhardt
>
> <ulrich.eckhardt@dominolaser.com> wrote:
>
> > Am 23.01.2013 05:06, schrieb Isaac Won:

>
> >

>
> >> I have tried to use different interpolation methods with Scipy. My

>
> >> code seems just fine with linear interpolation, but shows memory

>
> >> error with quadratic. I am a novice for python. I will appreciate any

>
> >> help.

>
> >

>
> [SNIP]
>
> >

>
> >

>
> > Concerning the rest of your problems, there is lots of code and the datafile

>
> > missing. However, there is also too much of it, try replacing the file with

>
> > generated data and remove everything from the code that is not absolutely

>
> > necessary.

>
>
>
> Also please copy paste the actual error message rather than paraphrasing it.
>
>
>
>
>
> Oscar


I really appreciate to both Ulich and Oscar.

To Oscar
My actual error message is:
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError
--------------------------------------------------------------------------
Thank you,

Isaac

Isaac Won 01-23-2013 02:26 PM

Re: Memory error with quadratic interpolation
 
On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 08:55, Ulrich Eckhardt
>
> <ulrich.eckhardt@dominolaser.com> wrote:
>
> > Am 23.01.2013 05:06, schrieb Isaac Won:

>
> >

>
> >> I have tried to use different interpolation methods with Scipy. My

>
> >> code seems just fine with linear interpolation, but shows memory

>
> >> error with quadratic. I am a novice for python. I will appreciate any

>
> >> help.

>
> >

>
> [SNIP]
>
> >

>
> >

>
> > Concerning the rest of your problems, there is lots of code and the datafile

>
> > missing. However, there is also too much of it, try replacing the file with

>
> > generated data and remove everything from the code that is not absolutely

>
> > necessary.

>
>
>
> Also please copy paste the actual error message rather than paraphrasing it.
>
>
>
>
>
> Oscar


I really appreciate to both Ulich and Oscar.

To Oscar
My actual error message is:
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError
--------------------------------------------------------------------------
Thank you,

Isaac

Isaac Won 01-23-2013 02:28 PM

Re: Memory error with quadratic interpolation
 
On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 08:55, Ulrich Eckhardt
>
>
>
> > Am 23.01.2013 05:06, schrieb Isaac Won:

>
> >

>
> >> I have tried to use different interpolation methods with Scipy. My

>
> >> code seems just fine with linear interpolation, but shows memory

>
> >> error with quadratic. I am a novice for python. I will appreciate any

>
> >> help.

>
> >

>
> [SNIP]
>
> >

>
> >

>
> > Concerning the rest of your problems, there is lots of code and the datafile

>
> > missing. However, there is also too much of it, try replacing the file with

>
> > generated data and remove everything from the code that is not absolutely

>
> > necessary.

>
>
>
> Also please copy paste the actual error message rather than paraphrasing it.
>
>
>
>
>
> Oscar


I really appreciate to both Ulich and Oscar.

To Oscar
My actual error message is:
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError
--------------------------------------------------------------------------
Thank you,

Hoonill

Isaac Won 01-23-2013 02:28 PM

Re: Memory error with quadratic interpolation
 
On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 08:55, Ulrich Eckhardt
>
>
>
> > Am 23.01.2013 05:06, schrieb Isaac Won:

>
> >

>
> >> I have tried to use different interpolation methods with Scipy. My

>
> >> code seems just fine with linear interpolation, but shows memory

>
> >> error with quadratic. I am a novice for python. I will appreciate any

>
> >> help.

>
> >

>
> [SNIP]
>
> >

>
> >

>
> > Concerning the rest of your problems, there is lots of code and the datafile

>
> > missing. However, there is also too much of it, try replacing the file with

>
> > generated data and remove everything from the code that is not absolutely

>
> > necessary.

>
>
>
> Also please copy paste the actual error message rather than paraphrasing it.
>
>
>
>
>
> Oscar


I really appreciate to both Ulich and Oscar.

To Oscar
My actual error message is:
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError
--------------------------------------------------------------------------
Thank you,

Hoonill

Oscar Benjamin 01-23-2013 02:40 PM

Re: Memory error with quadratic interpolation
 
On 23 January 2013 14:28, Isaac Won <winefrog@gmail.com> wrote:
> On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
>
> To Oscar
> My actual error message is:
> File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
> self._spline = splmake(x,oriented_y,order=order)
> File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
> coefs = func(xk, yk, order, conds, B)
> File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
> u,s,vh = np.dual.svd(B)
> File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
> full_matrices=full_matrices, overwrite_a = overwrite_a)
> MemoryError


Are you sure that's the *whole* error message? The traceback only
refers to the scipy modules. I can't see the line from your code that
is generating the error.


Oscar

Isaac Won 01-23-2013 02:47 PM

Re: Memory error with quadratic interpolation
 
On Wednesday, January 23, 2013 2:55:14 AM UTC-6, Ulrich Eckhardt wrote:
> Am 23.01.2013 05:06, schrieb Isaac Won:
>
> > I have tried to use different interpolation methods with Scipy. My

>
> > code seems just fine with linear interpolation, but shows memory

>
> > error with quadratic. I am a novice for python. I will appreciate any

>
> > help.

>
> >

>
> > #code

>
> > f = open(filin, "r")

>
>
>
> Check out the "with open(...) as f" syntax.
>
>
>
>
>
> > for columns in ( raw.strip().split() for raw in f ):

>
>
>
> For the record, this first builds a sequence and then iterates over that
>
> sequence. This is not very memory-efficient, try this instead:
>
>
>
> for line in f:
>
> columns = line.strip().split()
>
>
>
>
>
> Concerning the rest of your problems, there is lots of code and the
>
> datafile missing. However, there is also too much of it, try replacing
>
> the file with generated data and remove everything from the code that is
>
> not absolutely necessary.
>
>
>
> Good luck!
>
>
>
> Uli


Hi Ulich,

I tried to change the code following your advice, but it doesn't seem to work still.

My adjusted code is:

a = []

with open(filin, "r") as f:

for line in f:
columns = line.strip().split()

a.append(columns[5])
x = np.array(a, float)


not_nan = np.logical_not(np.isnan(x))
indices = np.arange(len(x))
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
p = interp(indices)
---------------------------------------------------------------------
And full error message is:
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError
-----------------------------------------------------------------------
Could you give me some advice for this situation?

Thank you always,

Isaac


All times are GMT. The time now is 06:51 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.