Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Negative array indicies and slice()

Reply
Thread Tools

Negative array indicies and slice()

 
 
Paul Rubin
Guest
Posts: n/a
 
      10-29-2012
Andrew <(E-Mail Removed)> writes:
> So: Why does python choose to convert them to positive indexes, and
> have slice operate differently than xrange


There was a thread a few years back, I think started by Bryan Olson,
that made the case that slice indexing is a Python wart for further
reasons than the above, and suggesting a notation like x[$-5] to denote
what we now call x[-5] (i.e. $ is the length of the string). So your
example x[$-4:3] would clearly be the same thing as x[6:3] and not give
any suggestion that it might wrap around.
 
Reply With Quote
 
 
 
 
Andrew Robinson
Guest
Posts: n/a
 
      10-29-2012
On 10/29/2012 04:19 AM, Steven D'Aprano wrote:
> On Mon, 29 Oct 2012 00:54:29 -0700, Andrew wrote:
>
> Slices and iterators have different purposes and therefore have not been
> made interchangeable. Yes, there are certain similarities between a slice
> and xrange, but there are also significant differences.

Aha, now were getting to the actual subject.

> [snip]
>> In 'C', where Python is written,

> That's a popular misapprehension. Python is written in Java, or Lisp, or
> Haskell, or CLR (dot Net), or RPython, or Ocaml, or Parrot. Each of those
> languages have, or had, at least one Python implementation. Oh, there's
> also a version written in C, or so I have heard.


I didn't say it was only written in "C", but in "C" where it is
implemented.
I will be porting Python 3.xx to a super low power embedded processor
(MSP430), both space and speed are at a premium.
Running Python on top of Java would be a *SERIOUS* mistake. .NET won't
even run on this system. etc.

>
>
>> Thank you for the code snippet; I don't think it likely that existing
>> programs depend on nor use a negative index and a positive index
>> expecting to take a small chunk in the center...

> On the contrary. That is the most straightforward and useful idea of
> slicing, to grab a contiguous slice of items.

Show me an example where someone would write a slice with a negative and
a positive index (both in the same slice);
and have that slice grab a contiguous slice in the *middle* of the list
with orientation of lower index to greater index.
I have asked before; It's not that I don't think it possible -- it's
that I can't imagine a common situation.

> Why would you want to grab a slice from the end of the list, and a slice
> from the start of the list, and swap them around? Apart from simulating
> card shuffles and cuts, who does that?

Advanced statistics programmers using lookup tables that are
symmetrical. Try Physicists too -- but they're notably weird.

>> My intended inferences about the iterator vs. slice question was perhaps
>> not obvious to you; Notice: an iterator is not *allowed* in
>> __getitem__().

> Actually, you can write __getitem__ for your own classes to accept
> anything you like.

Yes, I realize that.
But, why can't I just overload the existing __getitem__ for lists and
not bother writing an entire class?
Everything in Python is supposed to be an object and one of the big
supposed "selling" points is the ability to overload "any" object's methods.
The lists aren't special -- they're just a bunch of constant decimal
numbers, typically given as a large tuple.

>
> py> class Test:
> ... def __getitem__(self, index):
> ... return index
> ...

Better:
>>> class Test:

.... def __getitem__( self, *index ):
.... return index

No extra curlies required...

> You say that as if it were a bad thing.


hmmm... and you as if sarcastic?
It is a bad thing to have any strictly un-necessary and non-code saving
objects where memory is restricted.

> What existing class is that? It certainly isn't xrange.
>
> Because xrange represents a concrete sequence of numbers, all three of
> start, end and stride must be concrete, known, integers:
>


Let's up the ante. I'll admit xrange() won't do "later" fill in the
blank -- BUT --
xrange() is a subclass of an *existing* class called iterator.
Iterators are very general. They can even be made random.

>> The philosophy of Python is to have exactly one way to do something when
>> possible; so, why create a stand alone class that does nothing an
>> existing class could already do, and do it better ?

> py> xrange(4, None, 2)
> Traceback (most recent call last):
> File "<stdin>", line 1, in<module>
> TypeError: an integer is required


Hmmm..
Let's try your example exactly as shown...

"hello world"[aslice]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'aslice' is not defined

WOW. Cool.
Where did the blanks *actually* get filled in? Or HOW WILL they in your
next post?
>
> On the contrary, a simple list of three values not only could not do
> everything a slice does, but it's over twice the size!

Yeah, There is a definite issue there. But the case isn't decided by
that number alone.
A slice is storing three integers -- and an integer is size is 12.
So, slices don't use integers. If the type that *IS* used happens to be
a real Python type, we may merely typecast integers to that type --
insert them in a tuple and by definition, they must be the same size.

Looking at some of the online programming notes -- a slice apparently
doesn't use an integer storage variable that is capable of arbitrary
expansion. =-O -- and hence, won't work for very large sized lists.
That actually explains some crashes I have noted in the past when
working with 20 million element lists that I wanted a slice of. I had
*plenty* of ram on that system.
Besides: The program code to implement slice() is undoubtedly larger
than 12 bytes of savings!
How many slices() are typically found in memory simultaneously?

You're on the way to convincing me -- but you need to show off this fill
in the blank issue.
That's a difference of kind -- and not of size.
An iterator, could be an object with methods... So what common
application uses this "fill" in the blank stuff?




 
Reply With Quote
 
 
 
 
Andrew
Guest
Posts: n/a
 
      10-29-2012
On Sunday, October 28, 2012 9:26:01 PM UTC-7, Ian wrote:
> On Sun, Oct 28, 2012 at 10:00 PM, Andrew wrote:
>
> > Hi Ian,

>
> > Well, no it really isn't equivalent.

>
> > Consider a programmer who writes:

>
> > xrange(-4,3) *wants* [-4,-3,-2,-1,0,1,2]

>
> >

>
> > That is the "idea" of a range; for what reason would anyone *EVER* want-4 to +3 to be 6:3???

>
>
>
> That is what ranges do, but your question was about slices, not ranges.


Actually, I said in the OP:

"I also don't understand why slice() is not equivalent to an iterator, but can replace an integer in __getitem__() whereas xrange() can't."

=========================

Thank you for the code snippet; I don't think it likely that existing programs depend on nor use a negative index and a positive index expecting to take a small chunk in the center... hence, I would return the whole array; Orif someone said [-len(listX) : len(listX)+1 ] I would return the whole array twice.
That's the maximum that is possible.
If someone could show me a normal/reasonable script which *would* expect the other behavior, I'd like to know; compatibility is important.

=========================

My intended inferences about the iterator vs. slice question was perhaps not obvious to you; Notice: an iterator is not *allowed* in __getitem__().

The slice class when passed to __getitem__() was created to merely pass two numbers and a stride to __getitem__; As far as I know slice() itself does *nothing* in the actual processing of the elements. So, it's *redundant*functionality, and far worse, it's restrictive.

The philosophy of Python is to have exactly one way to do something when possible; so, why create a stand alone class that does nothing an existing class could already do, and do it better ?

A simple list of three values would be just as efficient as slice()!
xrange is more flexible, and can be just as efficient.

So, Have I misunderstood the operation of slice()? I think I might have...but I don't know.

In 'C', where Python is written, circularly linked lists -- and arrays are both very efficient ways of accessing data. Arrays can, in fact, have negative indexes -- perhaps contrary to what you thought. One merely defines avariable to act as the base pointer to the array and initialize it to the *end* of the array. Nor is the size of the data elements an issue, since inPython all classes are accessed by pointers which are of uniform size. I routinely do this in C.

Consider, also, that xrange() does not actually create a list -- but merelyan iterator generating integers which is exactly what __getitem__ works on..
So, xrange() does not need to incur a memory or noticeable time penalty.

From micro-python, it's clear that their implementation of xrange() is at the 'C' level; which is extremely fast.
 
Reply With Quote
 
Andrew
Guest
Posts: n/a
 
      10-29-2012
On Sunday, October 28, 2012 9:26:01 PM UTC-7, Ian wrote:
> On Sun, Oct 28, 2012 at 10:00 PM, Andrew wrote:
>
> > Hi Ian,

>
> > Well, no it really isn't equivalent.

>
> > Consider a programmer who writes:

>
> > xrange(-4,3) *wants* [-4,-3,-2,-1,0,1,2]

>
> >

>
> > That is the "idea" of a range; for what reason would anyone *EVER* want-4 to +3 to be 6:3???

>
>
>
> That is what ranges do, but your question was about slices, not ranges.


Actually, I said in the OP:

"I also don't understand why slice() is not equivalent to an iterator, but can replace an integer in __getitem__() whereas xrange() can't."

=========================

Thank you for the code snippet; I don't think it likely that existing programs depend on nor use a negative index and a positive index expecting to take a small chunk in the center... hence, I would return the whole array; Orif someone said [-len(listX) : len(listX)+1 ] I would return the whole array twice.
That's the maximum that is possible.
If someone could show me a normal/reasonable script which *would* expect the other behavior, I'd like to know; compatibility is important.

=========================

My intended inferences about the iterator vs. slice question was perhaps not obvious to you; Notice: an iterator is not *allowed* in __getitem__().

The slice class when passed to __getitem__() was created to merely pass two numbers and a stride to __getitem__; As far as I know slice() itself does *nothing* in the actual processing of the elements. So, it's *redundant*functionality, and far worse, it's restrictive.

The philosophy of Python is to have exactly one way to do something when possible; so, why create a stand alone class that does nothing an existing class could already do, and do it better ?

A simple list of three values would be just as efficient as slice()!
xrange is more flexible, and can be just as efficient.

So, Have I misunderstood the operation of slice()? I think I might have...but I don't know.

In 'C', where Python is written, circularly linked lists -- and arrays are both very efficient ways of accessing data. Arrays can, in fact, have negative indexes -- perhaps contrary to what you thought. One merely defines avariable to act as the base pointer to the array and initialize it to the *end* of the array. Nor is the size of the data elements an issue, since inPython all classes are accessed by pointers which are of uniform size. I routinely do this in C.

Consider, also, that xrange() does not actually create a list -- but merelyan iterator generating integers which is exactly what __getitem__ works on..
So, xrange() does not need to incur a memory or noticeable time penalty.

>From micro-python, it's clear that their implementation of xrange() is at the 'C' level; which is extremely fast.

 
Reply With Quote
 
andrewr3mail@gmail.com
Guest
Posts: n/a
 
      10-29-2012
On Sunday, October 28, 2012 10:14:03 PM UTC-7, Paul Rubin wrote:
> Andrew writes:
>
> > So: Why does python choose to convert them to positive indexes, and

>
> > have slice operate differently than xrange

>
>
>
> There was a thread a few years back, I think started by Bryan Olson,
>
> that made the case that slice indexing is a Python wart for further
>
> reasons than the above, and suggesting a notation like x[$-5] to denote
>
> what we now call x[-5] (i.e. $ is the length of the string). So your
>
> example x[$-4:3] would clearly be the same thing as x[6:3] and not give
>
> any suggestion that it might wrap around.


I'm getting very frustrated with the editor provided for this group... It keeps posting prematurely, and putting my email in even when I tell it not to each time; and there is no way to edit a post... but deleting is ok...

I think Olson makes a good point. The len() operator is so ubiquitous thatit would be very useful to have a shorthand like that.

I'll have to look for his thread.

I'm thinking that I might just patch my version of Python 3.x, in C, to allow iterators to be passed to __getitem__; I haven't ever seen someone wanting to use mixed sign indexes to extract a small chunk of an array in the middle; so I don't think my patch will break existing code.

The snippets of code given by other posters in the thread might also be used to make a compatibility wrapper; I'll have to study it closer; so that distributed code would still work on unpatched python, albeit much slower.
 
Reply With Quote
 
Chris Rebert
Guest
Posts: n/a
 
      10-29-2012
On Mon, Oct 29, 2012 at 12:54 AM, Andrew <(E-Mail Removed)> wrote:
> On Sunday, October 28, 2012 9:26:01 PM UTC-7, Ian wrote:
>> On Sun, Oct 28, 2012 at 10:00 PM, Andrew wrote:

<snip>
> The slice class when passed to __getitem__() was created to merely pass two numbers and a stride to __getitem__; As far as I know slice() itself does *nothing* in the actual processing of the elements. So, it's *redundant* functionality, and far worse, it's restrictive.
>
> The philosophy of Python is to have exactly one way to do something when possible; so, why create a stand alone class that does nothing an existing class could already do, and do it better ?
>
> A simple list of three values would be just as efficient as slice()!
> xrange is more flexible, and can be just as efficient.
>
> So, Have I misunderstood the operation of slice()? I think I might have.... but I don't know.


`slice` is intentionally lenient about the types of the start, stop, and step:
>>> class Foo:

.... def __getitem__(self, slice_):
.... print(slice_)
.... return 42
....
>>> Foo()["a":"b":"c"]

slice('a', 'b', 'c')
42
>>>

Thus, the thing being sliced is free to interpret the parts of the
slice however it wishes; hence, slice() is unable to contain the
"processing" you speak of.
By contrast, xrange() limits itself to integers.
To support the more general case, the slice syntax thus produces a
`slice` rather than an `xrange`.
Doubtlessly, there are also historical issues involved. As implied by
the ugliness of its name, `xrange` was added to the language
relatively later.

Cheers,
Chris
 
Reply With Quote
 
andrewr3mail@gmail.com
Guest
Posts: n/a
 
      10-29-2012
On Sunday, October 28, 2012 9:44:56 PM UTC-7, alex23 wrote:
> On Oct 29, 2:09*pm, Andrew <(E-Mail Removed)> wrote:
>
> > I use this arbitrary range code *often* so I need a general purpose solution.

>
> > I looked up slice() but the help is of no use, I don't even know how I might

>
> > overload it to embed some logic to concatenate ranges of data; nor evenif

>
> > it is possible.

>
>
>
> Slices are passed in if provided to __getitem__/__setitem__/
>
> __delitem__, so you'd need to override it at the list level:
>
>
>
> class RangedSlicer(list):
>
> def __getitem__(self, item):
>
> # map item.start, .stop and .step to your own semantics
>
>
>
> Then wrap your lists with your RangedSlicer class as needed.


Hmmm...

I began a test in an interactive shell:
>>> class RangedSlicer(list):

.... def __getitem__(self,item):
.... print item
....
>>> a=[1,2,3,4,5]
>>> a.__getitem__( slice(1,5) )

[2, 3, 4, 5]

Very odd... I would have expected [1,2,3,4]

>>> a.__getitem__( slice(1, )

[2, 3, 4, 5]

So, slice() somehow was truncated although it ought to have been executed first, and passed to __getitem__() before __getitem__ could affect it.
That requires some tricky programming!

Not only that, but,
a.__getitem__( xrange[1,8] )
Causes an exception before the __getitem__ shadowing received it.

I don't see how I can over-ride it with your suggestion, but that's very inconsistent.... for your idea seems to be normal python that would work for user defined classes.

 
Reply With Quote
 
Chris Rebert
Guest
Posts: n/a
 
      10-29-2012
On Mon, Oct 29, 2012 at 1:08 AM, <(E-Mail Removed)> wrote:
> On Sunday, October 28, 2012 10:14:03 PM UTC-7, Paul Rubin wrote:
>> Andrew writes:

<snip>
> I'm getting very frustrated with the editor provided for this group... It keeps posting prematurely, and putting my email in even when I tell it not to each time; and there is no way to edit a post... but deleting is ok...


This is a Usenet newsgroup[1], not a web forum. There are noteworthy
differences between the two.
FWICT, you happen to be accessing us via Google Groups, which is
widely acknowledged to suck. We are not hosted *by* Google Groups;
they just happen to carry our posts.
Personally, I'd suggest using our mailing list mirror instead:
http://mail.python.org/mailman/listinfo/python-list
Or use some other, better newsgroup provider that also carries us.

[1]: http://en.wikipedia.org/wiki/Usenet

Regards,
Chris
 
Reply With Quote
 
Chris Rebert
Guest
Posts: n/a
 
      10-29-2012
On Mon, Oct 29, 2012 at 1:24 AM, <(E-Mail Removed)> wrote:
> On Sunday, October 28, 2012 9:44:56 PM UTC-7, alex23 wrote:
>> On Oct 29, 2:09 pm, Andrew <(E-Mail Removed)> wrote:

<snip>
>> class RangedSlicer(list):

<snip>
>> Then wrap your lists with your RangedSlicer class as needed.

>
> Hmmm...
>
> I began a test in an interactive shell:
>>>> class RangedSlicer(list):

> ... def __getitem__(self,item):
> ... print item
> …


This just defines a class; it doesn't modify in-place the normal
behavior of plain lists. You have to actually *use* the class.

>>>> a=[1,2,3,4,5]


You never wrapped `a` in a RangedSlicer or otherwise made use of RangedSlicer!
You wanted:
a = RangedSlicer([1,2,3,4,5])

>>>> a.__getitem__( slice(1,5) )

> [2, 3, 4, 5]
>
> Very odd... I would have expected [1,2,3,4]


"[2, 3, 4, 5]" is the return value from `a.__getitem__( slice(1,5) )`
(or, equivalently, from `[1,2,3,4,5][1:5]`). It is not the result of
"print item"; that line of code is never executed since you never used
the RangedSlicer class at all.

Regards,
Chris
 
Reply With Quote
 
andrewr3mail@gmail.com
Guest
Posts: n/a
 
      10-29-2012
On Monday, October 29, 2012 1:38:04 AM UTC-7, Chris Rebert wrote:
> On Mon, Oct 29, 2012 at 1:24 AM,
>
> > On Sunday, October 28, 2012 9:44:56 PM UTC-7, alex23 wrote:

>
> >> On Oct 29, 2:09 pm, Andrew < wrote:

>
> You never wrapped `a` in a RangedSlicer or otherwise made use of RangedSlicer!
>
> You wanted:
>
> a = RangedSlicer([1,2,3,4,5])
>
>
>
> >>>> a.__getitem__( slice(1,5) )

>
> > [2, 3, 4, 5]

>
> >

>
> > Very odd... I would have expected [1,2,3,4]

>
>
>
> "[2, 3, 4, 5]" is the return value from `a.__getitem__( slice(1,5) )`
>
> (or, equivalently, from `[1,2,3,4,5][1:5]`). It is not the result of
>
> "print item"; that line of code is never executed since you never used
>
> the RangedSlicer class at all.
>
>
>
> Regards,
>
> Chris


My apology --- I deleted that post; yet it didn't delete... I saw my mistake seconds after posting.

***** gmail.

Note: I subscribed to the python-list, and am able to recieve e-mails, but I don't see how to write a post for this particular thread nor subscribe to this particular thread...

A brief suggestion, or link to a howto would be *much* appreciated.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Negative array indicies and slice() Ian Kelly Python 0 10-31-2012 07:43 AM
Re: Negative array indicies and slice() Dennis Lee Bieber Python 0 10-31-2012 03:36 AM
Re: Negative array indicies and slice() Ethan Furman Python 0 10-30-2012 09:21 PM
Re: Negative array indicies and slice() Mark Lawrence Python 0 10-29-2012 10:10 AM
Re: Negative array indicies and slice() Andrew Robinson Python 0 10-29-2012 02:31 AM



Advertisments