«

On 1/18/2013 21:31, glen herrmannsfeldt wrote:
> Keith Thompson <kst-> wrote:
>> Shao Miller <> writes:
>
> (snip)
>>> 2. Very consistently _not_ all-bits-zero, suggesting a practical use for
>>> a debugger to trap (as opposed to a same-sized integer/object with
>>> all-zeroes)
>
>> When you say it "behaves as a null pointer in C code", what exactly do
>> you mean by that?
>
>> Does it compare equal to NULL? If not, then it *doesn't* behave as a
>> null pointer in C code.
>
> I can imagine a system that masks off some bits before comparing
> it to NULL. Not that I know any that actually do it.
>

Yes well that's along the lines that it might be. For example, if a
'free'd pointer becomes 0x0000000C, in C terminology, it'd be an
"indeterminate value." But, using C terminology again, it'd be an
"unspecified value" rather than a "trap representation", as the value
can be read and passed around. In C terminology, it would "point to no
object", just as a "null pointer" does. Upon dereferencing though,
Windows could tell from the representation just what the problem was:
Use of a 'free'd pointer. And yes, the higher bits might indicate that
it was a pointer that points to no object ("null class", perhaps we
could call it).

--
- Shao Miller
--
"Thank you for the kind words; those are the kind of words I like to hear.

Cheerily," -- Richard Harter

Keith Thompson wrote:
> Shao Miller writes:
> > Keith Thompson wrote:
> >> Shao Miller writes:
> >>> I think all-bits-zero is one null pointer value representation,
> >>> but I was talking about "trap representations" in practice (as
> >>> opposed to a discussion of those that depend on padding bits).
> >>> In Windows NT kernel-land, more often than not I see that when
> >>> a null pointer is trapped, it's actually _not_ all-bits-zero;
> >>> differing in the LSB. The debugger still calls it a null pointer.
> >>
> >> A null pointer is not a trap representation; it's a perfectly valid
> >> pointer value (that can't legally be dereferenced).
> >
> > Agreed. And that's kind of what I was getting at... That this
> > representation is:
> >
> > 1. A valid null pointer representation (behaves as a null pointer in C
> > code; no undefined behaviour from using this pointer value unless
> > dereferencing)
> >
> > 2. Very consistently _not_ all-bits-zero, suggesting a practical use for
> > a debugger to trap (as opposed to a same-sized integer/object with
> > all-zeroes)
>
> When you say it "behaves as a null pointer in C code", what exactly do
> you mean by that?
>
> Does it compare equal to NULL? If not, then it *doesn't* behave as a
> null pointer in C code.

I would guess that he's seeing something like the following:

struct {
int a, b, c, d;
} *p = NULL;

p->d = 0;

This traps in the debugger, and the debugger reports a "null pointer
dereference" at address 0x0000000c.

On 1/19/2013 04:28, Philip Lantz wrote:
> Keith Thompson wrote:
>> Shao Miller writes:
>>> Keith Thompson wrote:
>>>> Shao Miller writes:
>>>>> I think all-bits-zero is one null pointer value representation,
>>>>> but I was talking about "trap representations" in practice (as
>>>>> opposed to a discussion of those that depend on padding bits).
>>>>> In Windows NT kernel-land, more often than not I see that when
>>>>> a null pointer is trapped, it's actually _not_ all-bits-zero;
>>>>> differing in the LSB. The debugger still calls it a null pointer.
>>>>
>>>> A null pointer is not a trap representation; it's a perfectly valid
>>>> pointer value (that can't legally be dereferenced).
>>>
>>> Agreed. And that's kind of what I was getting at... That this
>>> representation is:
>>>
>>> 1. A valid null pointer representation (behaves as a null pointer in C
>>> code; no undefined behaviour from using this pointer value unless
>>> dereferencing)
>>>
>>> 2. Very consistently _not_ all-bits-zero, suggesting a practical use for
>>> a debugger to trap (as opposed to a same-sized integer/object with
>>> all-zeroes)
>>
>> When you say it "behaves as a null pointer in C code", what exactly do
>> you mean by that?
>>
>> Does it compare equal to NULL? If not, then it *doesn't* behave as a
>> null pointer in C code.
>
> I would guess that he's seeing something like the following:
>
> struct {
> int a, b, c, d;
> } *p = NULL;
>
> p->d = 0;
>
> This traps in the debugger, and the debugger reports a "null pointer
> dereference" at address 0x0000000c.
>

Indeed! Or, perhaps more commonly:

typedef struct { UINT32 A, B, C, D; } FOO, * PFOO;

PUINT32 GetFooD(PFOO foo) {
return &foo->D;
}

(I don't particularly like pointer typedefs, but pretend they're
Microsoft's.)

Then the caller dereferences the returned pointer, or passes it around
until some other function does. That is, I find that it's not usually
quite as obvious as in your example, and sometimes due to opacity or
abstraction.

(And thanks for the set-up for the function name; I'm hungry. )

--
- Shao Miller
--
"Thank you for the kind words; those are the kind of words I like to hear.

Cheerily," -- Richard Harter

Shao Miller <> writes:
> On 1/18/2013 21:31, glen herrmannsfeldt wrote:
>> Keith Thompson <kst-> wrote:
>>> Shao Miller <> writes:
>>
>> (snip)
>>>> 2. Very consistently _not_ all-bits-zero, suggesting a practical use for
>>>> a debugger to trap (as opposed to a same-sized integer/object with
>>>> all-zeroes)
>>
>>> When you say it "behaves as a null pointer in C code", what exactly do
>>> you mean by that?
>>
>>> Does it compare equal to NULL? If not, then it *doesn't* behave as a
>>> null pointer in C code.
>>
>> I can imagine a system that masks off some bits before comparing
>> it to NULL. Not that I know any that actually do it.
>
> Yes well that's along the lines that it might be. For example, if a
> 'free'd pointer becomes 0x0000000C, in C terminology, it'd be an
> "indeterminate value."
[...]

How does a free'd pointer *become* anything?

For example:

int *p = malloc(sizeof *p);
// Assume malloc() succeeded
// Assume the representation of p, when viewed as
// bytes, is 0x12, 0x34, 0x56, 0x78
free(p);
// Any attempt to refer to the value of p has undefined behavior.
// But if you examine its representation, say by type-punning it
// as an array of unsigned char, it will still look like
// 0x12, 0x34, 0x56, 0x78

The argument to free() is passed *by value*, so free() can't modify the
contents of the pointer object. The argument needn't even be an lvalue;
free(p + 1 - 1) is perfectly valid.

An implementation *could* do some sort of compiler magic, changing the
representation of a pointer object passed to free(), but (a) I don't
think it would be worth the effort, and (b) it's not clear that it would
even be conforming (the bytes making up the representation of the
pointer are objects in their own right, whose values don't change unless
you write to them). In any case, I'm not aware that Microsoft's
implementation does this -- nor am I aware that it implements "==" on
pointers so that anything other that all-bits-zero would compare equal
to NULL.

There are a lot of strange things an implementation *could* do, but the
topic at hand is what a particular implementation actually does.
Storing some specific byte sequence in uninitialized objects is a great
idea, but it doesn't have much to do with null pointers as C defines the
term.

--
Keith Thompson (The_Other_Keith) kst- <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

On 1/19/2013 23:03, Keith Thompson wrote:
> Shao Miller <> writes:
>> On 1/18/2013 21:31, glen herrmannsfeldt wrote:
>>> Keith Thompson <kst-> wrote:
>>>> Shao Miller <> writes:
>>>
>>> (snip)
>>>>> 2. Very consistently _not_ all-bits-zero, suggesting a practical use for
>>>>> a debugger to trap (as opposed to a same-sized integer/object with
>>>>> all-zeroes)
>>>
>>>> When you say it "behaves as a null pointer in C code", what exactly do
>>>> you mean by that?
>>>
>>>> Does it compare equal to NULL? If not, then it *doesn't* behave as a
>>>> null pointer in C code.
>>>
>>> I can imagine a system that masks off some bits before comparing
>>> it to NULL. Not that I know any that actually do it.
>>
>> Yes well that's along the lines that it might be. For example, if a
>> 'free'd pointer becomes 0x0000000C, in C terminology, it'd be an
>> "indeterminate value."
> [...]
>
> How does a free'd pointer *become* anything?
>

The value of a pointer becomes indeterminate at the end of the
pointed-to object's lifetime. Defect Report #260 discusses this.

Windows "Checked" builds are more easily debugged than their "Free"
counterparts. I don't think a freed pointer's representation changing
is beyond the realm of either possible or useful. But "might," above,
means that this was just a guess, which now seems likely to be wrong,
thanks to Mr. Philip Lantz' more obvious explanation.

However, the masking of bits that Mr. Glen Herrmannsfeldt mentioned
still seems about right. Here is another guess: The "null class
pointer" that WinDbg talks about is some reasonably-sized window around
the unsigned value 0. Given a null pointer of all-zeroes, the
'CONTAINING_RECORD' macro could yield, let's say, 0xFFFFFFF0. Mr.
Lantz' example shows the other direction.

> [...]
>
> There are a lot of strange things an implementation *could* do, but the
> topic at hand is what a particular implementation actually does.
> Storing some specific byte sequence in uninitialized objects is a great
> idea, but it doesn't have much to do with null pointers as C defines the
> term.
>

Which of these most closely resembles the pointer value 0x0000000C in
this Windows scenario?:

1. A trap representation for some pointer type, invoking UB when read
2. A valid value for some pointer type; pointing to an object
3. A null pointer; pointing to no object, invoking UB when dereferenced

If it doesn't compare equal to 'NULL', then it doesn't appear to match
any of these. But if we can fuzz our way from Standard theory to
real-world practice, we might note some similarities to #3.

Perhaps interestingly, Mr. Lantz' explanation actually means the
representation gives hints about which member of a structure was
involved in a bug. In Windows, many "object types" (not to be confused
with the C notions) are C structures beginning with a common initial
sequence. A "bad" pointer might reveal which member of this common
initial sequence was involved in a bug. This isn't the "pointer was
freed" guess of above, but still useful!

(By the way, I've asked in another forum about possible documentation
for NULL_CLASS_PTR_DEREFERENCE, although I don't know if you or anyone
else cares.)

--
- Shao Miller
--
"Thank you for the kind words; those are the kind of words I like to hear.

Cheerily," -- Richard Harter

Shao Miller <> writes:
> On 1/19/2013 23:03, Keith Thompson wrote:
[...]
>> There are a lot of strange things an implementation *could* do, but the
>> topic at hand is what a particular implementation actually does.
>> Storing some specific byte sequence in uninitialized objects is a great
>> idea, but it doesn't have much to do with null pointers as C defines the
>> term.
>
> Which of these most closely resembles the pointer value 0x0000000C in
> this Windows scenario?:
>
> 1. A trap representation for some pointer type, invoking UB when read
> 2. A valid value for some pointer type; pointing to an object
> 3. A null pointer; pointing to no object, invoking UB when dereferenced
>
> If it doesn't compare equal to 'NULL', then it doesn't appear to match
> any of these. But if we can fuzz our way from Standard theory to
> real-world practice, we might note some similarities to #3.

It could easily be #1. For example, if a type `struct foo` has a
member `bar` at offset 12 (0xC), and a `struct foo*` object `ptr`
that has a null pointer value, then `&(ptr->bar)` could easily have
a representation that looks like 0x0000000C.

[...]

> (By the way, I've asked in another forum about possible documentation
> for NULL_CLASS_PTR_DEREFERENCE, although I don't know if you or anyone
> else cares.)

The name NULL_CLASS_PTR_DEREFERENCE wouldn't necessarily refer to
a *null pointer*.

I don't think there's anything strange going on here. None of the
exotic possibilities permitted by the standard (null pointers with a
representation other than all-bits-zero, pointer objects changing
representation after being passed to free(), pointers with different
representations appear to be equal) appear to be happening.

--
Keith Thompson (The_Other_Keith) kst- <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

On 1/20/2013 10:57, Keith Thompson wrote:
> Shao Miller <> writes:
>>
>> Which of these most closely resembles the pointer value 0x0000000C in
>> this Windows scenario?:
>>
>> 1. A trap representation for some pointer type, invoking UB when read
>> 2. A valid value for some pointer type; pointing to an object
>> 3. A null pointer; pointing to no object, invoking UB when dereferenced
>>
>> If it doesn't compare equal to 'NULL', then it doesn't appear to match
>> any of these. But if we can fuzz our way from Standard theory to
>> real-world practice, we might note some similarities to #3.
>
> It could easily be #1. For example, if a type `struct foo` has a
> member `bar` at offset 12 (0xC), and a `struct foo*` object `ptr`
> that has a null pointer value, then `&(ptr->bar)` could easily have
> a representation that looks like 0x0000000C.
>

It could be #1, but it isn't #1 in this Windows scenario. Using such a
pointer value doesn't yield undefined behaviour until it's dereferenced.
It'll still compare as unequal with valid pointer values, can still be
assigned to a pointer, etc.

> [...]
>
>> (By the way, I've asked in another forum about possible documentation
>> for NULL_CLASS_PTR_DEREFERENCE, although I don't know if you or anyone
>> else cares.)
>
> The name NULL_CLASS_PTR_DEREFERENCE wouldn't necessarily refer to
> a *null pointer*.
>

It depends on the definition. For the C definition, if it doesn't
compare equal to 'NULL', then it's probably not a null pointer, as you
pointed out earlier. For WinDbg users, it probably is. Hopefully I'll
find out just what Microsoft has to say about it.

> I don't think there's anything strange going on here. None of the
> exotic possibilities permitted by the standard (null pointers with a
> representation other than all-bits-zero, pointer objects changing
> representation after being passed to free(), pointers with different
> representations appear to be equal) appear to be happening.
>

I agree with you about this; seems pretty straight-forward. I had
wondered about the observed representation 0x0000000C (given Geoff's
post about trap representations[1]), but I'm sure Mr. Lantz' explanation
is right and that this is not such a case.

[1] (Not to be confused with the C definition, but simply meaning a
representation which the debugger can trap/detect.)

--
- Shao Miller
--
"Thank you for the kind words; those are the kind of words I like to hear.

Cheerily," -- Richard Harter

Shao Miller <> writes:

> On 1/20/2013 10:57, Keith Thompson wrote:
>> Shao Miller <> writes:
>>>
>>> Which of these most closely resembles the pointer value 0x0000000C in
>>> this Windows scenario?:
>>>
>>> 1. A trap representation for some pointer type, invoking UB when read
>>> 2. A valid value for some pointer type; pointing to an object
>>> 3. A null pointer; pointing to no object, invoking UB when dereferenced
>>>
>>> If it doesn't compare equal to 'NULL', then it doesn't appear to match
>>> any of these. But if we can fuzz our way from Standard theory to
>>> real-world practice, we might note some similarities to #3.
>>
>> It could easily be #1. For example, if a type `struct foo` has a
>> member `bar` at offset 12 (0xC), and a `struct foo*` object `ptr`
>> that has a null pointer value, then `&(ptr->bar)` could easily have
>> a representation that looks like 0x0000000C.
>>
>
> It could be #1, but it isn't #1 in this Windows scenario. Using such
> a pointer value doesn't yield undefined behaviour until it's
> dereferenced. It'll still compare as unequal with valid pointer
> values, can still be assigned to a pointer, etc.

All of that seems to be entirely consistent with #1.

Talking about whether something is UB in "this Windows scenario" looks
very odd to me because the term UB is defined by the language, not the
implementation. If the language says something is UB then it is and you
can't conclude from the behaviour of an implementation that it isn't.
All behaviours are consistent with UB.

<snip>
--
Ben.

Shao Miller wrote:
> Keith Thompson wrote:
> > Shao Miller writes:
> >>
> >> Which of these most closely resembles the pointer value 0x0000000C in
> >> this Windows scenario?:
> >>
> >> 1. A trap representation for some pointer type, invoking UB when read
> >> 2. A valid value for some pointer type; pointing to an object
> >> 3. A null pointer; pointing to no object, invoking UB when dereferenced
> >>
> >> If it doesn't compare equal to 'NULL', then it doesn't appear to match
> >> any of these. But if we can fuzz our way from Standard theory to
> >> real-world practice, we might note some similarities to #3.
> >
> > It could easily be #1. For example, if a type `struct foo` has a
> > member `bar` at offset 12 (0xC), and a `struct foo*` object `ptr`
> > that has a null pointer value, then `&(ptr->bar)` could easily have
> > a representation that looks like 0x0000000C.
>
> It could be #1, but it isn't #1 in this Windows scenario. Using such a
> pointer value doesn't yield undefined behaviour until it's dereferenced.
> It'll still compare as unequal with valid pointer values, can still be
> assigned to a pointer, etc.

The undefined behavior occurred when &ptr->bar was executed (with ptr
equal to NULL). You cannot refer to the language definition to make
sense of anything that happens after that.

Shao Miller <> writes:
> On 1/20/2013 10:57, Keith Thompson wrote:
>> Shao Miller <> writes:
>>>
>>> Which of these most closely resembles the pointer value 0x0000000C in
>>> this Windows scenario?:
>>>
>>> 1. A trap representation for some pointer type, invoking UB when read
>>> 2. A valid value for some pointer type; pointing to an object
>>> 3. A null pointer; pointing to no object, invoking UB when dereferenced
>>>
>>> If it doesn't compare equal to 'NULL', then it doesn't appear to match
>>> any of these. But if we can fuzz our way from Standard theory to
>>> real-world practice, we might note some similarities to #3.
>>
>> It could easily be #1. For example, if a type `struct foo` has a
>> member `bar` at offset 12 (0xC), and a `struct foo*` object `ptr`
>> that has a null pointer value, then `&(ptr->bar)` could easily have
>> a representation that looks like 0x0000000C.
>>
>
> It could be #1, but it isn't #1 in this Windows scenario.

I believe it is.

> Using such a
> pointer value doesn't yield undefined behaviour until it's dereferenced.

Yes, it does. Concretely:

int main(void) {
struct foo {
char c[12];
int i;
};

struct foo *fptr = NULL;

int *iptr = &(fptr->i);

return 0;
}

The evaluation of the expression `&(fptr->i)` has undefined behavior.
If you disagree, please show me where the standard defines its
behavior.

If the observed behavior is that the representation 0x0000000C is stored
in `iptr`, that doesn't mean the behavior is defined.

> It'll still compare as unequal with valid pointer values, can still be
> assigned to a pointer, etc.

It happens to do so in the implementation we're discussing, and probably
in most implementations.

>> [...]
>>
>>> (By the way, I've asked in another forum about possible documentation
>>> for NULL_CLASS_PTR_DEREFERENCE, although I don't know if you or anyone
>>> else cares.)
>>
>> The name NULL_CLASS_PTR_DEREFERENCE wouldn't necessarily refer to
>> a *null pointer*.
>
> It depends on the definition. For the C definition, if it doesn't
> compare equal to 'NULL', then it's probably not a null pointer, as you
> pointed out earlier.

A pointer that doesn't compare equal to NULL *definitely* isn't a null
pointer, by the definition of pointer equality in N1570 6.5.9p6.
(Unless the program's behavior is undefined for other reasons, in which
case anything goes.)

> For WinDbg users, it probably is. Hopefully I'll
> find out just what Microsoft has to say about it.

I don't believe so. As far as I can see, the only evidence you've
presented that this thing is a "null pointer" is the identifier
"NULL_CLASS_PTR_DEREFERENCE".

What I believe that name implies is that the pointer value whose
representation looks like 0x0000000C was most likely created as
the result of a null pointer. It doesn't imply that 0x0000000C
is itself a null pointer. Nobody other than you is saying that.
(Unless you can produce some Microsoft documentation that uses the
phrase "null pointer" -- with no other words between "null" and
"pointer" -- to refer to such a pointer value.)

--
Keith Thompson (The_Other_Keith) kst- <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"