Velocity Reviews > Decimal precision ?

# Decimal precision ?

Piotr
Guest
Posts: n/a

 04-02-2004
Hi

I'm working with Datagrid containg decimal (editable) numbers. The
user-entered numbers are read by decimal.Parse() method. However, Datagrid
formats some decimals as "1", but some as (eg) "1.00" - depends on what the
user had entered. The page has DISABLED viewstate.

So, let someone explain me why the number 1 is not the same as 1.0 or 1.000?
I cannot find any property to set the precision.

The number of decimal digits can be different in different rows, so no
static format option can be used. Currently the only option to get correct
formatting is (a bit stupid) construction like this:
((decimal)DataBinder.Eval(Container.DataItem, "Quantity")).ToString("F" +
((int)DataBinder.Eval(Container.DataItem, "DecimalPlaces")).ToString()

Regards.

Martin Dechev
Guest
Posts: n/a

 04-02-2004
Hi, Piotr,

"Piotr" <(E-Mail Removed)> wrote in message
news:c4j6g1\$ni8\$(E-Mail Removed)...
> Hi
>
> I'm working with Datagrid containg decimal (editable) numbers. The
> user-entered numbers are read by decimal.Parse() method. However, Datagrid
> formats some decimals as "1", but some as (eg) "1.00" - depends on what

the
> user had entered. The page has DISABLED viewstate.
>
> So, let someone explain me why the number 1 is not the same as 1.0 or

1.000?

There is no difference *between the values* indeed. The difference is in the
way these are stored.

1 is stored as 1 / 10^0
1.0 is stored as 10 / 10^1
1.000 is stored as 1000 / 10^3

See:
http://msdn.microsoft.com/library/en...ClassTopic.asp

This code illustrates it:

decimal d = decimal.One;
for(int i = 0; i < 20;i++)
{
d = -decimal.Parse(d.ToString("F" + i));
int[] strip = decimal.GetBits(d);
ulong value = Convert.ToUInt64(
((uint)strip[1]).ToString("x") +
((uint)strip[0]).ToString("x"), 16);
Console.WriteLine(
"{0,-22} = {1,-22} / 10^{2,-3} {3}",
d,
value,
(strip[3] & 0xFF0000) >> 16,
(((strip[3] >> 31) & 1) > 0 ? "Negative" : "Positive"));
}

See the topic on GetBits method:
http://msdn.microsoft.com/library/en...tBitsTopic.asp

> I cannot find any property to set the precision.
>
> The number of decimal digits can be different in different rows, so no
> static format option can be used. Currently the only option to get correct
> formatting is (a bit stupid) construction like this:
> ((decimal)DataBinder.Eval(Container.DataItem, "Quantity")).ToString("F" +
> ((int)DataBinder.Eval(Container.DataItem, "DecimalPlaces")).ToString()

You can try setting the DataFormatString property alternatively:

"F" + DataBinder.Eval(Container.DataItem, "DecimalPlaces").ToString()

should be fine.

Hope this helps
Martin

>
> Regards.
>