Syndicate this site: (RSS)

September 26, 2007

Floating point numbers aren't.

So what, specifically, are the values that cause this display problem? Of the 9.214*10^18 different floating point numbers (floating point on wikipedia) that Excel 2007 can store, there are 6 floating point numbers (using binary representation) between 65534.99999999995 and 65535, and 6 between 65535.99999999995 and 65536 that cause this problem. You can't actually enter these numbers into Excel directly (since Excel will round to 15 digits on entry), but any calculation returning one of those results will display this issue if the results of the calculation are displayed in a cell. All other calculation results are not affected.

David Gainer

September 26, 2007 Comments (0) TrackBack (0)

September 22, 2007

Standard Model

The physics was fucked up when I got here.

September 22, 2007 Comments (0) TrackBack (0)

September 16, 2007

Robert Jordan

The last embrace of the mother welcome you home.

September 16, 2007 Comments (0) TrackBack (0)