floating point weirdness
- Dominant language
- Macaulay2
- Stars
- 435
- Forks
- 297
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 11
Description
This is probably something about precision used while printing.
```
i3 : 2^53
o3 = 9007199254740992
i8 : (2^53-1) + 0.
o8 = 9007199254740990
o8 : RR (of precision 53)
i10 : (2^53-2) + 0.
o10 = 9007199254740990
o10 : RR (of precision 53)
i11 : (2^53-3) + 0.
o11 = 9007199254740990
o11 : RR (of precision 53)
i12 : (2^53-4) + 0.
o12 = 9007199254740990
o12 : RR (of precision 53)
i13 : (2^53-5) + 0.
o13 = 9007199254740990
o13 : RR (of precision 53)
i14 : (2^53-6) + 0.
o14 = 9007199254740990
o14 : RR (of precision 53)
i15 : (2^53-7) + 0.
o15 = 9007199254740980
o15 : RR (of precision 53)
i21 : (2^53-1) + 0.
o21 = 9007199254740990
o21 : RR (of precision 53)
i22 : toExternalString oo
o22 = .9007199254740991p53e16
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the Macaulay2 REPL transcript, starting with `(2^53-1) + 0.` and `toExternalString oo`. Compare the displayed values with the external string to determine whether the discrepancy is in floating-point arithmetic or printing; done means the reported behavior is explained and the affected output is handled consistently.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100