Macaulay2 / Macaulay2/M2

floating point weirdness

Open
#708 6 comments 0 reactions 0 assignees View on GitHub
Interpreter
Dominant language
Macaulay2
Stars
435
Forks
297
Avg merge
4d 20h
Merged PRs (30d)
11

Description

This is probably something about precision used while printing.

```
i3 : 2^53

o3 = 9007199254740992

i8 : (2^53-1) + 0.

o8 = 9007199254740990

o8 : RR (of precision 53)

i10 : (2^53-2) + 0.

o10 = 9007199254740990

o10 : RR (of precision 53)

i11 : (2^53-3) + 0.

o11 = 9007199254740990

o11 : RR (of precision 53)

i12 : (2^53-4) + 0.

o12 = 9007199254740990

o12 : RR (of precision 53)

i13 : (2^53-5) + 0.

o13 = 9007199254740990

o13 : RR (of precision 53)

i14 : (2^53-6) + 0.

o14 = 9007199254740990

o14 : RR (of precision 53)

i15 : (2^53-7) + 0.

o15 = 9007199254740980

o15 : RR (of precision 53)

i21 : (2^53-1) + 0.

o21 = 9007199254740990

o21 : RR (of precision 53)

i22 : toExternalString oo

o22 = .9007199254740991p53e16

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Macaulay2 REPL transcript, starting with `(2^53-1) + 0.` and `toExternalString oo`. Compare the displayed values with the external string to determine whether the discrepancy is in floating-point arithmetic or printing; done means the reported behavior is explained and the affected output is handled consistently.

Written by the indexing model from the issue text.

Assessment

Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.