Fix encode_phys precision loss for int64 with factor=1#648
Open
bizfsc wants to merge 1 commit intocanopen-python:masterfrom
Open
Fix encode_phys precision loss for int64 with factor=1#648bizfsc wants to merge 1 commit intocanopen-python:masterfrom
bizfsc wants to merge 1 commit intocanopen-python:masterfrom
Conversation
Skip the float division in encode_phys when factor is 1 (the default). Previously, `value /= self.factor` always performed float division, causing up to 10 bits of precision loss for large int64 values (e.g. 0x55554444AAAABBBB became 0x55554444AAAABC00). Use `!= 1` for the comparison since we are comparing values, not identity. Also clean up the redundant int(round(...)) to just round(), since round() already returns int when ndigits is omitted. Add unit tests covering: - int64 roundtrip with factor=1 (the original bug) - int type preservation with factor=1 - rounding with integer factor != 1 (existing behavior preserved) - float factor (existing behavior preserved) Based on the discussion in canopen-python#611.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Skip the float division in encode_phys when factor is 1 (the default). Previously,
value /= self.factoralways performed float division, causing up to 10 bits of precision loss for large int64 values (e.g. 0x55554444AAAABBBB became 0x55554444AAAABC00).Use
!= 1for the comparison since we are comparing values, not identity. Also clean up the redundant int(round(...)) to just round(), since round() already returns int when ndigits is omitted.Add unit tests covering:
Based on the discussion in #611.