Skip to content

fwrite: fix integer-backed POSIXct writing epoch instead of ISO #3535#7728

Open
rmwood0908 wants to merge 2 commits intoRdatatable:masterfrom
tjc234:master
Open

fwrite: fix integer-backed POSIXct writing epoch instead of ISO #3535#7728
rmwood0908 wants to merge 2 commits intoRdatatable:masterfrom
tjc234:master

Conversation

@rmwood0908
Copy link
Copy Markdown

Closes #3535

Problem
Previously, the fwrite() function would write POSIXct datetime columns
as epoch timestamps instead of the default ISO format when the column was
created with seq():

library(data.table)
Sys.setenv(TZ = 'UTC')
tmp <- tempfile()

# epoch instead of ISO
dt <- data.table(x = seq(as.POSIXct('1970-01-01'), by = '1 sec', length = 6))
fwrite(dt, tmp)
cat(readLines(tmp), sep = '\n')
#> x
#> 0
#> 1
#> 2
#> 3
#> 4
#> 5

# works correctly
dt[, x := x + 1]
fwrite(dt, tmp)
cat(readLines(tmp), sep = '\n')
#> x
#> 1970-01-01T00:00:01Z
#> 1970-01-01T00:00:02Z
#> 1970-01-01T00:00:03Z
#> 1970-01-01T00:00:04Z
#> 1970-01-01T00:00:05Z
#> 1970-01-01T00:00:06Z

This happens because seq() creates a POSIXct column backed by integers instead of doubles:

dt <- data.table(x = seq(as.POSIXct("1970-01-01"), by = "1 sec", length = 6))
typeof(dt$x)
# [1] "integer"

Fix

Two changes were required:

  1. src/fwriteR.c — Added a POSIXct check so integer-backed datetime columns are routed to the correct writer instead of being treated as plain integers.

  2. src/fwrite.c and src/fwrite.h — Added a new writePOSIXctInt function that reads integer values, converts them to double, then formats them as datetime strings.

Test Results

After applying the full fix, the output is now as desired

x
1970-01-01T00:00:00Z
1970-01-01T00:00:01Z
1970-01-01T00:00:02Z
1970-01-01T00:00:03Z
1970-01-01T00:00:04Z
1970-01-01T00:00:05Z

Test 2368.1 has been added to inst/tests/tests.Rraw and a NEWS entry has been added as per contributing guidelines

Copy link
Copy Markdown
Member

@aitap aitap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good start, thank you.

Comment thread inst/tests/tests.Rraw
Comment on lines +21589 to +21592
oldtz = Sys.getenv('TZ')
Sys.setenv(TZ = 'UTC')
tmp <- tempfile()
on.exit(Sys.setenv(TZ = oldtz), add = TRUE)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to adjust TZ, just give the tz = 'UTC' argument to as.POSIXct().

Comment thread inst/tests/tests.Rraw
Comment on lines +21594 to +21595
fwrite(DT, tmp)
test(2368.1, capture.output(cat(readLines(tmp), sep = '\n')), c(
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you don't give an output argument to fwrite(), it prints to the standard output. You can then test the value of capture.output(fwrite(...)) without going through a temporary file (which you won't have to remember to unlink()).

data.table:::test also has an output argument, but those are skipped when running with a non-English locale, and this test must succeed no matter whether or not message translation is enabled.

Comment thread src/fwrite.c
Comment on lines +489 to +490
const void *tmp = &x;
writePOSIXct(tmp, 0, pch);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can give &x straight to writePOSIXct() without going through a temporary variable. C allows this implicit conversion because the destination type is void *.

Comment thread src/fwrite.h
@@ -23,6 +23,7 @@ writer_fun_t writeITime;
writer_fun_t writeDateInt32;
writer_fun_t writeDateFloat64;
writer_fun_t writePOSIXct;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about renaming this function to writePOSIXctDouble?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fwrite ignores dateTimeAs for datetime sequences

3 participants