Debian Question posted in
The oficial documentation can be found here.

Parsing a date string reliably in jq – Debian

Overall aim: Parse a string in GMT as a time using jq and output both a formatted time and the difference of that time to "now". However, jqs (version 1.6, Debian testing) timezone handling seems very confused to me:

$ jq --version
jq-1.6
$ date
Sa 4. Jul 19:36:08 BST 2020
$ echo '""' | jq 'now | strftime("%H:%M")'
"18:36"        // OK, strftime is supposed to give GMT
$ echo '""' | jq 'now | strflocaltime("%H:%M")'
"19:36"        // also OK, British Summer time is one hour ahead, strflocaltime should give local time
$ echo '"2020-07-04T18:14:12Z"' | jq 'strptime("%Y-%m-%dT%H:%M:%SZ") | strftime("%H:%M")'
"18:14"        // strptime parses GMT, so this is fine
$ echo '"2020-07-04T18:14:12Z"' | jq 'strptime("%Y-%m-%dT%H:%M:%SZ") | strflocaltime("%H:%M")'
"18:14"        // but why is this not 19:14?!
$ echo '"2020-07-04T18:14:12Z"' | jq 'strptime("%Y-%m-%dT%H:%M:%SZ") | mktime | strftime("%H:%M")'
"19:14"        // and why does "mktime" change things around?
$ echo '"2020-07-04T18:14:12Z"' | jq 'strptime("%Y-%m-%dT%H:%M:%SZ") | mktime | strflocaltime("%H:%M")'
"20:14"       // and why does strflocaltime kick in after, but not before mktime?
$ echo '"2020-07-04T18:14:12Z"' | jq 'fromdate | strftime("%H:%M")'
"19:14"       // I thought fromdate was synonymous to strptime?
$ echo '"2020-07-04T18:14:12Z"' | jq 'fromdate | strflocaltime("%H:%M")'
"20:14"       // I suppose this is the same issue as above with mktime

Longer version: I’m playing around with an API to get a little display of arrival times at a nearby train station, in particular I want to show the next few trains and how many minutes from now they will leave. I want to use jq to parse that data. The data contains time strings of the format "2020-07-04T18:14:12Z". My understanding is that both fromdate and strptime in jq should parse that data as a GMT time stamp (from the man page: "In all cases these builtins deal exclusively with time in UTC.", the manpage seems to use GMT and UTC interchangeably) and any operations within jq use UTC, with only final output being in the local timezone if strflocaltime is used.

However, this understanding must be wrong, given the output of jq with various inputs shown above. In particular, I do not understand how to properly and reliably parse a time string as a GMT time stamp and b) once that is done, how the outputs of fromdate, mktime, now and strptime respectively differ when passed into strf[local]time to produce the array of outputs seen above.

Edit: Playing around further and with the information from the first two answers, it appears that the main problem is fromdate‘s application (or not) of Daylight Savings Time depending on the setting of the TZ environment variable:

$ TZ=BST jq -n '"2020-07-05T07:38:57Z" | fromdate'
1593934737
$ TZ=Etc/UTC jq -n '"2020-07-05T07:38:57Z" | fromdate'
1593934737
$ TZ=Europe/London jq -n '"2020-07-05T07:38:57Z" | fromdate'
1593938337
$ TZ=Asia/Tokyo jq -n '"2020-07-05T07:38:57Z" | fromdate'
1593934737
$ TZ=America/Los_Angeles jq -n '"2020-07-05T07:38:57Z" | fromdate'
1593938337
$ TZ=Asia/Kathmandu jq -n '"2020-07-05T07:38:57Z" | fromdate'
1593934737
$ unset TZ; jq -n '"2020-07-05T07:38:57Z" | fromdate'
1593938337

Note that London, Los Angeles and the unset TZ get a different Unix epoch timestamp than Tokyo, Kathmandu, UTC and the (I think malformed?) BST. I believe this should not happen, as timestamps should be timezone-independent. Unfortunately at the moment it appears to disregard the permanent timezone offset (Tokyo and Kathmandu give the same result as UTC, neither of the two have DST) but it does take into account DST unless running in a timezone which does not observe DST.

strflocaltime, when given a time stamp, seems to apply permanent and DST timezone corrections depending on the current value of TZ.

Unfortunately this seems to imply that I first need to set TZ to Etc/Utc to get fromdate to behave correctly and then when I want to print the local time, I need to re-set TZ to the local timezone.

Tags:

4

Answers


  1. Chosen as BEST ANSWER
    0

    I'd like to start building up an answer here, combining the different blocks:

    First, mktime takes into account DST but no other timezone information when taking a ‘broken-down time struct’:

    $ TZ=Etc/Utc jq -n '[2020,6,5,7,38,57,0,186] | mktime'
1593934737
$ TZ=Europe/London jq -n '[2020,6,5,7,38,57,0,186] | mktime'
1593938337
$ TZ=America/Los_Angeles jq -n '[2020,6,5,7,38,57,0,186] | mktime'
1593938337
$ TZ=Asia/Tokyo jq -n '[2020,6,5,7,38,57,0,186] | mktime'
1593934737
$ TZ=Asia/Kathmandu jq -n '[2020,6,5,7,38,57,0,186] | mktime'
1593934737
$ unset TZ; jq -n '[2020,6,5,7,38,57,0,186] | mktime'
1593938337

    Note that the only two ouputs are either 1593934737 or 1593938337, the difference of which is exactly 3600.

    Second, fromdate is identical to strptime() | mktime.

    Third, strflocaltime applies a time-zone offset (both permanent and DST) to unix-timestamp inputs, but not to broken-down-time inputs:

    $ TZ='Europe/London' jq -n '[2020,6,5,7,38,57,0,186] | strflocaltime("%H:%M")'
"07:38"
$ TZ='Asia/Tokyo' jq -n '[2020,6,5,7,38,57,0,186] | strflocaltime("%H:%M")'
"07:38"
$ TZ='Europe/London' jq -n '1593934737 | strflocaltime("%H:%M")'
"08:38"
$ TZ='Asia/Tokyo' jq -n '1593934737 | strflocaltime("%H:%M")'
"16:38"

    Fourth, now produces a unix-timestamp output which will be affected by strflocaltime's adjustment.

    Going over my original confusion-causing sequence in order:

    $ echo '""' | jq 'now | strftime("%H:%M")'
"18:36"        // OK, strftime is supposed to give GMT
$ echo '""' | jq 'now | strflocaltime("%H:%M")'
"19:36"        // also OK, British Summer time is one hour ahead, strflocaltime should give local time

    This is explained by (3) and (4) above: now produces a unix timestamp, strflocaltime adjusts this to the local time.

    $ echo '"2020-07-04T18:14:12Z"' | jq 'strptime("%Y-%m-%dT%H:%M:%SZ") | strftime("%H:%M")'
"18:14"        // strptime parses GMT, so this is fine
$ echo '"2020-07-04T18:14:12Z"' | jq 'strptime("%Y-%m-%dT%H:%M:%SZ") | strflocaltime("%H:%M")'
"18:14"        // but why is this not 19:14?!

    Here, strptime produces a broken-down time which is not adjusted by strflocaltime, by (3) above.

    $ echo '"2020-07-04T18:14:12Z"' | jq 'strptime("%Y-%m-%dT%H:%M:%SZ") | mktime | strftime("%H:%M")'
"19:14"        // and why does "mktime" change things around?
$ echo '"2020-07-04T18:14:12Z"' | jq 'strptime("%Y-%m-%dT%H:%M:%SZ") | mktime | strflocaltime("%H:%M")'
"20:14"       // and why does strflocaltime kick in after, but not before mktime?

    strptime produces the broken-down time and mktime in theory should convert this to a unix-timestamp time assuming it is in UTC, but mktime erroneously applies the one hour DST offset (by (1) above), leading to strftime producing the (accidentally correct) local time and strflocaltime – which corrects for both permanent and DST offset (by (3) above) – giving one further (for a total of two) hours offset.

    $ echo '"2020-07-04T18:14:12Z"' | jq 'fromdate | strftime("%H:%M")'
"19:14"       // I thought fromdate was synonymous to strptime?
$ echo '"2020-07-04T18:14:12Z"' | jq 'fromdate | strflocaltime("%H:%M")'
"20:14"       // I suppose this is the same issue as above with mktime

    This is simply a result of (2), that fromdate uses mktime internally.

    Compiling the latest commit on the master branch (a17dd32), this problem no longer appears as mktime no longer applies the one-hour offset. This is likely due to commit 3c5b1419.

    As a temporary workaround, we can get the offset introduced by mktime with: jq -n 'now | gmtime | mktime - (now | trunc)'. Subtracting this offset from any occurrence of fromdate will then reliably yield UTC timestamps.


  2. 0

    This probably isn’t the answer you’re looking for but it might clear some things up. builtin.jq defines

    def fromdateiso8601: strptime("%Y-%m-%dT%H:%M:%SZ")|mktime;
def todateiso8601: strftime("%Y-%m-%dT%H:%M:%SZ");
def fromdate: fromdateiso8601;
def todate: todateiso8601;

    the following test script

    #!/bin/bash
echo '"2020-07-04T18:14:12Z"' | jq -cr '
  def strptime_:  strptime("%Y-%m-%dT%H:%M:%SZ") ;
  def hour:       strftime("%H") ;
    ".                                    (.)"
  , ". | strptime_                        (strptime_)"
  , ". | fromdate                         (fromdate)"  
  , ". | fromdate | todate                (fromdate | todate)" 
  , ". | fromdate | hour                  (fromdate | hour)"   
'

    shows that on my mac (running jq 1.6) the %H strftime specifier appears sensitive to the setting of TZ.

    Without explicitly setting TZ (my system’s timezone is Pacific Daylight Time) I observe

    bash-3.2$ ./test.sh
.                                    2020-07-04T18:14:12Z
. | strptime_                        [2020,6,4,18,14,12,6,185]
. | fromdate                         1593890052
. | fromdate | todate                2020-07-04T19:14:12Z
. | fromdate | hour                  19

    explicitly setting TZ to America/Los_Angeles produces the same output

    bash-3.2$ env TZ=America/Los_Angeles ./test.sh
.                                    2020-07-04T18:14:12Z
. | strptime_                        [2020,6,4,18,14,12,6,185]
. | fromdate                         1593890052
. | fromdate | todate                2020-07-04T19:14:12Z
. | fromdate | hour                  19

    but explicitly setting TZ to Etc/UTC produces a different hour

    bash-3.2$ env TZ=Etc/UTC ./test.sh
.                                    2020-07-04T18:14:12Z
. | strptime_                        [2020,6,4,18,14,12,6,185]
. | fromdate                         1593886452
. | fromdate | todate                2020-07-04T18:14:12Z
. | fromdate | hour                  18

    I found it curious that the values from strptime are not quite the same as struct tm so digging a little deeper into builtin.c reveals some nontrivial platform-specific details along with the jv2tm which reveals the mapping from the struct tm to the json array strptime returns.

    static int jv2tm(jv a, struct tm *tm) {
  memset(tm, 0, sizeof(*tm));
  TO_TM_FIELD(tm->tm_year, a, 0);
  tm->tm_year -= 1900;
  TO_TM_FIELD(tm->tm_mon,  a, 1);
  TO_TM_FIELD(tm->tm_mday, a, 2);
  TO_TM_FIELD(tm->tm_hour, a, 3);
  TO_TM_FIELD(tm->tm_min,  a, 4);
  TO_TM_FIELD(tm->tm_sec,  a, 5);
  TO_TM_FIELD(tm->tm_wday, a, 6);
  TO_TM_FIELD(tm->tm_yday, a, 7);
  jv_free(a);

  // We use UTC everywhere (gettimeofday, gmtime) and UTC does not do DST.
  // Setting tm_isdst to 0 is done by the memset.
  // tm->tm_isdst = 0;

  // The standard permits the tm structure to contain additional members. We
  // hope it is okay to initialize them to zero, because the standard does not
  // provide an alternative.

  return 1;
}
    Login or Signup to reply.
  3. 0

    strflocaltime/1‘s behavior changes depending on the type of its input.

    If the input is an array (a "broken down time", this is what strptime returns), strflocaltime doesn’t correct it for the timezone and any seasonal time adjustments.

    $ TZ=UTC jq -n '[1970,0,1,0,0,1,4,0] | strflocaltime("%H")'
"00"
$ TZ=EST jq -n '[1970,0,1,0,0,1,4,0] | strflocaltime("%H")'
"00"

    But, if the input is a number (seconds since the Unix epoch, this is what mktime returns), strflocaltime feeds it to localtime first to get a broken down time; and localtime performs such corrections.

    $ TZ=UTC jq -n '1 | strflocaltime("%H")'
"00"
$ TZ=EST jq -n '1 | strflocaltime("%H")'
"19"

    In both cases, strftime is called with the broken down time structure, and the resulting string is returned.

    Login or Signup to reply.
  4. 0

    I have set the timezone to Europe/Amsterdam (+1).

    With JQ 1.6:

    This is expected:

    $ echo '"2020-03-28T11:04:04Z"' | jq 'fromdate | strflocaltime("%H:%M (%Z)")'
$ "12:04 CET"

    This is not expected:

    $ echo '"2020-03-29T11:04:04Z"' | jq 'fromdate | strflocaltime("%H:%M (%Z)")'
$ "14:04 CET"

    One would expect that at "2020-03-29 11:04" the time would be "13:04 CET" with summertime +1,
    but instead it gives me "14:04 CET"?

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search