proposal: encoding/json/v2: use "72h3m0.5s" as the default representation for time.Duration #71631
Comments
|
Please vote for what you believe is the most reasonable default representation of
|
|
Two reasons for ❤:
|
|
@timbray, you have experience writing RFCs, would you be interested in writing one similar to RFC 3339 that standardizes on a particular profile of ISO 8601 so that we can settle this problem of too many ISO 8601-like grammars that are all subtly different from each other? |
|
Related Issues
Related Code Changes (Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.) |
|
It's about midnight here now so I'll get back to you. I can probably write an Internet-Draft such as you propose. Nobody can “write an RFC”, to turn an Internet-Draft into an RFC you have to go through the IETF process which is fairly sane but not fast. Having said that, an Internet-Draft is immutable and lives forever and even though the front matter says it shouldn't be used as a reference, that happens regularly. |
gabyhelp
added
the
LibraryProposal
|
There is very little difference from the time.duration.string and the subset of ISO 8601 so it is worth to go the distance and make it the default. |
|
Since JSON is a serialization format, what does interoperability look like for other software, like those built on earlier versions of Go, or expecting to parse a field with the old representation? I view this as similar to #10275 , whether the change is to the underlying type or to just encoding/json/v2 , it has potential to break systems without careful auditing, especially with external types which may add in new struct tags. |
|
@seankhliao If I understood the parent proposal right, you would create an Unmarshaller with V1 options, and leave the Marshaler with default options. Anyway your question does not have anything to do with this default which in anycase would have been different in the original proposal. The v2 proposal containes several other breaking changes and their proposed migration options. |
|
Absolutely, there will be a V1 marshaller option no matter what that implements the V1 behavior. |
|
you can only do that if you only consume external messages in the old format, what about producing messages for external systems? |
|
json v1 will be entirely implemented in v2 with the right compativbility options so you can both consume and create messages according to the v1 spec using v2. |
|
It might be worth reviewing what other JSON serialisation implementations do with durations. Eg, Protobuf / ProtoJSON:
I think this option adding to the list above (but it won't get the voting attention of the initial options). It could be considered a subset of |
|
While I find ISO 8601 less readable compared to the Go My motivation with an RFC is to have something similar to RFC 3339 to specify a limited and concise grammar that the Internet agrees as the proper way to represent timestamps in order to resolve interoperability issues (which is the stated goal of why it was created and has largely accomplished its goal). When bugs are filed claiming problems parsing a timestamp, we can look at RFC 3339 to settle the issue. We need something similar for durations.
I'd be happy to help write the draft, but I'm unfamiliar to the RFC process. I would expect the RFC to satisfy several goals:
Many APIs or protocols implicitly support a limited grammar of ISO 8601 by functionally banning higher order units (e.g., years), banning commas, requiring adherence to "carry-over points" (or not), etc. However, exactly what is supported or not often isn't documented. RFC 3339 was written in 2002 for timestamps, and surprisingly we haven't had the equivalent for durations. I suspect an RFC for duration to be much simpler than RFC 3339 since a limited grammar is actually fairly simple. As a historical note: #43823 altered the Go "time" package to support commas to be in greater compliance with ISO 8601, which sadly broke Go's compliance with RFC 3339, which forbids commas. By transitive property, this bug persists in v1 "encoding/json" as fixing it unfortunately broke many users who were implicitly relying on commas in timestamps. In v2, we want to aim to have strict and precise support for various formats to support JSON's goal being a language agnostic, data interchange, format. |
|
Regarding protobuf's format for a duration, I'll point out that they're also in an inconsistent position where timestamps use RFC 3339, while durations use a custom format. The choice of format for duration predates my time working on protobufs, but I suspect it's because RFC 3339 existed, but an equivalent RFC for durations did not. |
At present, we're assuming that software that's already been serializing a That said, some closed systems (e.g., RPCs between two internal services) could theoretically migrate the wire representation since both sides of the RPC are controlled by the same organization. One could imagine a format grammar that specifies dual support for formats. For example, suppose there was This particular issue is about the default representation for |
Having checked around… what a mess. So yes, if you guys decide to adopt the ISO8601 profile approach, I would volunteer to write an Internet-Draft in the style of 3339 with the selected profile. I am not repeat NOT volunteering to try to push this through the IETF and make it an RFC. (There's a small but non-zero chance that enough people would be interested that this could end up happening anyhow.) This might be useful in that it would force us to be super-precise. What I'd want before I started working on this would be an agreed-on list of example duration literals that you would expect to be accepted, and another list that would be rejected as ill-formed. Building specs from examples is so much easier than from documents explaining what we think we want. |
Oops, wrote the above before I caught up on this morning's traffic. Yes, your 1-3 list seems sensible and achievable, and I agree that it would be much shorter than 3339, among other things almost all the ABNF needed can be taken from 3339 by reference. But I'd still want to see a list of good/bad literals to make sure thinking is clear. |
This is a very minor part of the proposal, but I will note that Thus I don't realistically see how |
True, but we want Go's implementation of a duration in JSON to be forwards compatible if there's ever a "time/v2" that supports higher precision. |
|
Here's my attempt at a particular profile of ISO 8601 that I believe should be parsable by most compliant ISO 8601 implementations. ExamplesValid:
Invalid:
Design considerations for the grammar:
If we're willing to lose the bijective property, here are some reasonable ways to loosen the parsing behavior and still be ISO 8601 compliant:
We define that:
With the above grammar, it is possible represent any finite duration. |
XML's duration format made this mistake. It's defined as: The "date" part of that format has a huge number of problems, like a duration of 3 months (P3M) can be a mixture of 28, 29, 30 or 31 days and you have no way to know unless you know the reference time to apply the duration to (3 months from May 2024) and have an up to date timezone database. Similarly, 2 years (P2Y) can be varying amount of days due to leap years without further context. The time section of the XML duration format is identical to what @dsnet is proposing. This is extremely convenient for services that once started out with XML but have grown JSON interfaces over time. The exception is that the XML format does not support fractional seconds. So for interoperability sake, it would be great to not support the following:
That would ensure that 1s always gets serialised as |
|
We need to be able to encode sub-second durations, though. Milliseconds, microseconds, and even nanoseconds are generally useful, and they can be encoded with encoding/json v1 today, just like they can with https://pkg.go.dev/time#Duration.String. |
|
Agreed. And that's fine. All I'm asking for is to not allow the unnecessary trailing zeros. That would break compatibility with existing formats, like XML duration, and doesn't seem to provide any benefit. |
Both of these have the problem that it's too hard to see what the field contains and too easy to get the unit wrong.
Too Go-specific. Trivial to parse for Go programs, but everyone else has to write a custom parser.
At least it's kind of a standard? But if we need to write an Internet Draft to standardize it enough to be usable, it doesn't seem like a great choice.
Everyone has to write a custom parser, so at least it's ecumenical? Of these options, attempting to set precedent for an ISO 8601 subset seems like the best. But I think they're all inferior to the very simple alternative used by the protobuf JSON encoding: Fractional seconds with an "s" suffix, e.g., "47683.228428s". This is trivial to parse, hard to misinterpret, and scales to whatever precision we want. |
The grammar I proposed above is going to encode as something that is almost certainly compatible with any decoder that claims compliance with ISO 8601. The challenge is what to do when inevitable bugs get filed claiming that the encoded output of some non-Go system could not be handled by Go's decoder for ISO 8601. We could keep loosening the decoder to support whatever bugs are filed, but this process of not knowing what "compliance" means is problematic, but not fatal. By virtue of everyone functionally approaching ISO 8601 with Postel's law, the format has become fairly usable, but landmines exist if you encode something that strays from the core grammar. One could imagine v2 using a restricted ISO 8601 profile, while also pursuing an RFC to codify that particular profile. While not unique to Go, the language has set the precedent of have strong specifications first and making the implementation follow the specification, rather than the other way around. We could be a trend setter and being the ones that advocate for a widely agreed duration format based on ISO 8601. |
Sure, but we can define a simple and precise grammar. I'm suggesting the general notion of "floating point seconds followed by the letter s", not precise compatibility with protobuf JSON implementations. |
|
I suppose you maybe have the best of both worlds with an absolutely brutally minimal subset of 8601, where all serializations have only seconds?
The regexp gibberish is trying to say that the numeric part can look like 3412 or 0.3412 or 34.12 but not 034.12 or 34.120 |
|
Oh, pardon me, that doesn't work, apparently you can't have more than 60 seconds. sigh So probably something close to what @dsnet proposed above. |
It's still unclear to me. There are several editions of ISO 8601. The 1988 edition mentions optional carry-over requirements, but at least one of the later editions does not seem to? Since ISO 8601 requires payment and has had 5 major editions, it's hard to check what's actually required or not. Your proposed syntax of just |
|
Also 3339 only allows 0-59. |
|
I'm not convinced there will be any momentum behind using a more complete ISO8601 encoding for JSON durations across the broader JSON ecosystem (beyond Go). Eg, that would need libraries for other languages and a desire to use them. Otherwise anyone interfacing with Go JSON durations will have to built it themselves - lots of opportunity for bugs/frustration. Making Go JSON durations more awkward to use/interoperate seems like a significant disadvantage. Standards date/time formats are great, but unfortunately I don't think we'll realise much broader benefit here. This leads me back to fractional seconds + trailing "s". It's unambiguous when encountered in the wild, and trivial to parse/encode. Much more likely to work well in the broader JSON ecosystem for interoperability. It's mostly the |
|
The ISO-style duration format,
The Schema.org Duration format also uses this, which means it's fairly well recognised on the web and used in microformats already. If we're talking interoperability with other platforms, that seems the format that provides the most convenient out of the box experience, as it wouldn't require any custom logic to parse a "1.0475s" string into something meaningful. One potential issue though is that some of those parses might do the same thing as XML's duration, so they wouldn't recognise a duration with more than 23H, 59M and 60S, expecting anything over that to turn into P1DT instead. That would be more annoying. |
|
Is there general agreement that ISO8601 durations across existing implementations that they must never use years/months/days/weeks? I suspect any usage of ISO8601 will face an awkward tension between supporting more than hours/minutes/seconds since someone else uses it, or frustration from interoperability issues - looks like an ISO8601 duration, but it doesn't work. Go's ..which brings me back to fractional seconds + trailing "s" as less ambiguous and better for interoperability. |
|
I'm fairly convinced now that ISO-8601 is not the right default. ISO-8601 durations explicitly support time intervals. "P1Y" is one year, and a variable number of seconds depending on the starting time. "P10S" is ambiguous as to whether it means ten seconds in monotonic or civil time. I just don't see any advantage to using ISO-8601 to marshal a Fractional seconds and a suffix is unambiguous and trivial to parse, and anyone who wants to use a different encoding can easily plug in a type-specific marshaler. |
This is the strangest thing to say when talking about an interoperability medium. Honestly sometimes it seems that go devs just live in a go walled garden. Interoparability is the main concern of JSON. |
To be fair, I guess I'm technically a Go developer, and while I earlier proposed use of My main concern with "fractional seconds and a suffix" is that nothing else (to my knowledge) uses this format except ProtoJSON, which is arguably a small use case since most protobuf messages are serialized using the binary wire format. OpenAPI, which is a relatively well-used system for JSON-based APIs uses ISO 8601 for time durations. It's on my TODO list to understand how it handles years, months, days, etc.
ISO 8601 seems to distinguish between "accurate" versus "nominal" duration values. Thus, this split of treating hours, minutes, seconds differently is at least called out within the standard:
I'm aware of at least some protocols that claim support for ISO 8601 and only accept hour, minute, and seconds. |
|
ISO8601 OpenAPI defines durations as ISO8601, sourced from JSON schema:
RFC3339 Appendix A includes support for civil time periods (years/months/days/week). I think any use of ISO8601 would need a good answer for how to handle civil time periods. Unlike Proto, JSON doesn't have type information so the form of the value may be used to indicate the type of data (eg, ISO8601 => civil time periods). Umarshalling into a If Go gains a Compatibility For me this is less about being compatible with ProtoJSON, and more about providing clear intent when looking at the encoded JSON. From what I've seen, many different encodings are used across the JSON ecosystem - with numeric encodings being most common (floating point/integer seconds/milliseconds/microseconds), but these are all problematic since the type is unclear looking at the numeric value itself. From a design POV, I think it would be clearer for real-time durations and civil time periods to be encoded differently. Fractional seconds + "s" is unambiguous and easier to encode/decode if you need to write it yourself. However, if the ecosystem is actively moving towards ISO8601 for real-time durations despite the ambiguity, then maybe aiming for "future" compatibility and throwing an error is the best we can do. ISO8601 durations seem problematic and it's not clear to me they are the "future" for JSON, so I'd lean towards the clearer encoding. That said, I can totally see people could come to a different conclusion about where this is heading. |
|
OpenAPI's reference of RFC 3339, Appendix A is problematic because RFC 3339 itself disavows being an authoritative grammar by saying: "This is informational only and may contain errors." In fact, it does contain errors. The grammatical construction makes it such that
I believe this is where we're fundamentally in disagreement (which is fine). I believe that compatibility with the wider ecosystem is more important than clarity. Clarity is a good goal (and it's entirely reasonable for others to hold to this), but at least for JSON, I believe interoperability is more important since JSON has become the de-facto langua fraca of how many machines or programs communicate with each other. Also, ISO 8601 isn't notably "unclear" relative to "fractional seconds and a suffix". For example, if I see "PT1M", it's guessable that it might mean "1 minute". Asking ChatGPT, Gemini, or Grok what "PT1M" means all results in them identifying it correctly as ISO 8601. (To be honest, I like the simplicity and aesthetics of Go's
I may be wrong, but I've been seeing increasingly more evidence of this being functionally true (e.g., Let's suppose an RFC one day exists that codifies a concise grammar. I can see a path forward where the Internet can actually stabilize on an interoperable format. I suspect:
|
|
I think the ISO-8601 definition is sufficiently clear enough to settle production: the This really only leaves a problem when we are decoded a Anyone who needs to support such time periods can be left to implement their own… or submit their own proposal to provide a solution to the standard library. (I recall that I’ve had to support this before.) |
|
Slightly off topic, but given the low voter turnout for the "base60" format (e.g., 👍 if you agree (remove it from v2), 👎 if you disagree (keep it in v2). |
|
SGTM with the assumption that we can always add base60 as an opt-in format later. |
|
XML duration turns out to actually allow fractional seconds, so From a cursory glance at some parsers, it seems some would accept a value like That may allow the use of the ISO-style format for serialisation, without the perils of civil time, if Go were to constraint the serialisation of |
|
Note that time.Duration.String returns "µs" instead of "us" (which time.ParseDuration accepts), which might be annoying (or impossible) to handle manually in another language. It would be nice to always produce "us" instead. |
I think the current intent is that we never generate either |
|
The debate regarding using To help resolve this, we (with the help of @timbray) wrote a RFC Internet-Draft to standardize on a concise grammar for durations based on ISO 8601 that we can circulate more widely outside the Go community. Regardless of where that effort goes, the wider reaction to the RFC draft will be helpful:
|
|
I would add that there is also a much prettier HTML version of the Internet-Draft at https://www.ietf.org/archive/id/draft-tsai-duration-00.html |
By popular demand (or rather lack thereof), remove support for the base60 representation of a time duration. There is no standard for this format and even the name is an attempt at naming something without clear industry basis for what this is called. Updates golang/go#71631
|
Great RFC; two notes: Typo? It seems there’s an extraneous
😂 Love this example, because it depends on 2000 being a leap year in the Gregorian calendar, which was a common Y2K bug that often made it into the year 2000. I also just had to go double-check, and from the sources I found, I can confirm there were no leap seconds in that interval. |
This comment has been minimized.
This comment has been minimized.
By popular demand (or rather lack thereof), remove support for the base60 representation of a time duration. There is no standard for this format and even the name is an attempt at naming something without clear industry basis for what this is called. Updates golang/go#71631
Proposal Details
This is a sub-issue of the "encoding/json/v2" proposal (#71497).
Here, we discuss the default JSON representation of
time.Duration. In #71497, we proposed using thetime.Duration.Stringrepresentation and continue to propose using that format, but want to provide an avenue for the community to find concensus on the right format if it should be something else. To be clear, v2 supports multiple formats, but the focus of this issue is determining what the right default is.The following is an overview of reasonable representations of a duration and their strengths and weaknesses. There is no obviously right approach. All have flaws of some kind. A major part of the problem is the lack of a standard like RFC 3339 for representing time durations.
JSON integer in nanoseconds (e.g.,
47683228428000)JSON fractional number in seconds (e.g.,
47683.228428)time.Duration(strconv.ParseFloat("0.000000015", 64)*1e9)results in 14ns (rather than the expected 15ns).time.Duration(math.Round(strconv.ParseFloat(...)*1e9)), then they get the right answer more often, but still run into precision errors eventually (~104 days). Also, the more correct parsing is less intuitive than the more trivial parsing.0.000000001as opposed to"1ns"fromtime.Duration.String).time.Durationusing v1 "encoding/json" may not result in an error, leading to silent data corruption. For example,1is a valid representation for 1 second, but parsing this intotime.Durationwith v1 will result in 1 nanosecond without an error. This may lead to significant interoperability issues between v1 and v2.JSON string using
time.Duration.String(e.g.,"13h14m43.228428s")"1ps"for 1 picosecond) or by using more fractional digits.time.Durationand there is argument for consistency.JSON string using ISO 8601 (e.g.,
"PT13H14M43.228428S")time.Time, which uses RFC 3339, which is a subset of ISO 8601.time.Durationsince the length of a year is ill-defined. At best, Go can use a subset of TC39's grammar that's constrained to only the units like hours, minutes, and seconds."PT0.000000001S"as opposed to"1ns"fromtime.Duration.String).JSON string using base60 representation (e.g.,
"13:14:43.228428")"0:00:00.000000001"as opposed to"1ns"fromtime.Duration.String).The text was updated successfully, but these errors were encountered: