proposal: encoding/json/v2: add support for user-defined format flags and option values

[dsnet]

Proposal Details

This is a sub-issue of the "encoding/json/v2" proposal (#71497).

Here we propose additional API to support user-defined format flags and option values.
This builds on top of the v2 API and does not block the acceptance of v2.

package json // encoding/json/v2

// WithFormat constructs an option specifying the format for type T,
// which must be a concrete (i.e., non-interface) type.
// The format alters the default representation of certain types
// (see [Marshal] and [Unmarshal] for which format flags are supported).
// Later occurrences of a format option for a particular type override
// prior occurrences of a format option of the exact same type.
//
// For example, to specify that [time.Time] types format as
// a JSON number of seconds since the Unix epoch:
//
//	opts := json.WithFormat[time.Time]("unix")
//
// The format can be retrieved using:
//
//	v, ok := json.GetOption(opts, json.WithFormat[MyType])
//
// The format option is automatically provided when a format flag
// is specified on a Go struct field, but is only present for
// the current JSON nesting depth.
// 
// For example, suppose we marshal a value of this type:
//
//  type MyStruct struct {
//      MyField MyType `json:",format:CustomFormat"`
//  }
//
// and the "json" package calls a custom [MarshalerTo] method:
//
//  func (MyType) MarshalJSONTo(enc *jsontext.Encoder, opts json.Options) error {
//      // Check whether any format is specified.
//      // Assuming this is called within the context of MyStruct.MyField,
//      // this reports "CustomFormat".
//      ... := json.GetOption(opts, json.WithFormat[MyType])
//
//      // Begin encoding of a JSON object.
//      ... := enc.WriteToken(jsontext.ObjectStart)
//
//      // Checking the format after starting a JSON object does not
//      // report "CustomFormat" since the nesting depth has changed.
//      // It may still report a format if WithFormat[MyType](...)
//      // was provided to the top-level marshal call.
//      ... := json.GetOption(opts, json.WithFormat[MyType])
//
//      // End encoding of a JSON object.
//      ... := enc.WriteToken(jsontext.ObjectStart)
//
//      // Checking the format reports "CustomFormat" again
//      // since the encoder is back at the original depth.
//      ... := json.GetOption(opts, json.WithFormat[MyType])
//
//      ...
//  }
//
// The format flag on a Go struct field takes precedence
// over any caller-specified format options.
//
// [WithFormat] and [WithOption] both support user-defined options,
// but the former can only represent options as a Go string,
// while the latter can represent arbitrary structured Go values.
func WithFormat[T any](v string) Options

// WithOption constructs a user-defined option value.
// The type T must be a declared, concrete (i.e., non-interface) type
// in a package or a pointer to such a type.
// Later occurrences of an option for a particular type override
// prior occurrences of an option of the exact same type.
//
// A user-defined option can be constructed using:
//
//	var v MyOptionsType = ...
//	opts := json.WithOption(v)
//
// The option value can be retrieved using:
//
//	v, ok := json.GetOption(opts, json.WithOption[MyOptionsType])
//
// User-defined options do not affect the default JSON representation
// of any type and is only intended to alter the representation for
// user-defined types with custom JSON representation.
//
// [WithOption] and [WithFormat] both support user-defined options,
// but the former can represent arbitrary structured Go values,
// while the latter can only represent options as a Go string.
func WithOption[T any](v T) Options

Example third-party package that supports custom formats:

package geo

// Coordinate represents a position on the earth.
type Coordinate struct { ... }

func (c Coordinate) MarshalJSONTo(enc *jsontext.Encoder, opts json.Options) error {
    format, _ := json.GetOption(opts, json.WithFormat[Coordinate])
    switch format {
    case DecimalDegrees: ...
    case PlusCodes:      ...
    case ...
    }
}

const (
    // DecimalDegrees formats a coordinate as decimal degrees.
    // E.g., "40.7128, -74.0060"
    DecimalDegrees = "DecimalDegrees" 

    // PlusCodes formats a coordinate as a plus code.
    // E.g., "87C8P3MM+XX"
    PlusCodes = "PlusCodes"

    ...
)

Example usage of custom formats supported by the geo package:

// Marshal a map of coordinates where each coordinate uses PlusCodes.
var locations map[string]geo.Coordinate = ...
json.Marshal(locations, json.WithFormat[geo.Coordinate](geo.PlusCodes))

// Marshal a Go struct with a field of a Coordinate type
// such that the field uses DecimalDegrees.
var person struct {
    Name     string 
    Location geo.Coordinate `json:",format:DecimalDegrees"`
}
json.Marshal(person)

Example third-party package that supports custom options:

package protojson

// MarshalOptions contains options to alter marshaling behavior
// specific to protobuf messages.
type MarshalOptions struct {
    AllowPartial bool
    UseProtoNames bool
    UseEnumNumbers bool
    ...
}

// MarshalEncode encodes message m to the provided JSON encoder e.
func MarshalEncode(e *jsontext.Encoder, m proto.Message, opts json.Options) error {
    protoOpts, _ := json.GetOption(opts, json.WithOption[MarshalOptions])
    ... // alter representation of protobuf message according to protoOpts 
}

Example usage of custom options supported by the protojson package:

var messages map[string]proto.Message
json.Marshal(messages,
    json.WithMarshalers(json.MarshalToFunc(protojson.MarshalEncode)),
    json.WithOption(protojson.MarshalOptions{AllowPartial: true},
))

Both WithFormat and WithOption support some way for user-defined options to alter the representation of options.

For now, we do not support interface types as it is unclear whether retrieval of an option (e.g., GetOption(opts, json.WithOption[MyType])) should also check whether any options are set for other interface types that MyType also implements. Trying to construct an option of an interface type panics. In the future, this restriction can be lifted, but allows providing value sooner for a suspected majority of use cases.

Alternatives considered

Instead of WithOption, we could consider making json.Option an interface that could be implemented by any declared type (i.e., all exported methods). However, it is unclear how the "json" package would merge options together and how to retrieve the custom options type back out of a combined json.Options. Also, this is more boilerplate for every user since they would need to implement at least one method to implement json.Option. The proposed WithValue API avoids unnecessary boilerplate for the user and only requires that the user declare a type, but no extra machinery.

An alternative API is to make the signature of WithValue similar to context.WithValue which accepts a user-provided key and value. In order to prevent conflicts between keys, the API requires that the key be a user-defined type. However, if we are going to require that, why not just make the user-defined value type the key itself? If so, we're back to a solution similar to the currently proposed WithValue API.

[mitar]

Thanks for opening this. Just for the record, while I support use cases and proposal above, I would prefer if custom marshal implementation could access full struct tags for a field currently being marshaled instead of just format (it is OK if format is exposed through additional/special flag because it is the most common case).

[dsnet]

prefer if custom marshal implementation could access full struct tags for a field

I'm not sure I understand why this is needed. Most of the field tag options are specific to the serialization of the parent Go struct and the child field type should not care.

The current set of all proposed tag options are omitzero, omitempty, string, case, inline, unknown, and format:

• omitzero and omitempty are evaluated in the context of the parent
• string is already forward via the StringifyNumbers option
• case only matters with regard to name of JSON member in the context of the parent
• inline and unknown currently only matter in the context of the parent, but could consider allowing types with custom implementations to be inline-able
• format is being forwarded with the proposed WithFormat option in this issue

[mitar]

The use case is that I could then do something like:

type Person struct {
	Name string `json:"name"`
	Surname string `json:"surname"`
	SSN string `json:"ssn,omitempty" private:""`
}

json.Marshal(Person{...},
	json.WithMarshalers(json.JoinMarshalers(
		json.MarshalToFunc(func(enc *jsontext.Encoder, _ string, opts json.Options) error {
			if _, private := opts.Tag().Lookup("private"); private {
				return enc.WriteToken(jsontext.String(""))
			}
			return json.SkipFunc
		}),
	)),
)

Here I use opts.Tag() to access reflect.StructTag value.

(BTW, I support that options would simply be accessible through enc.Options().)

If in the future context could be accessible during marshaling, then I could make private fields conditioned on the current user or user's permissions.

[dsnet]

I see, I was under the impression, access to struct tags was for the JSON-specific tag options, but your goal is to access user-specific information in the tags (in general). It's an interesting idea.

Technically, you could fold that information in the format tag option, but might be an overloading of what format was intended for.

[willfaught]

I like everything but the names. WithOption is odd because it's not the only func that returns an Option. The "With" parts are odd because they're not adding an option to an existing Options. TypeFormat[T any](string) Options and TypeValue[T any](any) Options seem clearer, in my humble opinion.