URI normalization

[jdesrosiers]

In the #1618 review discussion, I noticed that we may not be doing the right thing regarding URI normalization. RFC 3987 says,

Applications using IRIs as identity tokens with no relationship to a protocol MUST use the Simple String Comparison (see section 5.3.1). All other applications MUST select one of the comparison practices from the Comparison Ladder (see section 5.3

JSON Schema describes our use of IRIs as identity tokens, not locators, which means implementations should limit themselves to Simple String Comparison. That's not what our spec currently says. It says to use the normalization procedure defined in 5.3, which isn't quite correct either. We'd need to be more specific about which comparisons to use. For example, we wouldn't require scheme-based or protocol-based normalization because it would be too onerous a requirement for implementers to implement scheme-based requirements for any scheme they might encounter.

[riyap03]

Hi @json-schema-org, I’d like to work on this issue to clarify the correct IRI comparison for identity tokens in JSON Schema. I plan to update the spec to use Simple String Comparison as recommended by RFC 3987. @json-schema-org claim 💙

[notEthan]

This seems like a potentially significantly breaking change, anywhere schemas use URIs that normalize the same but are not string-equal.

I do agree requiring normalization for any scheme an implementation might encounter would be onerous. Though most implementations probably rely on a library for most URI functions including normalization and comparison, and don't spend a lot of time on those implementation details. (I do anyway; I don't have data but would suspect most do.)

Normalizing just http(s) is a thought, and simple comparison of other schemes. That would apply the the same normalization as always to the great majority of schemas that might be affected. I don't really like that, adding conditions to comparison adds complexity - less complexity than normalizing any scheme there may be, but more than necessary.

Yeah, simple string comparison is probably better. Upgrading schemas from one spec version to the next always takes some conversion, and ensuring URIs are string-equal can be part of that conversion.

[notEthan]

I keep typing URI out of habit. IRI.

[riyap03]

Thank you for pointing this out 💙 I understand the concern about this being a potentially breaking change, since implementations that relied on normalization may see different behavior. I agree that adding special cases like only normalizing http(s) would increase complexity without much long-term benefit.

Given that JSON Schema uses IRIs purely as identity tokens (not locators), I still feel Simple String Comparison aligns best with RFC 3987 and keeps the spec more consistent and easier to implement. And as you mentioned, upgrades between spec versions usually involve some migration effort, so ensuring strict string-equality of IRIs can reasonably be part of that process.

I’ll be happy to adjust wording if needed, but my current view is that Simple String Comparison is the clearer and less error-prone path forward. ✨

[gregsdennis]

I still feel Simple String Comparison ... keeps the spec more consistent and easier to implement.

As a counterpoint, many languages already have functionality (either built in or through packages) that handle IRI equivalence. Simple string comparison isn't necessarily easier to implement. Just as easy, perhaps...

That said, I don't think the desired behavior is that two schemas can share the same IRI, differing only by character casing or encoding, or maybe one has a query string. That's unexpected.

[jdesrosiers]

I've been doing some reading and trying to think about why URIs/IRIs are defined this way. Why "MUST" applications using IRIs as identity tokens only use string comparison?

I think the concept behind using IRIs as identity tokens is that they are entirely opaque. They're shaped like IRIs, but have none of the semantics of IRIs. That includes the standard parts like scheme, path, fragment, etc. So constructing an IRI from parts of an identity token (like resolving a relative IRI or simply adding a fragment) doesn't make sense because the identity token doesn't have a scheme or path or fragment, just strings that look like those things.

That, I think, is why it says we MUST use string comparison for identity tokens. They're really just strings and don't have the semantics of normal URIs/IRIs, so they MUST be treated as just strings. That's a problem for JSON Schema because we rely on our IRI parts having IRI semantics so we can use relative references (including local references). If we say IRIs in JSON Schema are identity tokens, then we can't have relative IRIs including fragments for referencing subschemas. (Technically, we could still have fragments, but they would be part of the document identification, the not subschema identification.)

I think we're running up against this conflict with RFC 3986/7 because we're misapplying the concept of identity tokens to JSON Schema. I think we can fix this in the spec without meaningfully changing the consensus on how JSON Schema should be implemented and used. IRIs could be defined as locators, which means their parts are all relevant, including the scheme and its locating semantics. But, we continue to recommend that implementations provide a way to load schemas and recommend that by default they don't retrieve schemas from anywhere other than the schemas provided. The schema registry is effectively a permanent cache layer that's loaded manually rather than caching what's retrieved. Users can still load schemas using whatever scheme IRIs they want, it's compatible with implementations that retrieve schemas for the network or file system based on scheme, and it's compatible with RFC 3967 (IRIs). Nothing functionally needs to change. It's just a change in the way we describe how it works in the spec (and probably a change in the way we talk about it).

Assuming we get consensus on that solution, the next question would be what normalization strategy would we require? RFC 3967 (IRIs) is clear that we MUST select one, which we don't right now. As others have pointed out, most implementations will use an off-the-shelf IRI library and making too strict normalization requirements might require some implementations to write custom IRI parsers/resolvers, which I don't think we want to do. I think it's reasonable to expect that off-the-shelf libraries will at least include Syntax normalization. I think we can require Syntax normalization and make the other two optional. Then make it a best practice to not rely on Scheme and/or Protocol normalization when authoring schemas. It shouldn't hurt if implementations support it, but not all will.

[notEthan]

I had sort of skimmed past it before, but:

JSON Schema describes our use of IRIs as identity tokens, not locators

I'm not seeing that the spec does this. The closest I see is in $id

Note that this IRI is an identifier and not necessarily a network locator.

(and similar language in $ref) but that doesn't sound like 'identity token' to me, that's just that it's an IRI and not a URL. And as you point out, the requirements to resolve relative references and parse fragments already aren't consistent with an opaque identity token.

Is it a locator? That debate has come up before, I recall, whether matching $ref to $id in a registry is 'locating' behavior. I don't see a significant distinction in practice for implementation, as long as relative references and fragments get resolved.

The question of normalization is a significant implementation requirement. Looking through the pages of different normalizations in 3986§5.3, it's hard to say the right rung of that ladder.

I think we can require Syntax normalization and make the other two optional. Then make it a best practice to not rely on Scheme and/or Protocol normalization when authoring schemas. It shouldn't hurt if implementations support it, but not all will.

Syntax normalization seems like a good minimum. I'm not sure about optional other rungs, that leads to schemas having different behaviors in different implementations, which is preferable to minimize. Maybe just specifying syntax normalization is good.

[jdesrosiers]

that doesn't sound like 'identity token' to me, that's just that it's an IRI and not a URL.

There's some vocabulary confusion going on here. Both the URI (IETF) spec and the URL (WHATWG) spec define that they can be used as locators or pure identifiers. The only difference is that they couldn't agree whether to call these them URIs or URLs. IETF says they should always be called URIs whether they are locators or just identifiers. WHATWG says they should always be called URLs whether they are locators or just identifiers. But, since it sounds logical, people think that URI means it's an identifier only and URL means it's a locator. Neither spec makes that distinction. RFC 3986/7 uses the term "identity token" to describe URIs that are identifier-only.

The JSON Schema spec doesn't use "URI" because its URIs are identifier-only, it uses "URI" because we use the URI spec (RFC 3986/7). Even if we make a change to consider them locators, we'll still call them "URIs" in the spec because we're referencing the URI spec, not the URL spec. We don't use the term "identity token" in the JSON Schema spec, but it's the same concept as what you are calling "URI".

(Note: I use the term URI throughout, but the same applies to IRI. Also, URLs (WHATWG) are internationalized, so they compare closer to IRI (RFC 3987) than URI (RFC 3986).)

[jdesrosiers]

Syntax normalization seems like a good minimum. I'm not sure about optional other rungs, that leads to schemas having different behaviors in different implementations, which is preferable to minimize. Maybe just specifying syntax normalization is good.

Yes, allowing for syntax and protocol normalization could change the way validators behave. But, you don't seem to be considering the argument of allowing implementations to use the standard library available in their language. We make the same allowance for regular expressions. We don't expect implementers to implement a custom regex engine so that all regular expressions are guaranteed to evaluate the same way. We accept that some implementations will support regex syntax that others done and advise that schema authors stick to subset likely to be supported everywhere.

I don't know how much variation there is in normalization done by standard library URI implementations. It many be that all or almost all do the same thing and allowing normalization other than Syntax isn't necessary. @riyap03, you've shown interest in helping out with this issue. Would you be interested in doing some research to find out what normalization standard library implementations in different languages actually do?

[riyap03]

Hey @jdesrosiers 💙, this makes a lot of sense. Letting implementations lean on their language’s standard library feels right—like how we handle regex, we don’t force everyone to reinvent the wheel.

I can dig into what normalization steps standard URI/IRI libraries actually do across popular languages. That way, we can set a minimum requirement, like syntax normalization, without over-complicating things or breaking implementations.

Once we have that, the spec can be clear about what’s mandatory vs. optional, keeping behavior consistent but still practical. I can start gathering the list of languages/libraries if that helps.

[notEthan]

@jdesrosiers I feel like you are dividing IRIs beween 'locator'/URL and 'identity token', but I don't think that's apt. Please excuse if I'm misreading that - as you say, we have some vocabulary confusion occurring and I'm not following all of how you're stating these. I'm going to back up and say my understanding of these terms from basics, hopefully you can tell me where we are misaligned.

An IRI is an identifier, used in JSON schema to identify a schema resource.

URL is a subset of IRI that is used to locate an identified resource; a URL is an identifier and a locator. For JSON schema, locating behavior (network retrieval etc) that requires using a schema IRI as a URL isn't recommended by the spec. Maybe registry lookup could be be considered locating behavior (I would say not, but debatable); either way IRI is a preferable term.

An identity token is an opaque string, compared by simple string comparison. An IRI can be an identity token, when used that way. This is not what JSON schema specifies, since it requires normalization, relative reference resolution, and fragment parsing. You say "JSON Schema describes our use of IRIs as identity tokens" but I (still) don't see that.

So a schema has an IRI that is not a URL (not typically used for locating) and is not an identity token (is normalized, resolved if relative). To my reading, this seems consistent with what's in the spec now.

---

Normalization, yeah, requiring syntax normalization, but allowing whatever else an implementation might implement, that seems good to me. (Looking at the URI library I rely on, it just has one normalize method with no options.)

[jdesrosiers]

You're saying that an identity token is just a kind of identifier-only URI/IRI and that it's possible to be an identifier-only URI and not be an identity token. That's where we disagree. The IRI spec uses "identity token" as a shorthand for "an identifier-only URI/IRI".

There is no term for an identifier-only URI/IRI. If I say something is a URI/IRI, it means that it's an identifier and might also be locator. The spec authors used the term "identity token" to describe an identifier-only URI. It's not an official term. It's a description. It's not introducing a new and distinct concept from an identifier.

Whether or not you agree with the premise that these terms are equivalent, hopefully you can at least see where my assertion about the spec is coming from. JSON Schema describes its use of IRIs as identifier-only, which is the same as saying it uses IRIs as identifier tokens, which the IRI spec necessitates string comparison for.

If you disagree, please provide an argument why you think "identity token" means something other than "identifier-only URI" in the IRI spec.