first complete pass at resolution/dereference clean up. still draft.#311
first complete pass at resolution/dereference clean up. still draft.#311
Conversation
|
Sorry to be a pain @jandrieu , but it looks to me like your editor has reformatted all of the HTML in the spec, making it pretty much impossible to determine what you have intentionally changed -- separating out reformatting from content changes. Would it be possible for you to identify the sections that you have substantially changed? Perhaps in the list of "What I changed" add a "sections impacted" detail. |
Can you speak to these two points some more. I do agree that the Resolver may need DID URL parameters that the dereferencer is unaware of. E.g. either the dereferencer calls I think you are saying that in this case |
To fix the terminology you have to add terms to the terms.html, e.g. see DID Resolution Result. https://github.com/w3c/did-resolution/blob/main/terms.html#L24 Then you can reference that term using a tags e.g. https://github.com/w3c/did-resolution/blob/main/index.html#L2086 |
Yes, I agree. Very difficult to follow. It looks like your formatting is enforcing the 80 char line length, plus some other additional newlines. One option could be, if the group agrees with this formatting, we push a change that only formats the spec. Then rebase after that is merged, that should identify what has actually changed. Ideally, this repo should define and enforce formatting. Rather than letting different code editors dictate based on their settings. |
| Using a [[DID URL]] is similar to any other URL: you first resolve | ||
| the identifier to get required metadata, then you use that | ||
| metadata to retrieve the resource using that metadata. For | ||
| DNS-based HTTP urls, you use DNS to get an IP address, then use | ||
| the IP address to query a webserver and retrieve a representation | ||
| of the resource. For DID URLs—and all DIDs are DID | ||
| URLs— you first resolve the DID URL to get the authoritative | ||
| DID document, then use that DID document to retrieve the | ||
| referenced resource. The process starts with a [[DID User]] who | ||
| has a DID URL they would like to dereference; the first step is | ||
| dereferencing is resolution. |
There was a problem hiding this comment.
| Using a [[DID URL]] is similar to any other URL: you first resolve | |
| the identifier to get required metadata, then you use that | |
| metadata to retrieve the resource using that metadata. For | |
| DNS-based HTTP urls, you use DNS to get an IP address, then use | |
| the IP address to query a webserver and retrieve a representation | |
| of the resource. For DID URLs—and all DIDs are DID | |
| URLs— you first resolve the DID URL to get the authoritative | |
| DID document, then use that DID document to retrieve the | |
| referenced resource. The process starts with a [[DID User]] who | |
| has a DID URL they would like to dereference; the first step is | |
| dereferencing is resolution. | |
| <p> | |
| Using a [[DID URL]] is similar to any other URL: you first resolve | |
| the identifier to get required metadata, then you use that | |
| metadata to retrieve the resource using that metadata. For | |
| DNS-based HTTP urls, you use DNS to get an IP address, then use | |
| the IP address to query a webserver and retrieve a representation | |
| of the resource. For DID URLs—and all DIDs are DID | |
| URLs— you first resolve the DID URL to get the authoritative | |
| DID document, then use that DID document to retrieve the | |
| referenced resource. The process starts with a [[DID User]] who | |
| has a DID URL they would like to dereference; the first step is | |
| dereferencing is resolution. | |
| </p> |
All this text should be in <p> tags I believe. Probably easier for you to address locally.
There was a problem hiding this comment.
@wip-abramson -- It looks like you've got something between be in and tags I believe which should be codefenced, like `<intended-tag>` instead of <intended-tag>, which is invisible to us without you doing that.
There was a problem hiding this comment.
I would first apply @wip-abramson's suggestion, but if I do that here, or address the line breaks, GitHub fails to properly highlight my own changes.
| Using a [[DID URL]] is similar to any other URL: you first resolve | |
| the identifier to get required metadata, then you use that | |
| metadata to retrieve the resource using that metadata. For | |
| DNS-based HTTP urls, you use DNS to get an IP address, then use | |
| the IP address to query a webserver and retrieve a representation | |
| of the resource. For DID URLs—and all DIDs are DID | |
| URLs— you first resolve the DID URL to get the authoritative | |
| DID document, then use that DID document to retrieve the | |
| referenced resource. The process starts with a [[DID User]] who | |
| has a DID URL they would like to dereference; the first step is | |
| dereferencing is resolution. | |
| Using a [[DID URL]] is similar to using any other URL: you first resolve | |
| the identifier to get required metadata, then you use that | |
| metadata to retrieve the resource. For | |
| DNS-based HTTP URLs, you use DNS to get an IP address, then use | |
| the IP address to query a web server and retrieve a representation | |
| of the resource. For DID URLs — and all DIDs are DID | |
| URLs — you first resolve the DID URL to get the authoritative | |
| DID document, then use that DID document to retrieve the | |
| referenced resource. The process starts with a [[DID User]] who | |
| has a DID URL they would like to dereference; the first step in | |
| dereferencing is resolution. |
A couple of comments on this. First, we can do this in CR. Second, what are the normative statements we would actually need to test. Noting that Meaning any normative MUST statements conditional on these parameters do not need to be tested per W3C requirements from what I am aware. @msporny can you confirm. So looking over the current algorithm in this PR that leaves
Now this feels like a bit of a cop out, but just wanted to point it out. I initially developed tests for DID Parameters, but then removed them due to the fact that they were optional. |
| handling the DID's method. This could be a [[remote | ||
| resolver]], which would use HTTPS for communications, or |
There was a problem hiding this comment.
| handling the DID's method. This could be a [[remote | |
| resolver]], which would use HTTPS for communications, or | |
| handling the DID's method. This could be a <a>remote resolver</a>, | |
| which would use HTTPS for communications, or |
All term references should use this pattern. Or maybe [remote resolver] (not sure). Definitely a tag works
There was a problem hiding this comment.
[= ... =] is equivalent to <a> ... </a> for linking to a term defn.
| handling the DID's method. This could be a [[remote | |
| resolver]], which would use HTTPS for communications, or | |
| handling the DID's method. This could be a [=remote | |
| resolver=], which would use HTTPS for communications, or |
| URL to create a [[Resolution DID URL]], e.g., | ||
| `did:ex:abc#key-1` becomes `did:ex:abc` .</li> | ||
| <li><strong>Prepare</strong> <var><a>resolution | ||
| options</a></var>. See the [[resolve]] algorithm for |
There was a problem hiding this comment.
| options</a></var>. See the [[resolve]] algorithm for | |
| options</a></var>. See the <a href="#resolve">resolve</a> algorithm for |
| URL, stop further processing and return an | ||
| error.</p> |
There was a problem hiding this comment.
Should specify what error I think in the https://w3c.github.io/did-resolution/#errors
Maybe it is INVALID_OPTIONS.
Or we define INVALID_PARAMETERS
There was a problem hiding this comment.
In this case, I've removed dereferencing as a function that returns any particular error. Dereferencing is an algorithm that any DID URL Client executes. What it does with the error is context dependent.
Hopefully this will be clearer in the new PR.
| property matches the value of the <code>serviceType</code> DID parameter.</li> | ||
| </ol> | ||
| The selected <a data-cite="did-core#services">services</a> are a list called the <var>selected <a>services</a></var>. | ||
| <li><strong>Service</strong> If <a>service parameter</a> is present as a DID |
There was a problem hiding this comment.
| <li><strong>Service</strong> If <a>service parameter</a> is present as a DID | |
| <li><strong>Service</strong> If <code>service</code> parameter is present as a DID |
Previous style has used the above.
I dont think we want service parameter as a term. We could link to its definition in the table perhaps.
There was a problem hiding this comment.
That service parameter does link to a definition. But, as I mentioned, the terms.html file is under utilized, so I deferred to the editors on that. We're going to need a vocabulary pass to pull the definitions together. And where possible, we should leverage xref for terms in other specs. This latter part: deciding between xref versus local definitions is where I am hoping the editors provide guidance.
| <strong>Service Type</strong> | ||
| If <a>serviceType parameter</a> is present as a DID | ||
| Parameter, use the <a>service-type selection | ||
| algorithm</a> to select the appropriate service | ||
| object for | ||
| retrieval. The retrieval strategy is determined by | ||
| the type property of the selected service.</li> | ||
| <li> | ||
| <strong>Property</strong> | ||
| If the DID document contains any properties known to |
There was a problem hiding this comment.
Thinking this through in terms of the PathService type.
This does not seem to capture that case, unless the DID URL explicitly includes serviceType=PathService. Is that reading correct
Or would a service with a type of PathService be also included in the property handling
There was a problem hiding this comment.
Under the current spec, serviceType=PathService is supported. That's what this explains.
A service of type PathService is not currently included in property handling if there is no parameter in the URL to specify that service.
The current PR from @swcurran is one way to address that, but at the cost of deprecating other property-based mechanisms in a manner that is unacceptable.
| determined by that property. For example, | ||
| linkedResource property uses its own retrieval | ||
| strategy as described at <a | ||
| href="https://www.w3.org/TR/did-extensions-properties/#linkedresource">Linked | ||
| Resource</a> in the [[DID-EXTENSIONS-PROPERTIES]] | ||
| specification. If the did document contains multiple properties that affect |
There was a problem hiding this comment.
If we go down this route I feel like we have to be more explicit somewhere about what it means to register an extension with a retrieval strategy.
E.g. if this text didn't explictly call out linked resource, I am not sure as an implementer from the extensions registry I would know this property defines a retrieval strategy. Even from the linked spec, it is not obvious.
Maybe this would be a different section in the extensions, or something explicit specs have to state and define. E.g. a retrival strategy algorithm.
There was a problem hiding this comment.
Agreed. This is wrapped up in @swcurran's proposed changes.
IMO, the services that affect path handling should add a property to the type array, e.g., "PathHandler" so that all such services can be evaluated at run time even if the detailed type is unknown to the DID URL Client.
But this PR is not trying to fix that problem. I'm just upgrading the current document to clarify resolution from dereferencing in a way that is consistent with how the rest of the web dereferences URLs.
| retrieval. The retrieval strategy is determined by | ||
| the type property of the selected service.</li> |
There was a problem hiding this comment.
I think this is implied, but may want to make explicit.
The selected service or services may not specify any retrieval algorithm.
E.g. they may be of the type SMTBeacon.
In this case, what I want to dereference to is the array of all services in the DID document that are of this type.
I think this is not fully handled currently
There was a problem hiding this comment.
Ummm...
Isn't this text making this explicit?
If your service type doesn't tell you how to use the properties of the service, it is underspecified. It is this statement in this specification that makes it explicit that such service definitions are responsible for clarifying the retrieval strategy.
I think it's useful to have an HTTPS binding for dereferencing DID URLs. This enables use cases such as the ones shown in #306 (comment). But dereference(), just like resolve() is an abstract function. Their algorithms can be implemented by client-side libraries or SDKs. In fact, I would argue that DID URL dereferencing is "even more client-side" than HTTPS URL dereferencing, since in some cases you can fully dereference a DID URL (including parameters such as "service" or "relativeRef") without ever calling a remote endpoint, which is something that is not possible with HTTPS URLs. |
I know you're going to hate this, but there are DID URLs that you could theoretically dereference without ever resolving the DID document, e.g. Note however that the current DID URL dereferencing algorithm DOES require resolution of the DID document as part of DID URL dereferencing, and I think that's fine: https://www.w3.org/TR/did-resolution/#resolving-algorithm |
No need to pass a DID URL to resolution, see this sentence in the specification:
|
I'm very surprised to hear this, didn't you just propose the opposite in #303 ? |
If I understand correctly, the concept of a "retrieval strategy" is a more modular, structured approach to what is currently a sequence of "if .. then .." statements in the DID URL Dereferencing section. I think this could be a good idea. I'm however concerned about the term "retrieval strategy", since to me that term sounds like something is being retrieved from a remote location, which isn't always the case, e.g. with I think some aspects in the "retrieval strategy" approach break what we have today. E.g. I'm not sure if your current PR properly supports DID URLs that have a "service" and "relativeRef" parameter. |
Good idea. I think this is pretty much implied in the current specification, but can't hurt to explicitly point it out. |
I don't like this term at all, it breaks the clear delineation we have had so far between "resolving a DID" and "dereferencing a DID URL". It's true that in the current specification, we don't really have a term for "the DID URL minus the fragment". I agree it could be useful to have a term for that, but I don't know what it could be. Right now, the specification simply says this:
|
|
This was discussed during yesterday's call |
Yes, confirmed. If the initial normative statement in a chain of normative statements isn't a MUST, then everything after it is optional and therefore doesn't need to be tested. That said, if there A LOT of normative statements that hang off of that single SHOULD/MAY, then it's a recipe for non-interoperability. The group can always decide to test some SHOULD statements, but implementers often object to tests that are not absolutely required for interop showing up in the WG test suite. |
|
Thanks for the feedback. Given the difficulty reviewing in Github, I'll make a new PR ASAP that not only cleans up the unnecessary format changes, but incorporates some new language to address some of the continued confusion. |
|
MUST statements SHOULD be tested. (See what I did there? Some MUSTs are not testable, and can only be judged as implemented and interoperable based on implementer assertion. This is suboptimal, but so is much of life.) SHOULD statements SHOULD be tested. (Implementations that demonstrate their support for SHOULD statements can increase their uptake by and appeal to deployers/end-users who might not themselves be implementers. Also, this helps increase interoperability, by showing that implementers who act on the SHOULD have done so according to the spec.) MAY statements MAY (though I daresay SHOULD) be tested. (Reasons are pretty much the same as the SHOULD above.) |
Because it isn't possible to reliably parse parameters you don't understand. The parsing step is an unnecessary complexity when the entire DID URL is passed to Resolve.
The problem is that parameters in the options override those in the DIDURL. Specifically, if you specify a different versionId or versionTime in the options, then that MUST prevail over any related property in the DID URL, including a versionId option overriding a versionTime in the DID URL. IMO, this is an unnecessary complexity that can only lead to inconsistent and incorrect usage, without improving the reliability of the call.
That is correct. What else do we do if the options disagree with parameters? |
Unfortunately, I think this is the heart of the confusion expressed by many. Dereferencing is something you do to bring the value referred to by a reference into the current context. See the introduction in the updated PR for better language explaining this. It isn't that the dereference() isn't an abstract function, it's that dereferencing is an algorithm performed by clients and not a function you call. |
Huh. I would think that the dereferencing algorithm might be implemented in some library/ies, and called thencefrom by some "client" applications in which this dereferencing algorithm isn't implemented. So I guess I disagree with the above quote. |
|
This PR has been overtaken by events (is being separated into smaller later PRs). Recommend close. |
|
Please keep this open until the other PR is resolved. This contains much of the discussion. |
This PR is primarily meant to address #226, clearing up confusion about dereferencing versus resolution.
My apologies for the huge PR. As I dove into the initial confusion, I became aware of several structural challenges that seemed to be at the root of the quagmire. I want to thank both @peacekeeper and @swcurran for their efforts to advance the work. I believe a lot of us heard of the edits and PRs in progress and they sounded good on the surface and we trusted our collaborators since, historically, they have done solid work.
Unfortunately, I think not enough of us have looked deeply enough at how dereferencing is described to fully realize the gaps in the current spec.
It wasn't until my own work on threat modeling required me to do a deeper dive, combined with the back-and-forth with Stephen on the path handling URL, created a perfect storm when I took on the job to try to address #226 from Jeffrey. These weren't linear events, but together they had a powerful synergy leading me to deeply rethink what is in the spec. Stephen and I initially tried to resolve our issues piecemeal within the context of his path handling PR, but that did more to expose the underlying structural problems than to resolve them.
When Stephen asserted that dereferencing and resolution were the same thing, I began to suspect that something fundamental is wrong in how we are describing the two functions. In particular, given the algorithms as described, it was incredibly difficult to separate the functions of resolution from dereferencing. Not only did Stephen bring that up, but when I made my own pass at Stephen's PR, I was also stumped by how to fit an extensible dereferencing mechanism within the algorithm as described.
As specific issues became apparent to me, I bubbled up a few issues for broader feedback. I think, on the whole those suggestions have largely been met with support, although some healthy debate has also engaged. One of those areas of feedback was asking for confirmation of the description of the workflow that I wrote for the threat model. Taken out of the resolution spec itself, it seemed there was general agreement of my understanding of the algorithm.
Addresses #308, #306, #305, #304, #303, #302, #246, #226
I think the biggest mistake was, unfortunately, catalyzed by my own addition in the last round of chartering, namely that there is something called "dereferencing" and therefore "dereferencers" that we should specify. This remains true, but I believe we made an understandable error when imagining the dereferencer as an equivalent component to the resolver. That is, when we imagined the dereferencer would also have an https binding and hence an API, we made a mistake. At least two. The first is that we imagined that we should describe a client in the language of the server. At the end of the day, the dereferencer is a client to a resolver who then performs additional steps to use the result of the resolver to return a resource. For example, while a web site MUST expose an HTTP(S) endpoint, the browser does not need to expose a dereferencing API. In fact, while some browsers to support a headless mode that enables external scripts and apps to dereference a URL to a local file, that interface is completely non-standard. And I definitely don't know of any browsers that offer such a feature over an HTTP(S) interface. (Opera may have back in the day, but that pattern didn't hold; most browsers DO NOT expose http ports as a fundamental security boundary.)
If you do some research on dereferencing, it is universally described as client-side functionality. Bundling a client-side capability into an HTTP endpoint simply defers the actual dereferencing to the entity calling *that* http endpoint, which is necessarily a dereferencing of the URL to reach the dereferencer.
FWIW, this shows up in the current spec, which has an anomalous reference to an otherwise underspecified "client" that does the final fragment dereferencing. So, even as currently designed what we call a dereferencer CAN'T actually complete the dereferencing. At the end of the day dereferencing remains a client side function, even if intermediate servers perform something that feels like dereferencing.
So, I revisited the algorithm from scratch and wrote what made sense to me with the insights of the previous work. I wanted an algorithm that cleanly separated resolution from dereferencing, one that would, in fact, make it easier to address extensible path handling approaches. That taught me a lot about features that probably ended up in the wrong spot. So I fixed those along the way.
I realize this is a lot to absorb at a stage in the game when Manu is pushing for closure, but the fact is the current spec has significant gaps and errors, and the only way I could even understand how to fix them was to rewrite so that algorithm was easier to read. When I set out to do so, I sincerely tried to just clean up without changing functionality; unfortunately I found places that seemed to be worth fixing.
Changes
1. Define dereferencing first, since that is the "outer" function in the algorithmic flow.
2. Pass a full DID URL to resolution, since the resolver may need DID URL parameters that the dereferencer is unaware of. This is especially true for method-specific extensions where resolvers know the special magic, e.g., BTCR, but the resolving client just wants to get back the right DID document, without needing to parse the incoming URL.
3. Remove guidance about why/when DID parameters should be used separately from the URL; instead discussed a specific reason when they should be used (without generalizing).
4. Removed the dereferencing API and its HTTPS binding to reframe dereferencing as an algorithm to be executed by any client that wants to use a DID URL.
5. Separated the algorithms for service and service type for simplicity.
6. Removed the `accept` options for dereferencing, since there is no longer an http endpoint to receive it. The dereferencer is free to apply whatever "accept" headers it might request when retrieving resources referenced in a DID Document or DID Metadata. Such headers should be based either on the business logic of the context the dereferencer is using the DID URL, or on the retrieval strategy defined by the properties in that document or metadata.
7. Updated the DID resolution image to clarify that resolution always returns a DID document and that dereferencing always presents a resource (which might mean display or serialized output), noting that the resource returned from dereferencing can be the DID document
8. Refactored the algorithm to support the explicit step of determinining the retrieval strategy based on the DID document. This is probably the biggest gap exposed by Stephen's contribution. The retrieval strategy depends not just on the DID method, but also on properties that can be found in either the DID document or DID document metadata.
9. Added the step where the dereferencer selects a resolver for a given DID method. This also led to the requirement that dereferencers MUST allow users to configure which resolver they want to use for particular DID methods. This is absolutely core to our security model and yet, that capability did not show up in the spec.
10. Removed the requirement for the dereferencer to parse DID URL parameters and put them into the options package. Given that we must pass the whole DID URL, this is just an opportunity for error to creep in. Rather, that step is for overriding the defaults and values from the DID URL, such as selecting a *different* version time. If there's no override, there's no need to set the options.
11. Name the "Resolution DID URL" which is sent to the resolver, which is the DID URL without the fragment part. In particular, this URL is NOT the input DID URL, so we should use a different name.
12. Added a coherent, prioritized heuristic for determining the retrieval strategy, based on the DID URL and the DID document. This explicitly suggests a hierarchy from
This is where I hope we can focus the path handling conversation to slide seamlessly into dereferencing without mucking with resolution.
13. Added a separate step for executing the retrieval strategy. Again, this gives us a clean way to support linkedResource, linkedResourceMetadata, pathService, and on-chain objects like ordinals.
14. Add the final step of using the resource, which is really the business purpose defined by the dereferencer. If the point is to display the resource visually, like in a browser, do that, and use the media type to handle any fragments. If the point is to use a verification method to verify a proof, do that. Fundamentally, the dereferencer is the client that manages the business logic based on the context of use. They put the retrieved resource to use, thus concluding dereferencing.
What's left to do?
Unfortunately, the terminology links are messed up. And the terminology section is empty. So, I think I need some guidance from the editors about how we want to define and refer to both existing terms and new ones like Resolution DID URL.
We also need to update some of the examples, including transforming my examples to the much better format the current spec uses with the EXAMPLE box and autonumber. That's editorial, but would make things much more readable.
We also need a test suite for the dereferencer/resolving client. This is not a new task, but it does beg the question about how we would test software that doesn't have an API. For which my response is that the test-suite isn't about testing software, it's about demonstrating the features of the spec. I can commit Legendary Requirements to producing a command-line dereferencer that a human tester could verify the features of the spec. The trick is demonstrating fragment handling. I do not have a way to automate such a test without co-opting a browser (with an extension) and using the browser for media presentation.
And the biggest to-do is to add the threat model to replace the architecture overview and security considerations sections. That's my next task. I'll have a draft in the DID Resolution Threat model repo this week and once we have some feedback on that, I'll propose how we might update the section of the spec.
Finally, I get that there is significant time pressure right now as our charter is due to expire. Even with an extension, we should be careful about bringing in such a huge change.
HOWEVER, I think we are in a worse position with the current spec than with this revision. This PR more clearly describes a dereferencing flow that I think is what most of us were imagining (minus the https endpoint), it adds some key features that, imo, were badly needed to support user choice, linkedResource, and linkedResourceMetadata, and removes several features that aren't needed. IMO, it is harder to understand and implement the current spec than it will be with this PR.
In any case, we still have the test suite to write, with all of its same complexities, especially wrt fragment handling, which cannot be outsourced to the other side of an https endpoint. So keeping the endpoint doesn't address how fragments are tested, while getting rid of the endpoint removes a ton of complexity.
In particular, I believe this conceptual framing of the dereferencer as a resolving client instead of a server matches more cleanly with the web architecture as it's existed since its inception.
That said, if the editors and the group would prefer to cherry pick the good stuff without throwing out the derereferencer server, I would just ask that at a minimum we include
Preview | Diff