objects/extension-definition schema member URL incorrect?

[dunkeki]

We are working an application related to (deeply validating objects using) custom extensions, beginning with those in development by the community specified in this repo. Extension definition objects in the objects/extension-definition folder all appear to have URLs in their schema members that point to (HTML user view) tree locations in this repo rather than directly to the (raw.githubusercontent.com/.../extension-definition-specifications/...) schema per the TC Extension Definition Policy. This makes using these awkward at best...

Perhaps I'm missing the intended use case for these objects? Has anyone developed a workflow for active live use of the extension definition objects in the objects/extension-definition folder? Thanks.

[rpiazza]

Thanks for pointing this out. This was an oversight when the latest STIX 1x extensions were merged into the main branch.
Are there other extensions you have noticed which also have this problem.

[rpiazza]

Hi. I was a little to quick to respond - there is a problem with the latest STIX 1x extensions, but you are referring to the general issue of the extension definition objects in the objects/extension-definition folder.

You are probably correct that these URLs are useless for obtaining the json schema for the extension, but could be used to find the extension manually.

Currently, there is only a theoretical use case of using these urls to obtain the json schemas automatically. It has not been implemented in the CTI STIX validator.

However, for that use case to be viable, it probably makes sense for it link to something which can be downloaded automatically. Right now those URLs point to the directory with all of the information (schema, documentation, examples) related to the extension, which even if we changed the link to point to the "raw data" might not be appropriate.

[dunkeki]

Thanks for your quick response. I was with you until your very last comment. And, WRT to the CTI STIX validator not currently using them, that is part of the impetus for what we are up to. Our server implementation currently uses your validator on object ingress to ensure data quality / integrity. As part of that (as well as diving directly into the validator source code), we "discovered" that while predefined extensions were validated, customs were not. So...we set off on an adventure and ended up here. The "STIX Editor" part of our client does validation as well on the front end for early indications to CTI creators that information is incomplete / non-comformant - another driver for desire to have thorough validation.

If you want to talk through any / all of this, including this issue (sorry if it was inappropriate), we can coordinate a live audio meet. I can be reached always at [email protected], thanks.

[rpiazza]

Hi. There is a lot to unpack in your response :-)

There was a recent release of the CTI STIX validator (release 3.1.3) that enhanced the validation of content that was based on extension definitions. In particular, the validator would use jsonschemas that were available locally (i.e., on the machine that was running the validator). The "theoretical" use case I mentioned previously was to use the URL in the schema property of the extension definition object to automatically find and download the schema from the internet. This HAS NOT been implemented in the current release of the validator.

It is important to note that the deprecated custom object/properties described in section 11 of the specification were never validated, because there was no jsonschema provided. With the introduction of extension definitions (section 7.3) the validator was improved to use jsonschemas specified on the command line to validate extension content. The latest version of the validator improved this feature based on section 3.7 of the Extension Policy document - under development by the STIX TC (see https://docs.google.com/document/d/1cGAQy93KuYZAgYUbzSomU_WIeDSUP4H7OVwbaBX5Szc). Notice that the MAY features may not be implemented.

Now to your original question (which certainly wasn't inappropriate). I agree that if the theoretical use case (i.e., finding jsonschemas on the internet automatically) is implemented the URL that is currently in the schema property of the extension definition objects in the objects/extension-definition folder is probably not useful, as it is a link to directory in the common object repository that contained the schema NOT directly to the schema file and its not even the raw file!

Because the validator doesn't use the URL at this time to obtain the jsonschema - it can best be thought of as a URL where one can manually obtain the schema. I hope this clears things up.

I think there is a discussion to be had about what URL should be specified in the extension definition object going forward.

[dunkeki]

I do appreciate (like!) the way the (this and the TC) discussion seems to be going in general. As I attempted to read and implement the latest specification, these are the items I inferred and the challenges that I encountered...

• our 'extended validator' first invokes the current cti-stix-validator and, if invalid, replies something that makes sense. Otherwise it moves on...
• it then looks for the extensions member in the candidate object
• if not present, it declares 'valid', otherwise it loops through the KVs
• if K doesn't begin with extension-definition--, it moves to the next KV pair
• at this point, it must be a custom extension. It looks for the extension-definition object IN THE CURRENT API ROOT AND COLLECTION ON THE LOCAL SERVER with no filter values (implying that the LAST version is the one we want). This is our first leap of faith...
• if not found, we return an INCONCLUSIVE result (there have been internal arguments about this, but that's irrelevant)
• If found, we attempt to parse the schema member as a URL. if invalid URL, we return INCONCLUSIVE
• if valid, we fetch the URL content - our second leap of faith - , apply JSON schema validation (including all sub schema loading per the JSON schema specification itself), and reply accordingly

Now that you are aware of what we implementing, let's focus on the challenges. First, where indeed should we be looking for the extension definition object? Thinking about api-root / collection pairs as authentication / authorization privileges, we look in the same api-root / collection pair as the object. Perhaps there should be a fixed api-root / collection pair for these things - I don't know... Perhaps we should be looking directly into the online cti-stix-common-objects/extension-definition-specifications as well, leveraging your stated purpose "...so that there's not duplicated objects..." [sic]

Then there's our basic inference that a valid schema URL points directly to the schema itself - this seemed to be clear in the TC Extension Policy, but we could be mistaken.

I do appreciate your and the group at large's opinion on these items. The specifications are so very clear on many / most items. I understand that these are emerging / growth items and I appreciate your patience with us as we push the bloody edge on our implementations...