[Requested feature] Server endpoint to interact with references saved in cache

[M-casado]

Summary

A feature (probably endpoint) to interact with references saved in cache of a Biovalidator server:

• To be able to refresh the stored references.
• To know when was the last time these were refreshed.

Motivation

When using URL-resolvable references for schemas, once any reference is used, it is stored in cache, avoiding its repeated download. This feature is very useful dealing with the delay of downloading referenced schemas. Nevertheless, at some point one may change the URL-resolved reference, and would want those modifications to propagate to the validation server without changing the referenced schema name ("$ref"). If the references are still stored in cache, the server will not grab the modified schemas, and would instead use the obsolete ones.
Therefore, we request an endpoint that allows for the server deployers to:

• Be able to refresh the stored references. As soon as this endpoint was contacted, save schemas would be removed from cache. This would cause for the references to be downloaded again when used, and thus grab the new modifications. In our use-case this endpoint could be reached by anyone, not only the server owners, but if credentials were needed it would also be fine.
• Be able to check when was the last time the references were refreshed. Again, a public endpoint would be fine. With this feature we would be able to interact with the refresh time to compare new changes, or for debugging purposes.

Examples

Example 1

My schema is used as the "latest version" reference. I notice a mistake and amend it. But I already used the validation tool, which stored the previous version of the schema. I would then want to manually refresh the cache of the server for it to grab the newer version.

Example 2

I would want to incorporate in my GitHub actions one that automatically pings the refresh endpoint whenever a new change is merged with the main branch. That way, changes in the GitHub repository I use to store my schemas would always propagate to the validation API.

Example 3

I would like to create a script that automatically checks if the schemas have not been refreshed for longer than a week (using the second requested feature) and, if so, automatically refresh them (using the first requested feature).

[theisuru]

This feature was back of my head for some time.
Currently, we have a custom cache and we are also using AJV cacheing. With custom cache these features are very straightforward. I just checked AJV API and it has both delete cache and get from cache features. But unfortunately we can get only from cache if we know the object key. I will see how we can move this forward. But as an first step, we will implement the cache clear feature.

[M-casado]

Thanks for the effort!

[M-casado]

@theisuru, even though this issue was closed, and I guess addressed in the PR, is there documentation anywhere on how to use this new feature to refresh the cache of the server?

[M-casado]

@theisuru if there is any documentation eventually for the cache interactions, please let me know!
Ideally we would set up a Webhook from my metadata repo so that the Cache is deleted every time my schema repo is updated!

[theisuru]

@M-casado sorry about the late response. We only have minimal entry in readme for cache interaction. I will add more details, but probably it will be in June.

[M-casado]

Hi @theisuru, thanks for reopening the issue: although the feature was technically implemente, without documentation it'll be difficult to make use of.

Besides, I would also like to address the other two features:

• Checking when the Cache was last deleted. This may be available already with the get /cache but I am missing that documentation, and haven't tried it on my own.
• Being able to allow deletion of the cache only if some kind of credentials are used. In our use-case, this could be done by only allowing the deletion of the cache by interacting with the server in a more local way (i.e. only we, who deployed the server, would be able to), or providing a password with the request of deletion.

Without the latter, I foresee an easy route of attack that would incredibly hinder the validation efficiency: the deletion of our cache would require the download of the schemas at every request, which would make requests go from mili-seconds to ~10s.