Define TR caching comms in ATD #353
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To make things more concrete about the interface we want with the backend. test plan: make
aryx
requested review from
a team,
emjin,
aaronmichaelacosta and
bkettle
and removed request for
a team and
emjin
|
Backwards compatibility summary: |
bkettle
reviewed
Feb 20, 2025
semgrep_output_v1.atd
Outdated
| type tr_cache_key = { | ||
| rule_id: rule_id; | ||
| (* ex: http://some-website/hello-world.0.1.2.tgz like in found_dependency *) | ||
| resolved_url: string; |
There was a problem hiding this comment.
I don't know if this will be easy to find in many cases, though I agree that if it's possible it makes the best key. I guess it is probably fine to start with this, and add another key later if needed.
There was a problem hiding this comment.
yes we can always refine. This is just defining the interface. Once we start the implementation we will discover
we need to refine it.
There was a problem hiding this comment.
Thoughts on or experience with PackageURL?
Also curious how we determine the url to fetch the code for a package, in the event that our dependency scanning logic doesn't yield a resolved_url?
semgrep_output_v1.atd
Outdated
| * and [transitive_unreachable] records? | ||
| * TODO? make it a list? match_results: ... list; ? | ||
| *) | ||
| match_result: sca_match_kind; |
There was a problem hiding this comment.
This is a bit odd, since when scanning a package with a rule it will result in direct code matches, not sca matches. Maybe we should just return the match here? Then, the CLI could treat those matches the same as matches that it receives from a call to Semgrep locally.
There was a problem hiding this comment.
We could. This is a bigger data structure to store then though, and for TR what we really need is actually just the sca_transitive_match_kind; that's the thing we try to optimize to avoid downloading the dependency and run semgrep on it.
There was a problem hiding this comment.
I think we should at least store the match locations. I don't think it makes sense to store sca_match_kind because if there are matches in multiple packages, we will somehow need to combine those into a single finding in the cli
gautambhat
approved these changes
Mar 6, 2025
mmcqd
pushed a commit
that referenced
this pull request
Mar 11, 2025
To make things more concrete about the interface we want with the backend. test plan: make - [x] I ran `make setup && make` to update the generated code after editing a `.atd` file (TODO: have a CI check) - [x] I made sure we're still backward compatible with old versions of the CLI. For example, the Semgrep backend need to still be able to *consume* data generated by Semgrep 1.50.0. See https://atd.readthedocs.io/en/latest/atdgen-tutorial.html#smooth-protocol-upgrades Note that the types related to the semgrep-core JSON output or the semgrep-core RPC do not need to be backward compatible!
mmcqd
added a commit
that referenced
this pull request
Mar 11, 2025
To make things more concrete about the interface we want with the backend. test plan: make - [x] I ran `make setup && make` to update the generated code after editing a `.atd` file (TODO: have a CI check) - [x] I made sure we're still backward compatible with old versions of the CLI. For example, the Semgrep backend need to still be able to *consume* data generated by Semgrep 1.50.0. See https://atd.readthedocs.io/en/latest/atdgen-tutorial.html#smooth-protocol-upgrades Note that the types related to the semgrep-core JSON output or the semgrep-core RPC do not need to be backward compatible!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To make things more concrete about the interface we want
with the backend.
test plan:
make
make setup && maketo update the generated code after editing a.atdfile (TODO: have a CI check)For example, the Semgrep backend need to still be able to consume data
generated by Semgrep 1.50.0.
See https://atd.readthedocs.io/en/latest/atdgen-tutorial.html#smooth-protocol-upgrades
Note that the types related to the semgrep-core JSON output or the
semgrep-core RPC do not need to be backward compatible!