|
| 1 | +--- |
| 2 | +slug: using-participle |
| 3 | +title: Why We Use Participle? |
| 4 | +description: Learn why we chose Participle and Roaring Bitmaps over GraphQL for our project. |
| 5 | +keywords: [Participle, Roaring Bitmaps, GraphQL, SBOM, dependency graphs] |
| 6 | +authors: [NeilNaveen, NaveenSrinivasan] |
| 7 | +tags: [participle, roaring bitmaps] |
| 8 | +--- |
| 9 | + |
| 10 | +We chose Participle and Roaring Bitmaps over GraphQL. Here's why. |
| 11 | + |
| 12 | +<!-- truncate --> |
| 13 | + |
| 14 | +--- |
| 15 | + |
| 16 | +**TLDR:** We had to make a crucial decision when developing **Minefield**, our tool for analyzing Software Bill of Materials (SBOMs) and their dependency graphs. Initially, we considered using **GraphQL** to query the intricate relationships between dependencies, dependents, and other relationships. However, as we delved deeper, it became evident that GraphQL wasn’t the ideal solution for our requirements, which included efficient, expressive, and high-performance querying on large, complex datasets. |
| 17 | +That’s when we opted to create a **custom Domain-Specific Language (DSL)** using **Participle**, a parsing library in Golang, and to enhance its performance, we backed it with **Roaring Bitmaps**. |
| 18 | + |
| 19 | +In this article, we will explore the reasons behind this decision and explain how it can help you tackle complex querying problems in your projects. |
| 20 | + |
| 21 | +--- |
| 22 | + |
| 23 | +## The Attraction of GraphQL: |
| 24 | + |
| 25 | +Our Initial Consideration is GraphQL, a formidable tool that offers flexibility, is driven by schemas, and is widely adopted in the development community. Initially, it appeared to be the perfect fit for querying Minefield’s dependency graphs. |
| 26 | +It allowed us to: |
| 27 | +- **Retrieve only the necessary data**, thereby reducing over-fetching commonly found in REST APIs. |
| 28 | +- **Utilize a robust schema**, which defines the structure of the data, making it simple to work with and validate queries. |
| 29 | +- **Leverage a vast ecosystem** equipped with tools for clients, servers, and documentation. |
| 30 | + |
| 31 | +Nevertheless, as we started experimenting, we encountered critical obstacles that led us to reconsider using GraphQL as a solution for querying SBOMs. |
| 32 | + |
| 33 | + |
| 34 | +## Why GraphQL Fell Short for Complex Dependency Graphs |
| 35 | + |
| 36 | +1. **Nested Relationships are Cumbersome** |
| 37 | + Dependency graphs are often deeply nested. You have to track which packages a library depends on and the entire chain of dependencies that follow. Writing deeply nested queries in GraphQL quickly became unwieldy and hard to manage. |
| 38 | + |
| 39 | +2. **Complex Set-Based Operations** |
| 40 | + GraphQL is great for fetching specific pieces of data, but it wasn't designed for complex set-based operations like intersections, unions, or complements. When dealing with millions of dependencies, performing these operations is crucial for answering questions like "Which dependencies are affected by this vulnerability but not by others?" |
| 41 | + |
| 42 | +3. **Performance Bottlenecks** |
| 43 | + As the dataset grew, performance became a concern. Querying large, complex graphs using GraphQL created a lot of overhead that slowed execution, especially when we needed to perform set-based operations on large lists of dependencies and vulnerabilities. |
| 44 | + |
| 45 | +--- |
| 46 | + |
| 47 | +We recognized the need for a tailored solution to query complex dependency graphs efficiently. |
| 48 | + |
| 49 | +To address this, we developed a **custom DSL** using **Participle**, a Golang parsing library that enabled us to define our own query language grammar with ease. |
| 50 | + |
| 51 | +Our DSL empowers us to construct potent, expressive queries that are simple to compose and execute efficiently. |
| 52 | + |
| 53 | +For instance, consider this straightforward example: ```dsl (dependencies library pkg:golang/net and not (dependents library pkg:golang/example)) ``` |
| 54 | +This query identifies all libraries dependent on `golang/net` but not relied upon by `golang/example`. |
| 55 | + |
| 56 | +Expressing such nuanced queries clearly in GraphQL was challenging, but our DSL effortlessly managed this task. |
| 57 | +### Why Participle? |
| 58 | + |
| 59 | +1. **Native to Go** |
| 60 | + We wanted something that felt natural for Go developers. We defined our query language with Participle using Go structs, making the DSL intuitive and type-safe. |
| 61 | + |
| 62 | + Example grammar definition: |
| 63 | + ```go |
| 64 | + type Expression struct { |
| 65 | + Left Term @@ |
| 66 | + Operator string @("and" | "or" | "xor")? |
| 67 | + Right *Expression @@? |
| 68 | + } |
| 69 | + ``` |
| 70 | + |
| 71 | +2. **Strong Typing** |
| 72 | + Leveraging Go's type system allowed us to create a robust DSL that's easy to extend and maintain. Adding new query operators or types is as simple as defining new Go structs and adding them to the parser. |
| 73 | + |
| 74 | +3. **Extensibility and Performance** |
| 75 | + Since we controlled the language's design, we could optimize it for exactly what we needed. With Participle, building a DSL that supports nested queries, conditional logic, and complex operations like set intersections was easy. |
| 76 | + |
| 77 | +--- |
| 78 | + |
| 79 | +## Roaring Bitmaps: Supercharging Query Performance |
| 80 | + |
| 81 | +Once Participle parses the queries, we apply them using Roaring Bitmaps, a data structure specifically designed for fast set operations. This was a key advantage over GraphQL, which doesn't natively support efficient set-based operations. |
| 82 | + |
| 83 | +### Why Roaring Bitmaps? |
| 84 | + |
| 85 | +Roaring Bitmaps are perfect for **large-scale datasets** like dependency graphs. They allow us to perform operations like **AND**, **OR**, and **XOR** on massive lists of dependencies in milliseconds, without sacrificing memory efficiency. |
| 86 | + |
| 87 | +Here's how we evaluate a parsed query using Roaring Bitmaps: |
| 88 | + |
| 89 | +```go |
| 90 | +func evaluateExpression(expr *Expression) *roaring.Bitmap { |
| 91 | + left := evaluateTerm(expr.Left) |
| 92 | + if expr.Operator == nil { |
| 93 | + return left |
| 94 | + } |
| 95 | + right := evaluateExpression(expr.Right) |
| 96 | + switch *expr.Operator { |
| 97 | + case "and": |
| 98 | + return roaring.And(left, right) |
| 99 | + case "or": |
| 100 | + return roaring.Or(left, right) |
| 101 | + case "xor": |
| 102 | + return roaring.Xor(left, right) |
| 103 | + } |
| 104 | + return nil |
| 105 | +} |
| 106 | +``` |
| 107 | + |
| 108 | +This allows us to calculate query results quickly and efficiently, even when dealing with millions of dependencies and vulnerabilities. |
| 109 | + |
| 110 | +--- |
| 111 | + |
| 112 | +## Real-World Use Case: Vulnerability Analysis at Scale |
| 113 | + |
| 114 | +Here’s an example of a real-world query you can run in Minefield using our DSL: |
| 115 | + |
| 116 | +```dsl |
| 117 | +(dependents vulns CVE-2023-12345 and not dependencies library pkg:golang/patchedlib) |
| 118 | +``` |
| 119 | + |
| 120 | +This query finds all dependents affected by a vulnerability (`CVE-2023-12345`) that haven’t yet patched the issue by using a specific library (`golang/patchedlib`). |
| 121 | + |
| 122 | +In a GraphQL world, expressing this kind of query would be cumbersome, and executing it would be slow due to the overhead of navigating through layers of nested data. With our custom DSL and Roaring Bitmaps, however, it runs quickly—even on large datasets. |
| 123 | + |
| 124 | +--- |
| 125 | + |
| 126 | +## Conclusion: Choosing the Right Tool for the Job |
| 127 | + |
| 128 | +GraphQL is an incredibly useful tool for many use cases, but it wasn’t the right fit for Minefield’s complex dependency graph queries. By using **Participle** to build a custom DSL and **Roaring Bitmaps** to optimize query performance, we created a solution that’s both **expressive** and **blazingly fast**. |
| 129 | + |
| 130 | +The takeaway? Always choose the right tool for the problem you’re solving. When you need something more tailored, building a custom solution can save you a lot of headaches down the road—especially when working with large, complex datasets. |
| 131 | + |
| 132 | +If you’re interested in digging deeper into how we built Minefield’s custom DSL or have your own experiences with dependency analysis, check out our [GitHub repository](https://github.com/bitbomdev/minefield) and let us know what you think! |
| 133 | + |
0 commit comments