Skip to content

Commit d462bc4

Browse files
r-nearr-near
authored
feat: Collections API (#4)
* feat: Collections API * Strip out generics for micropython * Switch to msgpack serialization * Linting * Move this to inside the SDK * Fix imports * Add docs * Rename this job * Switch to pickle (it's native) * Use generic Exception class
1 parent 204dbb8 commit d462bc4

28 files changed

+4417
-3
lines changed

‎.github/workflows/lint.yml

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Linting and Type Checking
1+
name: Linting, Type Checking, and Tests
22

33
on:
44
push:
@@ -26,3 +26,6 @@ jobs:
2626

2727
- name: Run mypy
2828
run: uv run mypy .
29+
30+
- name: Run pytest
31+
run: uv run pytest

‎docs/collections/README.md

+117
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# NEAR Python SDK Collections API
2+
3+
The Collections API provides efficient, persistent data structures for NEAR smart contracts written in Python. These collections help you manage contract state with a familiar Pythonic interface while optimizing for blockchain storage efficiency.
4+
5+
## Overview
6+
7+
When developing smart contracts, choosing the right data structures is critical for both functionality and gas efficiency. The Collections API offers specialized data structures that are:
8+
9+
- **Storage-efficient**: Values are serialized and persisted to blockchain storage
10+
- **Gas-optimized**: Data is lazy-loaded only when needed
11+
- **Pythonic**: Uses familiar syntax and methods similar to standard Python collections
12+
- **Type-safe**: Provides proper typing for better IDE support and code quality
13+
14+
## Available Collections
15+
16+
| Collection | Description | Similar to | When to Use |
17+
|------------|-------------|------------|-------------|
18+
| `Vector` | Ordered, indexable collection | Python's `list` | When you need ordered data with index-based access |
19+
| `LookupMap` | Non-iterable key-value store | Python's `dict` (without iteration) | When you only need fast key lookups and won't iterate |
20+
| `UnorderedMap` | Iterable key-value store | Python's `dict` | When you need both key lookups and iteration |
21+
| `LookupSet` | Non-iterable set of unique values | Python's `set` (without iteration) | When you only need to check for value existence |
22+
| `UnorderedSet` | Iterable set of unique values | Python's `set` | When you need to check for existence and iterate |
23+
| `TreeMap` | Ordered key-value store | Python's `SortedDict` | When you need keys in order or range queries |
24+
25+
## Quick Start
26+
27+
```python
28+
from near_sdk_py import view, call, init
29+
from near_sdk_py.collections import Vector, UnorderedMap, UnorderedSet
30+
31+
class TokenContract:
32+
def __init__(self):
33+
self.tokens = Vector("tokens")
34+
self.balances = UnorderedMap("balances")
35+
self.owners = UnorderedSet("owners")
36+
37+
@call
38+
def mint(self, token_id, owner_id):
39+
self.tokens.append(token_id)
40+
self.balances[owner_id] = self.balances.get(owner_id, 0) + 1
41+
self.owners.add(owner_id)
42+
43+
@view
44+
def get_tokens(self, from_index=0, limit=50):
45+
return self.tokens[from_index:from_index+limit]
46+
47+
@view
48+
def get_balance(self, account_id):
49+
return self.balances.get(account_id, 0)
50+
51+
@view
52+
def get_owners(self):
53+
return self.owners.values()
54+
```
55+
56+
## Import Options
57+
58+
You can import all collections at once or import specific collections:
59+
60+
```python
61+
# Import everything
62+
from near_sdk_py.collections import Vector, LookupMap, UnorderedMap, LookupSet, UnorderedSet, TreeMap
63+
64+
# Import specific collections
65+
from near_sdk_py.collections.vector import Vector
66+
from near_sdk_py.collections.unordered_map import UnorderedMap
67+
```
68+
69+
## Key Concepts
70+
71+
### Storage Prefixes
72+
73+
Each collection requires a unique prefix string to ensure data isn't accidentally overwritten. This prefix is used to create unique storage keys for each collection item.
74+
75+
```python
76+
tokens = Vector("tokens") # Uses prefix "tokens"
77+
owners = UnorderedSet("owners") # Uses prefix "owners"
78+
```
79+
80+
For more sophisticated prefix management, see the [Prefixes Guide](prefixes.md).
81+
82+
### Storage Efficiency
83+
84+
Collections store data on the blockchain, which has a cost in NEAR tokens. The API is designed to be storage-efficient:
85+
86+
- Data is only loaded when accessed (lazy loading)
87+
- Non-iterable collections (`LookupMap`, `LookupSet`) use less storage overhead
88+
- Iterable collections add metadata to enable iteration
89+
90+
See the [Storage Management Guide](storage_management.md) for best practices.
91+
92+
### Choosing the Right Collection
93+
94+
Choosing the right collection type is important for both functionality and gas efficiency:
95+
96+
- Need ordered, indexable data? Use `Vector`
97+
- Need key-value mapping without iteration? Use `LookupMap`
98+
- Need key-value mapping with iteration? Use `UnorderedMap`
99+
- Need to check for unique values without iteration? Use `LookupSet`
100+
- Need to check for unique values with iteration? Use `UnorderedSet`
101+
- Need ordered key-value mapping or range queries? Use `TreeMap`
102+
103+
For more details, see the [Choosing Collections Guide](choosing.md).
104+
105+
## Documentation
106+
107+
- [Vector Documentation](vector.md)
108+
- [Maps Documentation](maps.md) (LookupMap and UnorderedMap)
109+
- [Sets Documentation](sets.md) (LookupSet and UnorderedSet)
110+
- [TreeMap Documentation](tree_map.md)
111+
- [Storage Management Guide](storage_management.md)
112+
- [Prefixes Guide](prefixes.md)
113+
- [Examples](examples/)
114+
115+
## Reference
116+
117+
The Collections API is inspired by the [NEAR Rust SDK Collections](https://docs.near.org/build/smart-contracts/anatomy/collections) but adapted for Python's idioms and features.

‎docs/collections/choosing.md

+168
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
# Choosing the Right Collection
2+
3+
Selecting the appropriate collection type for your NEAR smart contract is crucial for both functionality and gas efficiency. This guide will help you understand the tradeoffs between different collection types.
4+
5+
## Collection Types Overview
6+
7+
| Collection | Iterable? | Ordered? | Unique Keys? | Memory Overhead | Use Case |
8+
|------------|-----------|----------|--------------|----------------|----------|
9+
| `Vector` | ✅ | ✅ (by insertion) | ❌ | Low | Ordered list of items |
10+
| `LookupMap` | ❌ | ❌ | ✅ | Lowest | Fast key-value lookups |
11+
| `UnorderedMap` | ✅ | ❌ | ✅ | Medium | Key-value with iteration |
12+
| `LookupSet` | ❌ | ❌ | ✅ | Lowest | Fast unique value lookups |
13+
| `UnorderedSet` | ✅ | ❌ | ✅ | Medium | Unique values with iteration |
14+
| `TreeMap` | ✅ | ✅ (by key) | ✅ | Highest | Ordered key-value pairs |
15+
16+
## Decision Flowchart
17+
18+
```
19+
Do you need to store key-value pairs?
20+
├── Yes → Do you need to iterate over all entries?
21+
│ ├── Yes → Do you need keys to be ordered?
22+
│ │ ├── Yes → Use TreeMap
23+
│ │ └── No → Use UnorderedMap
24+
│ └── No → Use LookupMap
25+
└── No → Do you need to store unique values?
26+
├── Yes → Do you need to iterate over all values?
27+
│ ├── Yes → Use UnorderedSet
28+
│ └── No → Use LookupSet
29+
└── No → Do you need indexed access or ordered items?
30+
├── Yes → Use Vector
31+
└── No → Use Vector (still the best choice)
32+
```
33+
34+
## When to Use Each Collection
35+
36+
### Use `Vector` when you need:
37+
38+
- An ordered list of items
39+
- Index-based access (e.g., `my_vector[5]`)
40+
- To frequently add items to the end of a list
41+
- To maintain insertion order
42+
43+
### Use `LookupMap` when you need:
44+
45+
- Fast key-value lookups
46+
- Gas efficiency (lowest overhead)
47+
- Only key-based access, never iteration
48+
- To check if a key exists
49+
50+
### Use `UnorderedMap` when you need:
51+
52+
- Both key-value lookups and iteration
53+
- To get all keys, values, or key-value pairs
54+
- A classic dictionary/map with full functionality
55+
56+
### Use `LookupSet` when you need:
57+
58+
- To check if a value exists (membership)
59+
- Gas efficiency (lowest overhead)
60+
- Only membership testing, never iteration
61+
62+
### Use `UnorderedSet` when you need:
63+
64+
- Both membership testing and iteration
65+
- To get all unique values in the set
66+
- A classic set with full functionality
67+
68+
### Use `TreeMap` when you need:
69+
70+
- Keys to be kept in sorted order
71+
- Range queries (get all keys between X and Y)
72+
- To find the nearest key (floor/ceiling operations)
73+
- Ordered iteration through keys
74+
75+
## Gas and Storage Considerations
76+
77+
Collections have different storage and gas cost profiles:
78+
79+
- **Non-iterable collections** (`LookupMap`, `LookupSet`) have the lowest storage overhead but cannot be iterated over
80+
- **Iterable collections** (`UnorderedMap`, `UnorderedSet`) require additional storage to track keys/values for iteration
81+
- **Ordered collections** (`Vector`, `TreeMap`) have additional overhead to maintain order
82+
83+
## Collection Size Recommendations
84+
85+
| Collection Size | Recommended Collection Type |
86+
|-----------------|---------------------------|
87+
| Small (< 100 items) | Any collection type is fine |
88+
| Medium (100-1,000 items) | Consider gas efficiency more carefully |
89+
| Large (> 1,000 items) | Use non-iterable collections when possible and implement pagination |
90+
91+
## Examples
92+
93+
### Tokens in an NFT Contract
94+
95+
```python
96+
# Store a list of all token IDs
97+
token_ids = Vector("token_ids")
98+
99+
# Store token metadata by token ID
100+
token_metadata = UnorderedMap("token_metadata")
101+
102+
# Track which tokens are owned by which accounts
103+
tokens_by_owner = UnorderedMap("tokens_by_owner")
104+
```
105+
106+
### Voting System
107+
108+
```python
109+
# Track valid voters
110+
eligible_voters = LookupSet("eligible_voters")
111+
112+
# Track who has already voted
113+
voted = LookupSet("voted")
114+
115+
# Store votes by proposal ID
116+
votes_by_proposal = UnorderedMap("votes_by_proposal")
117+
118+
# Store proposals in order by closing time
119+
proposals_by_end_time = TreeMap("proposals_by_end_time")
120+
```
121+
122+
### User Profiles
123+
124+
```python
125+
# Store user profiles by account ID
126+
profiles = LookupMap("profiles")
127+
128+
# Track verified users
129+
verified_users = LookupSet("verified_users")
130+
131+
# Featured profiles in priority order
132+
featured_profiles = Vector("featured_profiles")
133+
```
134+
135+
## Hybrid Approaches
136+
137+
Sometimes you might need multiple collections to achieve your goals efficiently:
138+
139+
```python
140+
# For a system needing both fast lookups and ordered access:
141+
142+
# Fast lookup by ID
143+
items_by_id = LookupMap("items_by_id")
144+
145+
# Ordered listing of IDs
146+
ordered_item_ids = Vector("ordered_item_ids")
147+
148+
# Usage:
149+
def get_item(item_id):
150+
return items_by_id.get(item_id)
151+
152+
def get_items_paginated(from_index, limit):
153+
# Get IDs in order
154+
ids = ordered_item_ids[from_index:from_index + limit]
155+
# Retrieve each item by ID
156+
return [items_by_id[id] for id in ids]
157+
```
158+
159+
## Summary
160+
161+
1. For simple ordered lists, use `Vector`
162+
2. For key-value lookups without iteration, use `LookupMap`
163+
3. For key-value with iteration, use `UnorderedMap`
164+
4. For unique value checking without iteration, use `LookupSet`
165+
5. For unique values with iteration, use `UnorderedSet`
166+
6. For ordered key-value data, use `TreeMap`
167+
168+
Choose the most restrictive collection that meets your needs to maximize gas efficiency.

0 commit comments

Comments
 (0)