Skip to content

Commit ff7b81d

Browse files
jhamonjhamon
authored
Add bulk import (#386)
## Problem Implement the following new methods: - `start_import` - `describe_import` - `list_imports` - `cancel_import` ## Solution #### Code generation changes Since these features are in prerelease, they only exist in the spec for the upcoming 2024-10 API version. This required me to make modifications to the codegen script that is now run as: ``` ./codegen/build-oas.sh 2024-07 false && ./codegen/build-oas.sh 2024-10 true ``` The second boolean argument is used to tell the codegen script whether the generated code should be stored in a new `pinecone/core_ea` subpackage. In the future we should probably do more to hide this complexity from the developer, but for now it is good enough. #### Code organization For the bespoke bits of the implementation that wrap the generated code, I have put them into a new class, `ImportFeatureMixin`, that the `Index` class inherits from. These functions could have all been implemented directly in the `Index` class, but I thought it a bit tidier to segregate these into a separate spot than just dump everything into one giant file. #### Overridden repr representation on generated objects The default print output in the generated classes comes from pprint and it looks quite poor for large objects. So I installed overrides that dump the objects into a formatted json style instead. I had previously done something similar for describe_index, etc, methods, so for this PR it was just a matter of cleaning up that logic a bit and moving it somewhere it could be reused. So far, I haven't tweaked the generated classes to do this approach across the board because it doesn't work well for long arrays of vector values. ## Type of Change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] This change requires a documentation update - [ ] Infrastructure change (CI configs, etc) - [ ] Non-code change (docs, etc) - [ ] None of the above: (explain here) ## Test Plan Manual testing with a dev release is in this [demo notebook](https://colab.research.google.com/drive/1W3OhMDG1yW2rdwx-ZulYH847m9R_IUuK#scrollTo=gGvVbfkYNz61 )
1 parent 4e9a40c commit ff7b81d

File tree

131 files changed

+19549
-202
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

131 files changed

+19549
-202
lines changed

Diff for: ‎.github/workflows/alpha-release.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -49,3 +49,4 @@ jobs:
4949
secrets:
5050
PYPI_USERNAME: __token__
5151
PYPI_PASSWORD: ${{ secrets.PROD_PYPI_PUBLISH_TOKEN }}
52+

Diff for: ‎CONTRIBUTING.md

+23
Original file line numberDiff line numberDiff line change
@@ -142,3 +142,26 @@ Hello, from your virtualenv!
142142
```
143143

144144
If you experience any issues please [file a new issue](https://github.com/pinecone-io/pinecone-python-client/issues/new).
145+
146+
147+
## Consuming API version upgrades
148+
149+
These instructions can only be followed by Pinecone employees with access to our private APIs repository.
150+
151+
Prerequisites:
152+
- You must be an employee with access to private Pinecone repositories
153+
- You must have [Docker Desktop](https://www.docker.com/products/docker-desktop/) installed and running. Our code generation script uses a dockerized version of the OpenAPI CLI.
154+
- You must have initialized the git submodules under codegen
155+
156+
```sh
157+
git submodule
158+
```
159+
160+
161+
To regenerate the generated portions of the client with the latest version of the API specifications, you need to have Docker Desktop running on your local machine.
162+
163+
164+
165+
```sh
166+
./codegen/
167+
```

Diff for: ‎Makefile

+1-1
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ develop:
1111

1212
test-unit:
1313
@echo "Running tests..."
14-
poetry run pytest --cov=pinecone --timeout=120 tests/unit
14+
poetry run pytest --cov=pinecone --timeout=120 tests/unit -s -vv
1515

1616
test-integration:
1717
@echo "Running integration tests..."

Diff for: ‎codegen/apis

Submodule apis updated from 062b114 to 3b7369b

Diff for: ‎codegen/build-oas.sh

+27-10
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,21 @@
33
set -eux -o pipefail
44

55
version=$1 # e.g. 2024-07
6-
modules=("control" "data")
6+
is_early_access=$2 # e.g. true
7+
8+
# if is_early_access is true, add the "ea" module
9+
if [ "$is_early_access" = "true" ]; then
10+
destination="pinecone/core_ea/openapi"
11+
modules=("db_control" "db_data")
12+
py_module_name="core_ea"
13+
template_dir="codegen/python-oas-templates/templates5.2.0"
14+
else
15+
destination="pinecone/core/openapi"
16+
modules=("control" "data")
17+
py_module_name="core"
18+
template_dir="codegen/python-oas-templates/templates5.2.0"
19+
fi
720

8-
destination="pinecone/core/openapi"
921
build_dir="build"
1022

1123
update_apis_repo() {
@@ -58,11 +70,9 @@ generate_client() {
5870
local module_name=$1
5971

6072
oas_file="codegen/apis/_build/${version}/${module_name}_${version}.oas.yaml"
61-
openapi_generator_config="codegen/openapi-config.${module_name}.json"
62-
template_dir="codegen/python-oas-templates/templates5.2.0"
73+
package_name="pinecone.${py_module_name}.openapi.${module_name}"
6374

6475
verify_file_exists $oas_file
65-
verify_file_exists $openapi_generator_config
6676
verify_directory_exists $template_dir
6777

6878
# Cleanup previous build files
@@ -73,13 +83,20 @@ generate_client() {
7383
docker run --rm -v $(pwd):/workspace openapitools/openapi-generator-cli:v5.2.0 generate \
7484
--input-spec "/workspace/$oas_file" \
7585
--generator-name python \
76-
--config "/workspace/$openapi_generator_config" \
86+
--additional-properties=packageName=$package_name,pythonAttrNoneIfUnset=true \
7787
--output "/workspace/${build_dir}" \
7888
--template-dir "/workspace/$template_dir"
7989

90+
# Hack to prevent coercion of strings into datetimes within "object" types while still
91+
# allowing datetime parsing for fields that are explicitly typed as datetime
92+
find "${build_dir}" -name "*.py" | while IFS= read -r file; do
93+
sed -i '' "s/bool, date, datetime, dict, float, int, list, str, none_type/bool, dict, float, int, list, str, none_type/g" "$file"
94+
done
95+
8096
# Copy the generated module to the correct location
8197
rm -rf "${destination}/${module_name}"
82-
cp -r "build/pinecone/core/openapi/${module_name}" "${destination}/${module_name}"
98+
mkdir -p "${destination}"
99+
cp -r "build/pinecone/$py_module_name/openapi/${module_name}" "${destination}/${module_name}"
83100
}
84101

85102
extract_shared_classes() {
@@ -118,13 +135,13 @@ extract_shared_classes() {
118135

119136
# Adjust import paths in every file
120137
find "${destination}" -name "*.py" | while IFS= read -r file; do
121-
sed -i '' 's/from \.\.model_utils/from pinecone\.core\.openapi\.shared\.model_utils/g' "$file"
138+
sed -i '' "s/from \.\.model_utils/from pinecone\.$py_module_name\.openapi\.shared\.model_utils/g" "$file"
122139

123140
for module in "${modules[@]}"; do
124-
sed -i '' "s/from pinecone\.core\.openapi\.$module import rest/from pinecone\.core\.openapi\.shared import rest/g" "$file"
141+
sed -i '' "s/from pinecone\.$py_module_name\.openapi\.$module import rest/from pinecone\.$py_module_name\.openapi\.shared import rest/g" "$file"
125142

126143
for sharedFile in "${sharedFiles[@]}"; do
127-
sed -i '' "s/from pinecone\.core\.openapi\.$module\.$sharedFile/from pinecone\.core\.openapi\.shared\.$sharedFile/g" "$file"
144+
sed -i '' "s/from pinecone\.$py_module_name\.openapi\.$module\.$sharedFile/from pinecone\.$py_module_name\.openapi\.shared\.$sharedFile/g" "$file"
128145
done
129146
done
130147
done

Diff for: ‎codegen/openapi-config.control.json

-4
This file was deleted.

Diff for: ‎codegen/openapi-config.data.json

-4
This file was deleted.

Diff for: ‎codegen/python-oas-templates

Diff for: ‎pinecone/config/openapi.py

+2
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@ def build(cls, api_key: str, host: Optional[str] = None, **kwargs):
2424
openapi_config.host = host
2525
openapi_config.ssl_ca_cert = certifi.where()
2626
openapi_config.socket_options = cls._get_socket_options()
27+
openapi_config.discard_unknown_keys = True
28+
2729
return openapi_config
2830

2931
@classmethod

Diff for: ‎pinecone/control/repr_overrides.py

+2-3
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,7 @@
1+
from pinecone.utils import install_json_repr_override
12
from pinecone.models.index_model import IndexModel
23
from pinecone.core.openapi.control.models import CollectionModel
34

4-
import json
5-
65

76
def install_repr_overrides():
87
"""
@@ -14,4 +13,4 @@ def install_repr_overrides():
1413
query results.
1514
"""
1615
for model in [IndexModel, CollectionModel]:
17-
model.__repr__ = lambda self: json.dumps(self.to_dict(), indent=4, sort_keys=False)
16+
install_json_repr_override(model)

Diff for: ‎pinecone/core/openapi/control/model/collection_list.py

-2
Original file line numberDiff line numberDiff line change
@@ -72,8 +72,6 @@ def additional_properties_type():
7272
lazy_import()
7373
return (
7474
bool,
75-
date,
76-
datetime,
7775
dict,
7876
float,
7977
int,

Diff for: ‎pinecone/core/openapi/control/model/collection_model.py

-2
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,6 @@ def additional_properties_type():
7676
"""
7777
return (
7878
bool,
79-
date,
80-
datetime,
8179
dict,
8280
float,
8381
int,

Diff for: ‎pinecone/core/openapi/control/model/configure_index_request.py

-2
Original file line numberDiff line numberDiff line change
@@ -74,8 +74,6 @@ def additional_properties_type():
7474
lazy_import()
7575
return (
7676
bool,
77-
date,
78-
datetime,
7977
dict,
8078
float,
8179
int,

Diff for: ‎pinecone/core/openapi/control/model/configure_index_request_spec.py

-2
Original file line numberDiff line numberDiff line change
@@ -72,8 +72,6 @@ def additional_properties_type():
7272
lazy_import()
7373
return (
7474
bool,
75-
date,
76-
datetime,
7775
dict,
7876
float,
7977
int,

Diff for: ‎pinecone/core/openapi/control/model/configure_index_request_spec_pod.py

-2
Original file line numberDiff line numberDiff line change
@@ -69,8 +69,6 @@ def additional_properties_type():
6969
"""
7070
return (
7171
bool,
72-
date,
73-
datetime,
7472
dict,
7573
float,
7674
int,

Diff for: ‎pinecone/core/openapi/control/model/create_collection_request.py

-2
Original file line numberDiff line numberDiff line change
@@ -70,8 +70,6 @@ def additional_properties_type():
7070
"""
7171
return (
7272
bool,
73-
date,
74-
datetime,
7573
dict,
7674
float,
7775
int,

Diff for: ‎pinecone/core/openapi/control/model/create_index_request.py

-2
Original file line numberDiff line numberDiff line change
@@ -89,8 +89,6 @@ def additional_properties_type():
8989
lazy_import()
9090
return (
9191
bool,
92-
date,
93-
datetime,
9492
dict,
9593
float,
9694
int,

Diff for: ‎pinecone/core/openapi/control/model/deletion_protection.py

-2
Original file line numberDiff line numberDiff line change
@@ -66,8 +66,6 @@ def additional_properties_type():
6666
"""
6767
return (
6868
bool,
69-
date,
70-
datetime,
7169
dict,
7270
float,
7371
int,

Diff for: ‎pinecone/core/openapi/control/model/embed_request.py

-2
Original file line numberDiff line numberDiff line change
@@ -74,8 +74,6 @@ def additional_properties_type():
7474
lazy_import()
7575
return (
7676
bool,
77-
date,
78-
datetime,
7977
dict,
8078
float,
8179
int,

Diff for: ‎pinecone/core/openapi/control/model/embed_request_inputs.py

-2
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,6 @@ def additional_properties_type():
6565
"""
6666
return (
6767
bool,
68-
date,
69-
datetime,
7068
dict,
7169
float,
7270
int,

Diff for: ‎pinecone/core/openapi/control/model/embed_request_parameters.py

-2
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,6 @@ def additional_properties_type():
6565
"""
6666
return (
6767
bool,
68-
date,
69-
datetime,
7068
dict,
7169
float,
7270
int,

Diff for: ‎pinecone/core/openapi/control/model/embedding.py

-2
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,6 @@ def additional_properties_type():
6565
"""
6666
return (
6767
bool,
68-
date,
69-
datetime,
7068
dict,
7169
float,
7270
int,

Diff for: ‎pinecone/core/openapi/control/model/embeddings_list.py

-2
Original file line numberDiff line numberDiff line change
@@ -74,8 +74,6 @@ def additional_properties_type():
7474
lazy_import()
7575
return (
7676
bool,
77-
date,
78-
datetime,
7977
dict,
8078
float,
8179
int,

Diff for: ‎pinecone/core/openapi/control/model/embeddings_list_usage.py

-2
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,6 @@ def additional_properties_type():
6565
"""
6666
return (
6767
bool,
68-
date,
69-
datetime,
7068
dict,
7169
float,
7270
int,

Diff for: ‎pinecone/core/openapi/control/model/error_response.py

-2
Original file line numberDiff line numberDiff line change
@@ -72,8 +72,6 @@ def additional_properties_type():
7272
lazy_import()
7373
return (
7474
bool,
75-
date,
76-
datetime,
7775
dict,
7876
float,
7977
int,

Diff for: ‎pinecone/core/openapi/control/model/error_response_error.py

+3-5
Original file line numberDiff line numberDiff line change
@@ -87,8 +87,6 @@ def additional_properties_type():
8787
"""
8888
return (
8989
bool,
90-
date,
91-
datetime,
9290
dict,
9391
float,
9492
int,
@@ -112,7 +110,7 @@ def openapi_types():
112110
return {
113111
"code": (str,), # noqa: E501
114112
"message": (str,), # noqa: E501
115-
"details": ({str: (bool, date, datetime, dict, float, int, list, str, none_type)},), # noqa: E501
113+
"details": ({str: (bool, dict, float, int, list, str, none_type)},), # noqa: E501
116114
}
117115

118116
@cached_property
@@ -169,7 +167,7 @@ def _from_openapi_data(cls, code, message, *args, **kwargs): # noqa: E501
169167
Animal class but this time we won't travel
170168
through its discriminator because we passed in
171169
_visited_composed_classes = (Animal,)
172-
details ({str: (bool, date, datetime, dict, float, int, list, str, none_type)}): Additional information about the error. This field is not guaranteed to be present.. [optional] # noqa: E501
170+
details ({str: (bool, dict, float, int, list, str, none_type)}): Additional information about the error. This field is not guaranteed to be present.. [optional] # noqa: E501
173171
"""
174172

175173
_check_type = kwargs.pop("_check_type", True)
@@ -262,7 +260,7 @@ def __init__(self, code, message, *args, **kwargs): # noqa: E501
262260
Animal class but this time we won't travel
263261
through its discriminator because we passed in
264262
_visited_composed_classes = (Animal,)
265-
details ({str: (bool, date, datetime, dict, float, int, list, str, none_type)}): Additional information about the error. This field is not guaranteed to be present.. [optional] # noqa: E501
263+
details ({str: (bool, dict, float, int, list, str, none_type)}): Additional information about the error. This field is not guaranteed to be present.. [optional] # noqa: E501
266264
"""
267265

268266
_check_type = kwargs.pop("_check_type", True)

Diff for: ‎pinecone/core/openapi/control/model/index_list.py

-2
Original file line numberDiff line numberDiff line change
@@ -72,8 +72,6 @@ def additional_properties_type():
7272
lazy_import()
7373
return (
7474
bool,
75-
date,
76-
datetime,
7775
dict,
7876
float,
7977
int,

Diff for: ‎pinecone/core/openapi/control/model/index_model.py

-2
Original file line numberDiff line numberDiff line change
@@ -91,8 +91,6 @@ def additional_properties_type():
9191
lazy_import()
9292
return (
9393
bool,
94-
date,
95-
datetime,
9694
dict,
9795
float,
9896
int,

Diff for: ‎pinecone/core/openapi/control/model/index_model_spec.py

-2
Original file line numberDiff line numberDiff line change
@@ -74,8 +74,6 @@ def additional_properties_type():
7474
lazy_import()
7575
return (
7676
bool,
77-
date,
78-
datetime,
7977
dict,
8078
float,
8179
int,

Diff for: ‎pinecone/core/openapi/control/model/index_model_status.py

-2
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,6 @@ def additional_properties_type():
7676
"""
7777
return (
7878
bool,
79-
date,
80-
datetime,
8179
dict,
8280
float,
8381
int,

Diff for: ‎pinecone/core/openapi/control/model/pod_spec.py

-2
Original file line numberDiff line numberDiff line change
@@ -82,8 +82,6 @@ def additional_properties_type():
8282
lazy_import()
8383
return (
8484
bool,
85-
date,
86-
datetime,
8785
dict,
8886
float,
8987
int,

Diff for: ‎pinecone/core/openapi/control/model/pod_spec_metadata_config.py

-2
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,6 @@ def additional_properties_type():
6565
"""
6666
return (
6767
bool,
68-
date,
69-
datetime,
7068
dict,
7169
float,
7270
int,

Diff for: ‎pinecone/core/openapi/control/model/serverless_spec.py

-2
Original file line numberDiff line numberDiff line change
@@ -71,8 +71,6 @@ def additional_properties_type():
7171
"""
7272
return (
7373
bool,
74-
date,
75-
datetime,
7674
dict,
7775
float,
7876
int,

0 commit comments

Comments
 (0)