Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid spurious backtrace / log message on critical error #291

Merged
merged 1 commit into from
Jan 13, 2025

Conversation

daverigby
Copy link
Collaborator

@daverigby daverigby commented Jan 10, 2025

Problem

On a critical error (e.g. existing index and --overwrite not specified), VSB prints a large amount of spurious / unnecessary output - e.g:

vsb --database=pinecone --pinecone_api_key=XXX
    --workload=synthetic-proportional \
    --synthetic_records=1000 --synthetic_requests=100 \
    --synthetic_insert_ratio=0.3 --synthetic_query_ratio=0.5 \
    --synthetic_delete_ratio=0.1 --synthetic_update_ratio=0.1 \
    --synthetic_dimensions=768 --synthetic_query_distribution=zipfian \
    --synthetic_metadata=id:10n --synthetic_metadata=tags:5s10l \
    --pinecone_index_name=daver-synthetic-test
2025-01-10T14:44:11 INFO     Vector Search Bench: Starting experiment with backend='pinecone', workload='synthetic-proportional', users=1, requests_per_sec=unlimited
...
2025-01-10T14:44:12 CRITICAL PineconeDB: Index 'daver-synthetic-test' already exists - cowardly refusing to overwrite existing data. Specify --overwrite to delete it, or specify --skip-populate to skip population phase.
Traceback (most recent call last):
  File "/Users/dave/repos/pinecone-io/VSB/vsb/users.py", line 64, in setup
    self.database.initialize_population()
  File "/Users/dave/repos/pinecone-io/VSB/vsb/databases/pinecone/pinecone.py", line 155, in initialize_population
    raise StopUser()
locust.exception.StopUser
Performing Setup phase                         0/? ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0:00:00   %

This causes the "CRITICAL" message to get somewhat lost.

Solution

Modify the handling of critical errors to use an explicit DoneFailed state in the state machine, which handles the shutdown without adding backtraces, also also updating messages to be more applicable to a failed run:

2025-01-10T14:47:20 INFO     Vector Search Bench: Starting experiment with backend='pinecone', workload='synthetic-proportional', users=1, requests_per_sec=unlimited
...
2025-01-10T14:47:22 CRITICAL PineconeDB: Index 'daver-synthetic-test' already exists - cowardly refusing to overwrite existing data. Specify --overwrite to delete it, or specify --skip-populate to skip population phase.
✘ Setup cancelled                              1/1 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 00:00 ~

Type of Change

  • Bug fix (non-breaking change which fixes an issue)

Base automatically changed from daver/grpc_spurious_messages to main January 10, 2025 16:20
On a critical error (e.g. existing index and --overwrite not specified), VSB prints a large amount of spurious / unnecessary output - e.g:

```
vsb --database=pinecone --pinecone_api_key=XXX
    --workload=synthetic-proportional \
    --synthetic_records=1000 --synthetic_requests=100 \
    --synthetic_insert_ratio=0.3 --synthetic_query_ratio=0.5 \
    --synthetic_delete_ratio=0.1 --synthetic_update_ratio=0.1 \
    --synthetic_dimensions=768 --synthetic_query_distribution=zipfian \
    --synthetic_metadata=id:10n --synthetic_metadata=tags:5s10l \
    --pinecone_index_name=daver-synthetic-test
2025-01-10T14:44:11 INFO     Vector Search Bench: Starting experiment with backend='pinecone', workload='synthetic-proportional', users=1, requests_per_sec=unlimited
...
2025-01-10T14:44:12 CRITICAL PineconeDB: Index 'daver-synthetic-test' already exists - cowardly refusing to overwrite existing data. Specify --overwrite to delete it, or specify --skip-populate to skip population phase.
Traceback (most recent call last):
  File "/Users/dave/repos/pinecone-io/VSB/vsb/users.py", line 64, in setup
    self.database.initialize_population()
  File "/Users/dave/repos/pinecone-io/VSB/vsb/databases/pinecone/pinecone.py", line 155, in initialize_population
    raise StopUser()
locust.exception.StopUser
Performing Setup phase                         0/? ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0:00:00   %
```

This causes the "CRITICAL" message to get somewhat lost. Modify the handling of critical errors to use an explicit DoneFailed state in the state machine, which handles the shutdown without adding backtraces, also also updating messages to be more applicable to a failed run:

```
2025-01-10T14:47:20 INFO     Vector Search Bench: Starting experiment with backend='pinecone', workload='synthetic-proportional', users=1, requests_per_sec=unlimited
...
2025-01-10T14:47:22 CRITICAL PineconeDB: Index 'daver-synthetic-test-for-edo' already exists - cowardly refusing to overwrite existing data. Specify --overwrite to delete it, or specify --skip-populate to skip population phase.
✘ Setup cancelled                              1/1 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 00:00 ~
```
@daverigby daverigby force-pushed the daver/spurious_log_msgs_on_error branch from d54e357 to 867549e Compare January 10, 2025 20:24
@daverigby daverigby merged commit 8766e51 into main Jan 13, 2025
7 checks passed
@daverigby daverigby deleted the daver/spurious_log_msgs_on_error branch January 13, 2025 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant