Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(replays): Add buffered consumer implementation #85356

Open
wants to merge 46 commits into
base: master
Choose a base branch
from

Conversation

cmanallen
Copy link
Member

@cmanallen cmanallen commented Feb 18, 2025

TODO description...

Partially addresses the DACI: https://www.notion.so/sentry/DACI-Session-Replay-Recording-Consumer-Stability-and-Performance-Improvements-19e8b10e4b5d80a192a1ecd46f13eebb

How it works:

  • As messages come in they are processed and their processed results are stored on a queue.
  • When the queue fills up the processed messages are flushed.
    • Flushing involves committing data to GCS, ClickHouse, BigQuery, DataDog.
    • Anything I/O related.
  • Flushing happens in a thread-pool.

This closely mirrors current production behavior except processing is no longer done in the thread-pool.

This PR also introduces a new RunTime abstraction for managing state changes in the consumer. Which I will document in a Notion doc.

@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Feb 18, 2025
Copy link

codecov bot commented Feb 20, 2025

Codecov Report

Attention: Patch coverage is 98.68132% with 6 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/sentry/replays/consumers/buffered/platform.py 96.36% 4 Missing ⚠️
src/sentry/replays/consumers/buffered/consumer.py 98.03% 1 Missing ⚠️
...ests/sentry/replays/unit/consumers/test_runtime.py 98.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           master   #85356    +/-   ##
========================================
  Coverage   87.89%   87.89%            
========================================
  Files        9701     9711    +10     
  Lines      550004   550638   +634     
  Branches    21376    21376            
========================================
+ Hits       483412   484007   +595     
- Misses      66283    66322    +39     
  Partials      309      309            

@cmanallen cmanallen marked this pull request as ready for review February 28, 2025 16:58
@cmanallen cmanallen requested review from a team as code owners February 28, 2025 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant