Scheduler subsystem high level design proposal #603
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: smarterclayton The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
size/L
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
This sets down basic design principles of the current gateway scheduler. We also highlight who we are targeting as users, and why we prioritize the current approach. It also selects standard terminology for scheduling that the implementation should adopt. This is a high level design and thus sets general scope, without expecting to fully address all problems.
|
|
||
| #### Replacement Scheduler | ||
|
|
||
| The replacement scheduler will be a low-latency mechanism for out-of-process execution of the core endpoint selection option. The replacement scheduler will accept one or more requests to schedule, a list of endpoints, and optionally the associated informer state for those endpoints. The replacement scheduler will return one or zero endpoints per request. |
There was a problem hiding this comment.
You mention above:
the scheduler will consult a list of configured scorers to score the matches into a prioritized list of endpoints.
Should that just be the expected output of any scheduler?
It could allow the Personas to do cool parity comparisons, i.e. (run these 3 schedulers in parallel, but treat the reference scheduler as the source of truth, and only block on the response of that one) Which could be a useful/safe way to roll out algorithms in production traffic
There was a problem hiding this comment.
Yeah, that layering is that filtering and scoring and prioritization are three phases.
Being able to run a shadow schedule is indeed valuable and aligns to the rollout use cases.
|
|
||
| A benchmarking harness will be provided to capture and reproduce a production trace, primarily to aid algorithmic contributors. A small but diverse set of production traces will be used initially to anchor expectations, and scaling both the number of supported traces and efficient regression testing at scale will be critical. | ||
|
|
||
| We anticipate that accelerator availability will limit the scale of e2e testing and contribution. We will develop a **model server stub** that can emulate the behavior of the core expected algorithm for model servers and does not require accelerators. We will support both time-accurate and configurable ratio emulation to allow fast execution. |
| ## Non-Goals | ||
|
|
||
| - Dynamic reconfiguration of the reference scheduler algorithms at runtime | ||
| - Being a general scheduler framework for load balancing |
There was a problem hiding this comment.
I agree with this. I'm trying to think through how we enforce/where we draw the line. What do we consider in scope? There will be some overlap likely, but any addition/improvement should probably have an inference-specific justification.
| - The scheduler should be educatable - extending the [model server protocol](../003-model-server-protocol/) with new metrics or adding a new source of data should be minimally invasive | ||
| - The scheduler should be replaceable - the reference endpoint picker implementation should support delegating scheduling decisions per pool to an alternative **replacement scheduler** | ||
|
|
||
| ## Non-Goals |
There was a problem hiding this comment.
I think we should specify that a non-goal would be the concept of:
How to graduate ideas from: fork -> included in reference scheduler -> exposed as EPP config -> config expressed in our API
Something we should figure out, but not in this doc.
|
|
||
| We desire the following outcomes from the reference scheduler: | ||
|
|
||
| 1. Keep model servers optimally utilized without saturating |
There was a problem hiding this comment.
this is mostly dictated by the qps though, the algorithm can't ensure that the model servers don't saturate
There was a problem hiding this comment.
Yeah, should be “allow model servers to more predictably approach saturation” instead
|
|
||
| 1. Keep model servers optimally utilized without saturating | ||
| 2. Make user-visible request latency more predictable | ||
| 3. Provide isolation between multiple workloads on the same model servers before saturation |
There was a problem hiding this comment.
I recommend to define saturation in the proposal
There was a problem hiding this comment.
Good point, might as well open that doc up
|
|
||
| #### Replacement Scheduler | ||
|
|
||
| The replacement scheduler will be a low-latency mechanism for out-of-process execution of the core endpoint selection option. The replacement scheduler will accept one or more requests to schedule, a list of endpoints, and optionally the associated informer state for those endpoints. The replacement scheduler will return one or zero endpoints per request. |
There was a problem hiding this comment.
Why does this look like a batch scheduler?
There was a problem hiding this comment.
1k qps will require either streaming or batching, fair point we don’t want to constrain that design yet
|
|
||
| Given that we anticipate a significant amount of future work to integrate heterogenous hardware (different generations / topologies) and heterogeous server roles (prefill-heavy, prefill/decode split, latency objectives), we expect that there will be an **assignment** informer that partitions the candidate endpoints over multiple dimensions for the scheduler. This will decouple the scheduling algorithm from the process of determining the capacity and suitability of different model servers to different dimensions of request cost. | ||
|
|
||
| #### Replacement Scheduler |
There was a problem hiding this comment.
I am not following what this section is proposing, what do we mean by replacement? replacing the reference scheduler? why is this proposal trying to define that?
There was a problem hiding this comment.
It’s kind of defined above, but implementing a whole EPP is a lot, and i don’t want to duplicate everything in another language.
There was a problem hiding this comment.
I guess my question is: are we proposing to build a second reference scheduler in this repo?
There was a problem hiding this comment.
Just the tools to do so: I'm gonna clean up & finish implementing: https://github.com/kfswain/go-py-interface/tree/main (it's hideous and incomplete, which is why I've been so cagey about it)
So the Replacement Scheduler could be a Python based scheduler, that is called via an EPP fork. If we create a simple ingress interface:
- endpoint map w/metrics
- config params
And a simple egress interface:
- scored endpoints
- maybe specify how many endpoints to duplicate to? (was mentioned as a potential need by prodstack)
implementing a new algo in Python should be straightforward
This sets down basic design principles of the current gateway scheduler. We also highlight who we are targeting as users, and why we prioritize the current approach. It also selects standard terminology for scheduling that the implementation should adopt.
This is a high level design and thus sets general scope, without expecting to fully address all problems.