Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RayService] Trim Redis Cleanup job less than 63 chars #2846

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

[RayService] Trim Redis Cleanup job less than 63 chars #2846

wants to merge 6 commits into from
+28 −5

Conversation

aviadshimoni
Copy link
Contributor

@aviadshimoni aviadshimoni commented Jan 28, 2025 •
edited
Loading

Why are these changes needed?

When the RayCluster name is too long, the Redis cleanup job name and associated labels exceed Kubernetes' 63-character limit. This causes the cleanup job to fail during creation with the following error:

Warning  FailedToCreateRedisCleanupJob  raycluster/text-translation-model-rayservice-raycluster-g9gps        Failed to create Redis cleanup Job text-translation-model/text-translation-model-rayservice-raycluster-g9gps-redis-cleanup, Job.batch "text-translation-model-rayservice-raycluster-g9gps-redis-cleanup" is invalid: spec.template.labels: Invalid value: "text-translation-model-rayservice-raycluster-g9gps-redis-cleanup": must be no more than 63 characters

Related issue number

Closes #2845

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(
image

Update: Tested ✅ , great development doc! 🥇

@aviadshimoni aviadshimoni changed the title Trim Redis Cleanup job and label to less than 63 Characters [RayService] Trim Redis Cleanup job to less than 63 chars Jan 30, 2025
@aviadshimoni
1. trim labels value like here: https://github.com/ray-project/kubera…
c219805 
…y/blob/master/ray-operator/controllers/ray/common/service.go#L27

2. interface function TrimJobName for clarity

3. Keep job for 3 minutes after completion for debugging
@aviadshimoni aviadshimoni changed the title [RayService] Trim Redis Cleanup job to less than 63 chars [RayService] Trim Redis Cleanup job less than 63 chars Jan 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DanielChern DanielChern DanielChern left review comments

@andrewsykim andrewsykim andrewsykim left review comments

@rueian rueian rueian left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@andrewsykim andrewsykim

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] Fix long job names and labels exceeding 63 characters in Redis cleanup jobs
4 participants
@aviadshimoni @rueian @andrewsykim @DanielChern