Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UfsSyncPathCache Cache Eviction Recursion Issue #18641

Open
ssyssy opened this issue Jun 28, 2024 · 1 comment
Open

UfsSyncPathCache Cache Eviction Recursion Issue #18641

ssyssy opened this issue Jun 28, 2024 · 1 comment
Labels
type-bug This issue is about a bug

Comments

@ssyssy
Copy link
Contributor

ssyssy commented Jun 28, 2024

Alluxio Version:
master-2.x

Describe the bug
I found a bug in master-2.x, in the onCacheEviction handler, when the guava cache in UfsSyncPathCache is full and there are 1000 threads concurrently syncing paths, there will be a recursion loop which would lead to stack overflow and lock all the rpcs threads.

To Reproduce
Having a large number(~1000) of threads, concurrently syncing different paths.

Expected behavior
Threads won't get locked, even there is a large traffic.

Urgency
This could influence users when the scale is large, leading to all the rpc threads in the master is dead, and thus block all the services.

Are you planning to fix it
#18640

Additional context
Here is the Jstack of one thread we found in the bug:
"master-rpc-executor-TPE-thread-31670" #73207 daemon prio=5 os_prio=0 cpu=490.13ms elapsed=114135.12s tid=0x0000ffc5c8545800 nid=0x539fb waiting on condition [0x0000ffc3fedf9000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x0000ffc8cdfed8b0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt([email protected]/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued([email protected]/AbstractQueuedSynchronizer.java:917)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:1240)
at java.util.concurrent.locks.ReentrantLock.lock([email protected]/ReentrantLock.java:267)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2199)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:312)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:312)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:312)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:312)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:312)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:312)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:312)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:312)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifyInvalidationInternal(UfsSyncPathCache.java:294)
at alluxio.master.file.meta.UfsSyncPathCache.onCacheEviction(UfsSyncPathCache.java:119)
at alluxio.master.file.meta.UfsSyncPathCache.lambda$new$0(UfsSyncPathCache.java:101)
at alluxio.master.file.meta.UfsSyncPathCache$$Lambda$354/0x0000ffc5e5e104b0.onRemoval(Unknown Source)
at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1814)
at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3467)
at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3443)
at com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2270)
at com.google.common.cache.LocalCache.compute(LocalCache.java:4212)
at alluxio.master.file.meta.UfsSyncPathCache.notifySyncedPath(UfsSyncPathCache.java:355)
at alluxio.master.file.InodeSyncStream.syncInternal(InodeSyncStream.java:712)
at alluxio.master.file.InodeSyncStream.sync(InodeSyncStream.java:481)
at alluxio.master.file.DefaultFileSystemMaster.syncMetadata(DefaultFileSystemMaster.java:4370)
at alluxio.master.file.DefaultFileSystemMaster.getFileInfo(DefaultFileSystemMaster.java:1036)
at alluxio.master.file.PrivilegedFileSystemMaster.getFileInfo(PrivilegedFileSystemMaster.java:261)
at alluxio.master.file.FileSystemMasterClientServiceHandler.lambda$getStatus$12(FileSystemMasterClientServiceHandler.java:343)
at alluxio.master.file.FileSystemMasterClientServiceHandler$$Lambda$1597/0x0000ffc3c5f3bd08.call(Unknown Source)
at alluxio.RpcUtils.callAndReturn(RpcUtils.java:129)
at alluxio.RpcUtils.call(RpcUtils.java:87)
at alluxio.master.file.FileSystemMasterClientServiceHandler.getStatus(FileSystemMasterClientServiceHandler.java:340)
at alluxio.grpc.FileSystemMasterClientServiceGrpc$MethodHandlers.invoke(FileSystemMasterClientServiceGrpc.java:3532)
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
at alluxio.security.authentication.ClientContextServerInjector$1.onHalfClose(ClientContextServerInjector.java:94)
at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
at alluxio.security.authentication.AuthenticatedUserInjector$1.onHalfClose(AuthenticatedUserInjector.java:67)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:797)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at java.lang.Thread.run([email protected]/Thread.java:829)

Locked ownable synchronizers:
- <0x0000ffcd968bc288> (a java.util.concurrent.ThreadPoolExecutor$Worker)
@ssyssy ssyssy added the type-bug This issue is about a bug label Jun 28, 2024
@ljluestc
Copy link

ljluestc commented Jul 4, 2024 

package alluxio.master.file.meta;

import com.google.common.cache.CacheBuilder;
import com.google.common.cache.CacheLoader;
import com.google.common.cache.LoadingCache;
import com.google.common.cache.RemovalListener;
import com.google.common.cache.RemovalNotification;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;

public class UfsSyncPathCache {

    private final LoadingCache<String, String> mCache;
    private final LinkedBlockingQueue<String> mEvictionQueue = new LinkedBlockingQueue<>();
    private boolean mEvictionInProgress = false;
    private final ExecutorService mEvictionExecutor = Executors.newSingleThreadExecutor();

    public UfsSyncPathCache() {
        mCache = CacheBuilder.newBuilder()
                .maximumSize(1000)
                .removalListener(new RemovalListener<String, String>() {
                    @Override
                    public void onRemoval(RemovalNotification<String, String> notification) {
                        handleEviction(notification.getKey());
                    }
                })
                .build(new CacheLoader<String, String>() {
                    @Override
                    public String load(String key) throws Exception {
                        return syncPath(key);
                    }
                });
    }

    private void handleEviction(String key) {
        mEvictionQueue.add(key);
        processEvictionQueue();
    }

    private synchronized void processEvictionQueue() {
        if (mEvictionInProgress) {
            return;
        }
        mEvictionInProgress = true;
        mEvictionExecutor.submit(() -> {
            try {
                while (!mEvictionQueue.isEmpty()) {
                    String key = mEvictionQueue.poll();
                    if (key != null) {
                        notifyInvalidationInternal(key);
                    }
                }
            } finally {
                mEvictionInProgress = false;
            }
        });
    }

    private void notifyInvalidationInternal(String key) {
        // Existing invalidation logic
        System.out.println("Invalidating path: " + key);
    }

    private String syncPath(String path) {
        // Simulate path syncing logic
        System.out.println("Syncing path: " + path);
        return path;
    }

    public void sync(String path) {
        mCache.getUnchecked(path);
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type-bug This issue is about a bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ssyssy @ljluestc