[server] Added adaptive throttler for tuning ingestion bandwidth dyn… #3990
majisourav99
Matrix: Clients / UT & CodeCov
Matrix: Controller / UT & CodeCov
Matrix: Integrations / UT & CodeCov
Matrix: Internal / UT & CodeCov
Matrix: Router / UT & CodeCov
Matrix: Server / UT & CodeCov
Matrix: StaticAnalysis
ValidateGradleWrapper
11s
StaticAnalysisAndUnitTestsCompletionCheck
0s
Annotations
8 errors and 21 warnings
|
SITWithTWiseAndBufferAfterLeaderTest.testRecordLevelMetricForCurrentVersion[0](false):
clients/da-vinci-client/src/test/java/com/linkedin/davinci/kafka/consumer/SITWithTWiseAndBufferAfterLeaderTest.java#L1
org.mockito.exceptions.verification.TooFewActualInvocations:
hostLevelIngestionStats.recordTotalBytesConsumed(
<any long>
);
Wanted 2 times:
-> at com.linkedin.davinci.stats.HostLevelIngestionStats.recordTotalBytesConsumed(HostLevelIngestionStats.java:492)
But was 1 time:
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerRecord(StoreIngestionTask.java:2595)
|
|
SITWithTWiseAndBufferAfterLeaderTest.testRecordLevelMetricForCurrentVersion[0](false):
clients/da-vinci-client/src/test/java/com/linkedin/davinci/kafka/consumer/SITWithTWiseAndBufferAfterLeaderTest.java#L1
org.mockito.exceptions.verification.TooFewActualInvocations:
hostLevelIngestionStats.recordTotalBytesConsumed(
<any long>
);
Wanted 2 times:
-> at com.linkedin.davinci.stats.HostLevelIngestionStats.recordTotalBytesConsumed(HostLevelIngestionStats.java:492)
But was 1 time:
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerRecord(StoreIngestionTask.java:2595)
|
|
SITWithPWiseAndBufferAfterLeaderTest.testCorruptMessagesFailFast[167](AA_OFF):
clients/da-vinci-client/src/test/java/com/linkedin/davinci/kafka/consumer/SITWithPWiseAndBufferAfterLeaderTest.java#L1
Wanted but not invoked:
logNotifier.completed(
"TestTopic_50ef6ff88b_c1ed5809_v1",
1,
LongEqualOrGreaterThanMatcher(lowerBound=3),
"STANDBY"
);
-> at com.linkedin.davinci.notifier.LogNotifier.completed(LogNotifier.java:26)
Actually, there were zero interactions with this mock.
|
|
SITWithPWiseAndBufferAfterLeaderTest.testNotifier[1](AA_OFF):
clients/da-vinci-client/src/test/java/com/linkedin/davinci/kafka/consumer/SITWithPWiseAndBufferAfterLeaderTest.java#L1
org.mockito.exceptions.verification.ArgumentsAreDifferent: Argument(s) are different! Wanted:
storageMetadataService.put(
"TestTopic_64203ef6a0_4712f31f_v1",
2,
OffsetRecord{localVersionTopicOffset=3, upstreamOffset=-1, leaderTopic=null, offsetLag=0, eventTimeEpochMs=-1, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=true, databaseInfo={}, realTimeProducerState={}}
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTaskTest.lambda$testNotifier$35(StoreIngestionTaskTest.java:1623)
Actual invocations have different arguments:
storageMetadataService.getLastOffset(
"TestTopic_64203ef6a0_4712f31f_v1",
1
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.getLastOffset(DeepCopyOffsetManager.java:49)
storageMetadataService.getLastOffset(
"TestTopic_64203ef6a0_4712f31f_v1",
2
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.getLastOffset(DeepCopyOffsetManager.java:49)
storageMetadataService.computeStoreVersionState(
"TestTopic_64203ef6a0_4712f31f_v1",
com.linkedin.venice.offsets.DeepCopyStorageMetadataService$$Lambda$700/364408566@7580c45d
);
-> at com.linkedin.venice.offsets.DeepCopyStorageMetadataService.computeStoreVersionState(DeepCopyStorageMetadataService.java:33)
storageMetadataService.put(
"TestTopic_64203ef6a0_4712f31f_v1",
1,
OffsetRecord{localVersionTopicOffset=1, upstreamOffset=-1, leaderTopic=null, offsetLag=9223372036854775807, eventTimeEpochMs=1736991097381, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=false, databaseInfo={}, realTimeProducerState={}}
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.put(DeepCopyOffsetManager.java:38)
storageMetadataService.computeStoreVersionState(
"TestTopic_64203ef6a0_4712f31f_v1",
com.linkedin.venice.offsets.DeepCopyStorageMetadataService$$Lambda$700/364408566@5cab11db
);
-> at com.linkedin.venice.offsets.DeepCopyStorageMetadataService.computeStoreVersionState(DeepCopyStorageMetadataService.java:33)
storageMetadataService.put(
"TestTopic_64203ef6a0_4712f31f_v1",
1,
OffsetRecord{localVersionTopicOffset=2, upstreamOffset=-1, leaderTopic=null, offsetLag=9223372036854775807, eventTimeEpochMs=1736991097381, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=false, databaseInfo={}, realTimeProducerState={}}
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.put(DeepCopyOffsetManager.java:38)
storageMetadataService.computeStoreVersionState(
"TestTopic_64203ef6a0_4712f31f_v1",
com.linkedin.venice.offsets.DeepCopyStorageMetadataService$$Lambda$700/364408566@4658b4ef
);
-> at com.linkedin.venice.offsets.DeepCopyStorageMetadataService.computeStoreVersionState(DeepCopyStorageMetadataService.java:33)
storageMetadataService.put(
"TestTopic_64203ef6a0_4712f31f_v1",
2,
OffsetRecord{localVersionTopicOffset=1, upstreamOffset=-1, leaderTopic=null, offsetLag=9223372036854775807, eventTimeEpochMs=1736991097381, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=false, databaseInfo={}, realTimeProducerState={}}
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.put(DeepCopyOffsetManager.java:38)
storageMetadataService.put(
"TestTopic_64203ef6a0_4712f31f_v1",
1,
OffsetRecord{localVersionTopicOffset=3, upstreamOffset=-1, leaderTopic=null, offsetLag=9223372036854775807, eventTimeEpochMs=1736991097381, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=true, databaseInfo={}, realTimeProducerState={}}
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.put(DeepCopyOffsetManager.java:38)
storageMetadataService.put(
"TestTopic_64203ef6a0_4712f31f_v1",
2,
OffsetRecord{localVersionTopicOffset=2, upstreamOffset=-1, leaderTopic=null, offsetLag=9223372036854775807, eventTimeEpochMs=1736991097381, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=false, databaseInfo={}, realTimeProducerState={}}
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.put(DeepCopyOffsetManager.java:38)
storageMetadataService.computeStoreVersionState(
"TestTopic_64203ef6a0_4712f31f_v1",
com.linkedin.venice.offset
|
|
NativeMetadataRepositoryTest.testNativeMetadataRepositoryStats:
clients/da-vinci-client/src/test/java/com/linkedin/davinci/repository/NativeMetadataRepositoryTest.java#L133
java.lang.AssertionError: expected [2000.0] but found [1000.0]
|
|
SITWithPWiseAndBufferAfterLeaderTest.testCorruptMessagesFailFast[167](AA_OFF):
clients/da-vinci-client/src/test/java/com/linkedin/davinci/kafka/consumer/SITWithPWiseAndBufferAfterLeaderTest.java#L1
Wanted but not invoked:
logNotifier.completed(
"TestTopic_50ef6ff88b_c1ed5809_v1",
1,
LongEqualOrGreaterThanMatcher(lowerBound=3),
"STANDBY"
);
-> at com.linkedin.davinci.notifier.LogNotifier.completed(LogNotifier.java:26)
Actually, there were zero interactions with this mock.
|
|
SITWithPWiseAndBufferAfterLeaderTest.testNotifier[1](AA_OFF):
clients/da-vinci-client/src/test/java/com/linkedin/davinci/kafka/consumer/SITWithPWiseAndBufferAfterLeaderTest.java#L1
org.mockito.exceptions.verification.ArgumentsAreDifferent: Argument(s) are different! Wanted:
storageMetadataService.put(
"TestTopic_64203ef6a0_4712f31f_v1",
2,
OffsetRecord{localVersionTopicOffset=3, upstreamOffset=-1, leaderTopic=null, offsetLag=0, eventTimeEpochMs=-1, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=true, databaseInfo={}, realTimeProducerState={}}
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTaskTest.lambda$testNotifier$35(StoreIngestionTaskTest.java:1623)
Actual invocations have different arguments:
storageMetadataService.getLastOffset(
"TestTopic_64203ef6a0_4712f31f_v1",
1
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.getLastOffset(DeepCopyOffsetManager.java:49)
storageMetadataService.getLastOffset(
"TestTopic_64203ef6a0_4712f31f_v1",
2
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.getLastOffset(DeepCopyOffsetManager.java:49)
storageMetadataService.computeStoreVersionState(
"TestTopic_64203ef6a0_4712f31f_v1",
com.linkedin.venice.offsets.DeepCopyStorageMetadataService$$Lambda$700/364408566@7580c45d
);
-> at com.linkedin.venice.offsets.DeepCopyStorageMetadataService.computeStoreVersionState(DeepCopyStorageMetadataService.java:33)
storageMetadataService.put(
"TestTopic_64203ef6a0_4712f31f_v1",
1,
OffsetRecord{localVersionTopicOffset=1, upstreamOffset=-1, leaderTopic=null, offsetLag=9223372036854775807, eventTimeEpochMs=1736991097381, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=false, databaseInfo={}, realTimeProducerState={}}
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.put(DeepCopyOffsetManager.java:38)
storageMetadataService.computeStoreVersionState(
"TestTopic_64203ef6a0_4712f31f_v1",
com.linkedin.venice.offsets.DeepCopyStorageMetadataService$$Lambda$700/364408566@5cab11db
);
-> at com.linkedin.venice.offsets.DeepCopyStorageMetadataService.computeStoreVersionState(DeepCopyStorageMetadataService.java:33)
storageMetadataService.put(
"TestTopic_64203ef6a0_4712f31f_v1",
1,
OffsetRecord{localVersionTopicOffset=2, upstreamOffset=-1, leaderTopic=null, offsetLag=9223372036854775807, eventTimeEpochMs=1736991097381, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=false, databaseInfo={}, realTimeProducerState={}}
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.put(DeepCopyOffsetManager.java:38)
storageMetadataService.computeStoreVersionState(
"TestTopic_64203ef6a0_4712f31f_v1",
com.linkedin.venice.offsets.DeepCopyStorageMetadataService$$Lambda$700/364408566@4658b4ef
);
-> at com.linkedin.venice.offsets.DeepCopyStorageMetadataService.computeStoreVersionState(DeepCopyStorageMetadataService.java:33)
storageMetadataService.put(
"TestTopic_64203ef6a0_4712f31f_v1",
2,
OffsetRecord{localVersionTopicOffset=1, upstreamOffset=-1, leaderTopic=null, offsetLag=9223372036854775807, eventTimeEpochMs=1736991097381, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=false, databaseInfo={}, realTimeProducerState={}}
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.put(DeepCopyOffsetManager.java:38)
storageMetadataService.put(
"TestTopic_64203ef6a0_4712f31f_v1",
1,
OffsetRecord{localVersionTopicOffset=3, upstreamOffset=-1, leaderTopic=null, offsetLag=9223372036854775807, eventTimeEpochMs=1736991097381, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=true, databaseInfo={}, realTimeProducerState={}}
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.put(DeepCopyOffsetManager.java:38)
storageMetadataService.put(
"TestTopic_64203ef6a0_4712f31f_v1",
2,
OffsetRecord{localVersionTopicOffset=2, upstreamOffset=-1, leaderTopic=null, offsetLag=9223372036854775807, eventTimeEpochMs=1736991097381, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=false, databaseInfo={}, realTimeProducerState={}}
);
-> at com.linkedin.venice.offsets.DeepCopyOffsetManager.put(DeepCopyOffsetManager.java:38)
storageMetadataService.computeStoreVersionState(
"TestTopic_64203ef6a0_4712f31f_v1",
com.linkedin.venice.offset
|
|
NativeMetadataRepositoryTest.testNativeMetadataRepositoryStats:
clients/da-vinci-client/src/test/java/com/linkedin/davinci/repository/NativeMetadataRepositoryTest.java#L133
java.lang.AssertionError: expected [2000.0] but found [1000.0]
|
|
ValidateGradleWrapper
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Integrations / UT & CodeCov (8)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Integrations / UT & CodeCov (17)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Integrations / UT & CodeCov (11)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Controller / UT & CodeCov (17)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Controller / UT & CodeCov (8)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Controller / UT & CodeCov (11)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Router / UT & CodeCov (8)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Router / UT & CodeCov (17)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
StaticAnalysis (17)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Router / UT & CodeCov (11)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Internal / UT & CodeCov (17)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Internal / UT & CodeCov (11)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Internal / UT & CodeCov (8)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Server / UT & CodeCov (17)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Server / UT & CodeCov (11)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Server / UT & CodeCov (8)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Clients / UT & CodeCov (8)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Clients / UT & CodeCov (17)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
Clients / UT & CodeCov (11)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
|
StaticAnalysisAndUnitTestsCompletionCheck
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
Artifacts
Produced during runtime
| Name | Size | |
|---|---|---|
|
StaticAnalysis
|
651 KB |
|
|
clients-jdk11
|
2.52 MB |
|
|
clients-jdk17
|
2.54 MB |
|
|
clients-jdk8
|
2.48 MB |
|
|
controller-jdk11
|
1.44 MB |
|
|
controller-jdk17
|
1.44 MB |
|
|
controller-jdk8
|
1.44 MB |
|
|
integrations-jdk11
|
515 KB |
|
|
integrations-jdk17
|
519 KB |
|
|
integrations-jdk8
|
502 KB |
|
|
internal-jdk11
|
3.23 MB |
|
|
internal-jdk17
|
3.24 MB |
|
|
internal-jdk8
|
3.21 MB |
|
|
router-jdk11
|
1.05 MB |
|
|
router-jdk17
|
1.03 MB |
|
|
router-jdk8
|
1.02 MB |
|
|
server-jdk11
|
9.11 MB |
|
|
server-jdk17
|
9.14 MB |
|
|
server-jdk8
|
9.37 MB |
|