Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[server][controller] Add MaterializedViewWriter and support view writers in L/F #1296

Merged
merged 11 commits into from
Jan 17, 2025
Merged

[server][controller] Add MaterializedViewWriter and support view writers in L/F #1296

merged 11 commits into from
Jan 17, 2025

Conversation

xunyin8
Copy link
Contributor

@xunyin8 xunyin8 commented Nov 12, 2024 •
edited
Loading

[server][controller] Add MaterializedViewWriter and support view writers in L/F

  1. View writers will be invoked in L/F SIT too instead of only in A/A SIT. We rely on view config validation to ensure views that do require A/A are only added to stores with A/A enabled.

  2. This PR only includes creation of materialized view topics, writing of data records and control messages to the materialized view topics in server and controller.

  • Materialized view topics are created during version creation time along with other view topics.
  • SOP is sent during view topic creation time with same chunking and compression configs as the store version.
  • EOP is sent when servers have reported EOP in every partition.
  • Incremental push control messages SOIP and EOIP are not propagated to the view topic for now because the end to end incremental push tracking story for view topics is not clear yet. Store owners will likely just disable the requirement to wait for view consumers to fully ingest the incremental push.
  • Ingestion heartbeats will not be propagated. We will broadcast the heartbeat and leader complete state in a separate PR.
  • Version swap for CDC users will be implemented in a separate PR to keep this PR somewhat short for review.
  1. One issue to be resolved is that during processing of batch records in the native replication source fabric, where we consume local VT, a leader transfer could result in missing records in the materialized view topic. This is because we don't do any global checkpointing across leader and followers when consuming local VT. To solve this we will be producing to the view topic from the VPJ itself.

How was this PR tested?

Unit and integration tests

Does this PR introduce any user-facing changes?

  • No. You can skip the rest of this section.
  • Yes. Make sure to explain your proposed changes and call out the behavior change.

@xunyin8 xunyin8 force-pushed the RePartitionViewWriter branch 2 times, most recently from e43850a to a711263 Compare November 13, 2024 02:32
@xunyin8 xunyin8 force-pushed the RePartitionViewWriter branch 6 times, most recently from 791712b to bcee9bc Compare November 25, 2024 17:47
@xunyin8 xunyin8 force-pushed the RePartitionViewWriter branch from bcee9bc to 3c59c8c Compare December 2, 2024 23:14
@xunyin8 xunyin8 force-pushed the RePartitionViewWriter branch from 3c59c8c to 37c15cc Compare December 3, 2024 18:34
gaojieliu
gaojieliu reviewed Dec 4, 2024
View reviewed changes
...ient/src/main/java/com/linkedin/davinci/kafka/consumer/LeaderFollowerStoreIngestionTask.java Outdated Show resolved Hide resolved
...client/src/main/java/com/linkedin/davinci/kafka/consumer/ActiveActiveStoreIngestionTask.java Outdated Show resolved Hide resolved
...client/src/main/java/com/linkedin/davinci/kafka/consumer/ActiveActiveStoreIngestionTask.java Outdated Show resolved Hide resolved
...ient/src/main/java/com/linkedin/davinci/kafka/consumer/LeaderFollowerStoreIngestionTask.java Outdated Show resolved Hide resolved
...client/src/main/java/com/linkedin/davinci/kafka/consumer/ActiveActiveStoreIngestionTask.java Outdated Show resolved Hide resolved
clients/venice-push-job/src/main/java/com/linkedin/venice/hadoop/VenicePushJob.java Outdated Show resolved Hide resolved
...-job/src/main/java/com/linkedin/venice/hadoop/mapreduce/datawriter/reduce/VeniceReducer.java Outdated Show resolved Hide resolved
internal/venice-common/src/main/java/com/linkedin/venice/views/ChangeCaptureView.java Show resolved Hide resolved
internal/venice-common/src/main/java/com/linkedin/venice/writer/CompositeVeniceWriter.java Outdated Show resolved Hide resolved
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java Show resolved Hide resolved
@xunyin8 xunyin8 force-pushed the RePartitionViewWriter branch 4 times, most recently from 51028b4 to 1e7493f Compare January 7, 2025 16:25
gaojieliu
gaojieliu reviewed Jan 8, 2025
View reviewed changes
Copy link
Contributor

@gaojieliu gaojieliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few more comments.

...ient/src/main/java/com/linkedin/davinci/kafka/consumer/LeaderFollowerStoreIngestionTask.java Show resolved Hide resolved
...ient/src/main/java/com/linkedin/davinci/kafka/consumer/LeaderFollowerStoreIngestionTask.java Outdated Show resolved Hide resolved
...s/da-vinci-client/src/main/java/com/linkedin/davinci/store/view/VeniceViewWriterFactory.java Outdated Show resolved Hide resolved
internal/venice-common/src/main/java/com/linkedin/venice/writer/CompositeVeniceWriter.java Outdated Show resolved Hide resolved
internal/venice-common/src/main/java/com/linkedin/venice/writer/CompositeVeniceWriter.java Show resolved Hide resolved
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java Show resolved Hide resolved
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java Show resolved Hide resolved
...ces/venice-controller/src/main/java/com/linkedin/venice/pushmonitor/AbstractPushMonitor.java Show resolved Hide resolved
@xunyin8 xunyin8 force-pushed the RePartitionViewWriter branch 7 times, most recently from 3ad1097 to 88ec04b Compare January 14, 2025 19:56
gaojieliu
gaojieliu reviewed Jan 15, 2025
View reviewed changes
Copy link
Contributor

@gaojieliu gaojieliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a major comment regarding LeaderCompleteState HB messages and we don't need to address it in this PR and all other comments are very minor.

Thanks a lot for addressing the comments quickly!

...ient/src/main/java/com/linkedin/davinci/kafka/consumer/LeaderFollowerStoreIngestionTask.java Outdated Show resolved Hide resolved
...ts/da-vinci-client/src/main/java/com/linkedin/davinci/store/view/MaterializedViewWriter.java Show resolved Hide resolved
...venice-push-job/src/test/java/com/linkedin/venice/hadoop/mapreduce/AbstractTestVeniceMR.java Show resolved Hide resolved
internal/venice-common/src/main/java/com/linkedin/venice/writer/CompositeVeniceWriter.java Outdated Show resolved Hide resolved
internal/venice-common/src/main/java/com/linkedin/venice/writer/CompositeVeniceWriter.java Outdated Show resolved Hide resolved
internal/venice-common/src/main/java/com/linkedin/venice/writer/CompositeVeniceWriter.java Outdated Show resolved Hide resolved
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java Outdated Show resolved Hide resolved
...ces/venice-controller/src/main/java/com/linkedin/venice/pushmonitor/AbstractPushMonitor.java Outdated Show resolved Hide resolved
...ces/venice-controller/src/main/java/com/linkedin/venice/pushmonitor/AbstractPushMonitor.java Outdated Show resolved Hide resolved
@xunyin8 xunyin8 force-pushed the RePartitionViewWriter branch 2 times, most recently from 979089c to a5260e7 Compare January 16, 2025 19:30
xunyin8 added 10 commits January 16, 2025 17:21
@xunyin8
[server][controller] Add MaterializedViewWriter and support view writ…
08106ed 
…ers in L/F

1. View writers will be invoked in L/F SIT too instead of only in A/A SIT. We rely on
view config validation to ensure views that do require A/A are only added to stores
with A/A enabled.

2. This PR only includes creation of materialized view topics, writing of data
records and control messages to the materialized view topics  in server and controller.
  - Materialized view topics are created during version creation time along with other
    view topics.
  - SOP is sent during view topic creation time with same chunking and compression
    configs as the store version.
  - EOP is sent when servers have reported EOP in every partition.
  - Incremental push control messages SOIP and EOIP are not propagated to the view topic
    for now because the end to end incremental push tracking story for view topics is
    not clear yet. Store owners will likely just disable the requirement to wait for
    view consumers to fully ingest the incremental push.
  - Ingestion heartbeats will be propagated in a broadcast manner. See implementation
    for details.
  - Version swap for CDC users will be implemented in a separate PR to keep this PR
    somewhat short for review.

3. TODO: one pending issue to be resolved is that during processing of batch records
in the native replication source fabric, where we consume local VT, a leader transfer
could result in missing records in the materialized view topic. This is because we
don't do any global checkpointing across leader and followers when consuming local VT.
@xunyin8
Add unit tests and make VeniceViewWriter version specific (it already…
42b0ea6 
… is implicitly)
@xunyin8
Added CompositeVeniceWriter to write batch records from VPJ in NR sou…
51d0a6c 
…rce fabric
@xunyin8
Added unit tests
1bb1d14
@xunyin8
Unit test coverage
d24aeb6
@xunyin8
Address review comments
8bcccb9
@xunyin8
Address more review comments
ff52b5b
@xunyin8
Fixed E2E test to validate all child fabrics and added a test for re-…
215d6a6 
…push
@xunyin8
Addressed more comments
d1864f5 
1. Removed heartbeat propagation
2. Refactored CompositeVeniceWriter and reduced code dup
3. Better CM coordination in L/F SIT for segment and heartbeat CMs
@xunyin8
Handle leader complete state and refactoring for code dup and testing
e201bdc
@xunyin8 xunyin8 force-pushed the RePartitionViewWriter branch 2 times, most recently from ddc56a0 to 4e07678 Compare January 17, 2025 18:41
@xunyin8 xunyin8 force-pushed the RePartitionViewWriter branch from 4e07678 to dc8be78 Compare January 17, 2025 20:35
gaojieliu
gaojieliu approved these changes Jan 17, 2025
View reviewed changes
Copy link
Contributor

@gaojieliu gaojieliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thanks!

@xunyin8 xunyin8 merged commit 5a81893 into linkedin:main Jan 17, 2025
57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lusong64 lusong64 lusong64 left review comments

@gaojieliu gaojieliu gaojieliu approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@xunyin8 @lusong64 @gaojieliu