feat(webhook): webhook waiting for persistency. #20164
Conversation
KeXiangWang
force-pushed
the
wkx/async-insert
branch
from
0e058d1 to
df3011a
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
license-eye has checked 5561 files.
| Valid | Invalid | Ignored | Fixed |
|---|---|---|---|
| 2346 | 3 | 3212 | 0 |
Click to see the invalid file list
- src/batch/src/executor/fast_insert.rs
- src/frontend/src/scheduler/distributed/fast_insert.rs
- src/frontend/src/scheduler/fast_insert.rs
Use this command to fix any missing license headers
```bash
docker run -it --rm -v $(pwd):/github/workspace apache/skywalking-eyes header fix
</details>
|
This pull request has been modified. If you want me to regenerate unit test for any of the files related, please find the file in "Files Changed" tab and add a comment |
src/frontend/src/webhook/mod.rs
Outdated
| // Can be any address, we use the port of meta to indicate that it's a internal request. | ||
| let dummy_addr = Address::Tcp(SocketAddr::new(IpAddr::V4(Ipv4Addr::new(0, 0, 0, 0)), 5691)); | ||
|
|
||
| // FIXME(kexiang): the dummy_session can lead to memory leakage |
There was a problem hiding this comment.
I think we should get rid of Session in this fast_insert approach. Any reason that blocks us from doing so?
There was a problem hiding this comment.
+1, Can we bypass creating a session to get catalog_reader? I think the catalog is just inside the SESSION_MANAGER's frontendEnv
proto/batch_plan.proto
Outdated
| // A special insert node for non-pgwire insert, not really a batch node. | ||
| message FastInsertNode { |
There was a problem hiding this comment.
not really a batch node.
Yeah, recommend to flatten these fields inside message FastInsertRequest
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
This pr does several things:
task_servicecalledfast_insert(maybe should be calledlightweight_insert).With this interface,
a. RW can insert dml data directly into tables without the complicated optimization process and batch execution process, which is more efficient.
b. the new interfaces allow callers to wait for persistency, which means, if enabled, the function call will only return after the data has been commited to persistency layer. To be specific, it will wait until the epoch when the insert happens has been commited in hummock.
One
ongoingrefactor: withFastInsertContextsaved as a map, we don't need to create a new sessionimpl every time anymore. After the communication with @st1page, pause for now.Two known bugs to fix:
alter table.Checklist
Documentation
Release note