Update libp2p to 0.53.1 #2212

nazar-pc · 2023-11-09T02:07:50Z

This doesn't seem to help with networking issues in the end, but we have to update to it at some point, so why not now, right?

There were a few type changes, ConnectionHandlerEvent::Close is gone, ConnectionEvent is (very annoyingly) non-exhausive (but there are improvements upstream like libp2p/rust-libp2p#4825) and keep-alive mechanism has changed.

Interesting changes:

handler in connected peers had to be updated to properly update its keep alive status asynchronously after timeout
tests in request response were failing likely due to libp2p changes, I have fixed and simplified them

Code contributor checklist:

I have read, understood and followed contributing guide

shamil-gadelshin

It looks OK. However, I would recommend to postpone the merge of this PR until we stabilize gemini-3g, otherwise we introduce a whole new line of issues connected to the new libp2p version. Another issue of this PR, that is shouldn't target the connected-peers protocol change PR. These changes should be separated.

nazar-pc · 2023-11-09T13:10:27Z

Another issue of this PR, that is shouldn't target the connected-peers protocol change PR. These changes should be separated.

It would be if there wasn't any merge conflicts otherwise and I don't see why we shouldn't merge connected-peers PR anyway.

nazar-pc · 2023-11-09T19:28:36Z

Rebased on main, no other changes

nazar-pc · 2023-11-13T14:59:45Z

Same stuff as before, just rebased on main

nazar-pc · 2023-11-13T20:30:10Z

Converting to draft, caught following panic locally:

====================

Version: 0.1.0-38353aeda25

   0: sp_panic_handler::panic_hook
             at /home/nazar-pc/.cargo/git/checkouts/polkadot-sdk-3e27f0ebe1cdca51/892bf8e/substrate/primitives/panic-handler/src/lib.rs:166:18
   1: sp_panic_handler::set::{{closure}}
             at /home/nazar-pc/.cargo/git/checkouts/polkadot-sdk-3e27f0ebe1cdca51/892bf8e/substrate/primitives/panic-handler/src/lib.rs:62:12
   2: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/alloc/src/boxed.rs:2021:9
      std::panicking::rust_panic_with_hook
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/panicking.rs:735:13
   3: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/panicking.rs:601:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/sys_common/backtrace.rs:170:18
   5: rust_begin_unwind
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/panicking.rs:597:5
   6: core::panicking::panic_fmt
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/core/src/panicking.rs:72:14
   7: core::panicking::panic
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/core/src/panicking.rs:127:5
   8: subspace_networking::protocols::peer_info::protocol::recv::{{closure}}
             at crates/subspace-networking/src/protocols/peer_info/protocol.rs:29:1
   9: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/core/src/future/future.rs:125:9
  10: futures_util::future::future::FutureExt::poll_unpin
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.29/src/future/future/mod.rs:558:9
  11: <subspace_networking::protocols::peer_info::handler::Handler as libp2p_swarm::handler::ConnectionHandler>::poll
             at crates/subspace-networking/src/protocols/peer_info/handler.rs:163:19
  12: <libp2p_swarm::behaviour::toggle::ToggleConnectionHandler<TInner> as libp2p_swarm::handler::ConnectionHandler>::poll
             at /home/nazar-pc/.cargo/git/checkouts/rust-libp2p-54a9adfb8fd2de49/64f3baf/swarm/src/behaviour/toggle.rs:304:13
  13: <libp2p_swarm::handler::select::ConnectionHandlerSelect<TProto1,TProto2> as libp2p_swarm::handler::ConnectionHandler>::poll
             at /home/nazar-pc/.cargo/git/checkouts/rust-libp2p-54a9adfb8fd2de49/64f3baf/swarm/src/handler/select.rs:240:15
  14: <libp2p_swarm::handler::select::ConnectionHandlerSelect<TProto1,TProto2> as libp2p_swarm::handler::ConnectionHandler>::poll
             at /home/nazar-pc/.cargo/git/checkouts/rust-libp2p-54a9adfb8fd2de49/64f3baf/swarm/src/handler/select.rs:223:15
  15: <libp2p_swarm::handler::select::ConnectionHandlerSelect<TProto1,TProto2> as libp2p_swarm::handler::ConnectionHandler>::poll
             at /home/nazar-pc/.cargo/git/checkouts/rust-libp2p-54a9adfb8fd2de49/64f3baf/swarm/src/handler/select.rs:223:15
  16: <libp2p_swarm::handler::select::ConnectionHandlerSelect<TProto1,TProto2> as libp2p_swarm::handler::ConnectionHandler>::poll
             at /home/nazar-pc/.cargo/git/checkouts/rust-libp2p-54a9adfb8fd2de49/64f3baf/swarm/src/handler/select.rs:223:15
  17: libp2p_swarm::connection::Connection<THandler>::poll
             at /home/nazar-pc/.cargo/git/checkouts/rust-libp2p-54a9adfb8fd2de49/64f3baf/swarm/src/connection.rs:274:19
  18: libp2p_swarm::connection::pool::task::new_for_established_connection::{{closure}}::{{closure}}
             at /home/nazar-pc/.cargo/git/checkouts/rust-libp2p-54a9adfb8fd2de49/64f3baf/swarm/src/connection/pool/task.rs:181:26
  19: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.29/src/future/poll_fn.rs:56:9
  20: futures_util::future::future::FutureExt::poll_unpin
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.29/src/future/future/mod.rs:558:9
  21: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.29/src/future/select.rs:118:35
  22: libp2p_swarm::connection::pool::task::new_for_established_connection::{{closure}}
             at /home/nazar-pc/.cargo/git/checkouts/rust-libp2p-54a9adfb8fd2de49/64f3baf/swarm/src/connection/pool/task.rs:183:10
  23: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  24: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/core/src/future/future.rs:125:9
  25: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/core.rs:328:17
  26: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/loom/std/unsafe_cell.rs:16:9
      tokio::runtime::task::core::Core<T,S>::poll
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/core.rs:317:13
  27: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/harness.rs:485:19
  28: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/core/src/panic/unwind_safe.rs:271:9
  29: std::panicking::try::do_call
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/panicking.rs:504:40
  30: __rust_try
  31: std::panicking::try
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/panicking.rs:468:19
  32: std::panic::catch_unwind
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/panic.rs:142:14
  33: tokio::runtime::task::harness::poll_future
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/harness.rs:473:18
  34: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/harness.rs:208:27
  35: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/harness.rs:153:15
  36: tokio::runtime::task::raw::poll
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/raw.rs:276:5
  37: tokio::runtime::task::raw::RawTask::poll
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/raw.rs:200:18
  38: tokio::runtime::task::LocalNotified<S>::run
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/mod.rs:408:9
  39: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/scheduler/multi_thread/worker.rs:639:17
  40: tokio::runtime::coop::with_budget
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/coop.rs:107:5
      tokio::runtime::coop::budget
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/coop.rs:73:5
      tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
  41: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
  42: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  43: tokio::runtime::context::scoped::Scoped<T>::set
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/context/scoped.rs:40:9
  44: tokio::runtime::context::set_scheduler::{{closure}}
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/context.rs:176:26
  45: std::thread::local::LocalKey<T>::try_with
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/thread/local.rs:270:16
  46: std::thread::local::LocalKey<T>::with
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/thread/local.rs:246:9
  47: tokio::runtime::context::set_scheduler
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/context.rs:176:9
  48: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  49: tokio::runtime::context::runtime::enter_runtime
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/context/runtime.rs:65:16
  50: tokio::runtime::scheduler::multi_thread::worker::run
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  51: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  52: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/blocking/task.rs:42:21
  53: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/core.rs:328:17
  54: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/loom/std/unsafe_cell.rs:16:9
      tokio::runtime::task::core::Core<T,S>::poll
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/core.rs:317:13
  55: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/harness.rs:485:19
  56: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/core/src/panic/unwind_safe.rs:271:9
  57: std::panicking::try::do_call
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/panicking.rs:504:40
  58: __rust_try
  59: std::panicking::try
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/panicking.rs:468:19
  60: std::panic::catch_unwind
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/panic.rs:142:14
  61: tokio::runtime::task::harness::poll_future
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/harness.rs:473:18
  62: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/harness.rs:208:27
  63: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/harness.rs:153:15
  64: tokio::runtime::task::raw::poll
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/raw.rs:276:5
  65: tokio::runtime::task::raw::RawTask::poll
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/raw.rs:200:18
  66: tokio::runtime::task::UnownedTask<S>::run
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/mod.rs:445:9
  67: tokio::runtime::blocking::pool::Task::run
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/blocking/pool.rs:159:9
  68: tokio::runtime::blocking::pool::Inner::run
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/blocking/pool.rs:513:17
  69: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /home/nazar-pc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/blocking/pool.rs:471:13
  70: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/sys_common/backtrace.rs:154:18
  71: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/thread/mod.rs:529:17
  72: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/core/src/panic/unwind_safe.rs:271:9
  73: std::panicking::try::do_call
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/panicking.rs:504:40
  74: __rust_try
  75: std::panicking::try
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/panicking.rs:468:19
  76: std::panic::catch_unwind
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/panic.rs:142:14
      std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/thread/mod.rs:528:30
  77: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/core/src/ops/function.rs:250:5
  78: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/alloc/src/boxed.rs:2007:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/alloc/src/boxed.rs:2007:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/42b1224e9eb37177f608d3f6a6f2be2ee13902e4/library/std/src/sys/unix/thread.rs:108:17
  79: start_thread
             at ./nptl/pthread_create.c:442:8
  80: clone3
             at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81


Thread 'tokio-runtime-worker' panicked at '`async fn` resumed after completion', crates/subspace-networking/src/protocols/peer_info/protocol.rs:29

nazar-pc · 2023-11-14T07:57:23Z

There were a few nice bug fixes according to change logs

nazar-pc requested review from vedhavyas, NingLin-P, rg3l3dr and shamil-gadelshin as code owners November 9, 2023 02:07

shamil-gadelshin reviewed Nov 9, 2023

View reviewed changes

nazar-pc force-pushed the fix-connected-peers-bursts branch from 73874a0 to 41c9880 Compare November 9, 2023 13:23

Base automatically changed from fix-connected-peers-bursts to main November 9, 2023 15:52

nazar-pc force-pushed the update-libp2p branch from 738d8c2 to c26af09 Compare November 9, 2023 19:28

nazar-pc enabled auto-merge November 10, 2023 00:09

Update libp2p to 0.53.0

38353ae

nazar-pc force-pushed the update-libp2p branch from c26af09 to 38353ae Compare November 13, 2023 14:59

nazar-pc requested a review from shamil-gadelshin November 13, 2023 14:59

nazar-pc marked this pull request as draft November 13, 2023 20:29

auto-merge was automatically disabled November 13, 2023 20:29
Pull request was converted to draft

Make sure to not poll the future again after it is ready

95f0345

nazar-pc marked this pull request as ready for review November 14, 2023 00:42

shamil-gadelshin previously approved these changes Nov 14, 2023

View reviewed changes

nazar-pc added this pull request to the merge queue Nov 14, 2023

nazar-pc removed this pull request from the merge queue due to a manual request Nov 14, 2023

Switch to upstream released version of libp2p

f749dbf

nazar-pc dismissed shamil-gadelshin’s stale review via f749dbf November 14, 2023 07:57

nazar-pc changed the title ~~Update libp2p to 0.53.0~~ Update libp2p to 0.53.1 Nov 14, 2023

nazar-pc requested a review from shamil-gadelshin November 14, 2023 07:57

nazar-pc enabled auto-merge November 14, 2023 07:57

shamil-gadelshin approved these changes Nov 14, 2023

View reviewed changes

nazar-pc added this pull request to the merge queue Nov 14, 2023

Merged via the queue into main with commit b2f2b50 Nov 14, 2023
10 checks passed

nazar-pc deleted the update-libp2p branch November 14, 2023 11:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update libp2p to 0.53.1 #2212

Update libp2p to 0.53.1 #2212

nazar-pc commented Nov 9, 2023 •

edited

Loading

shamil-gadelshin left a comment

nazar-pc commented Nov 9, 2023

nazar-pc commented Nov 9, 2023

nazar-pc commented Nov 13, 2023

nazar-pc commented Nov 13, 2023

nazar-pc commented Nov 14, 2023

Update libp2p to 0.53.1 #2212

Update libp2p to 0.53.1 #2212

Conversation

nazar-pc commented Nov 9, 2023 • edited Loading

Code contributor checklist:

shamil-gadelshin left a comment

Choose a reason for hiding this comment

nazar-pc commented Nov 9, 2023

nazar-pc commented Nov 9, 2023

nazar-pc commented Nov 13, 2023

nazar-pc commented Nov 13, 2023

nazar-pc commented Nov 14, 2023

nazar-pc commented Nov 9, 2023 •

edited

Loading