Experiment Creation (WIP)

[bonedaddy]

Not the official PR, working out of this PR for my own purposes and task tracking, etc...

Session Bug

• session calls update -> update calls supervise
  • labagent/agentrouter/router.go
• benchmark is ran -> calls /labapp/appapi/appapi.go to run the actual benchmark
• run function is located in labapp/approuter/router.go

Looks like the race is happening within the supevisor???

Next Focus

• Distribute cue templates as package hinshun already did this so not needed
• cleanup repo for upstream merge

TODO:

• Distribute cue templates as an actual package
  • Example

Session Bug Description (largely copy & pasted from hinshun)

• When calling the /update API with the intended version ref to labagent to start supervising labapp process, it's usually an asynchronous process.
• In order to capture opentracing spans in bitswap, the context.Context object must be propagated to the ROOT context object used to start the libp2p peer.
• This context must have the trace ID and other metadata, which is plumbed across the network by being injected/extracted from HTTP headers. In order to achieve this, there is a synchronous version of /update that involves calling the API without a ref.
• This call to /update forwards the context.Context from the client, all the way to the root context of the libp2p peer, which means that this peer lives and dies with the request.
• The code I commented out calls this synchronous version of /update and cancels the context to kill the peer. For some reason, it's killing a peer in an unrelated cluster (other trial in the experiment), and it seems to be racy (not every time).

[bonedaddy]

Useful commands for bulk running tests:

1. for i in $(ps aux | grep -i labd | awk '{print $2}'); do kill -9 "$i"; done
2. rm -rf labd tmp output.log && go build && ./labd 2>&1 | tee --append output.log
3. rm -rf labctl output.log && go build && ./labctl --log-level debug e create ../../cue/cue.mod/p2plab_example1.cue 2>&1 | tee --append output.log

[bonedaddy]

Trace errors

Overview

These trace errors lead me to labagent/agentrouter/router.go:putUpdate which leads me to think that maybe it's the peer definition handling? I've shoe-horned a temporary helper function to create peer definitions for experiments and im wondering if this is the problem

labd scenarios.Session

labd HTTP POST

[bonedaddy]

this is the buggy part

[bonedaddy]

Session Debugging

labagent/agentrouter/router.go is what gets called when session updates and fails

[bonedaddy]

Build labd with -race and captured a race
output.log

[bonedaddy]

Session Bug

• session calls update -> update calls supervise
  • labagent/agentrouter/router.go
• benchmark is ran -> calls /labapp/appapi/appapi.go to run the actual benchmark
• run function is located in labapp/approuter/router.go

Looks like the race is happening within the supevisor???