Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lotus-miner: expose Config in StorageMiner; add SealingPipelineState API endpoint #7513

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions api/api_storage.go
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,8 @@ type StorageMiner interface {
// SectorCommitPending returns a list of pending Commit sectors to be sent in the next aggregate message
SectorCommitPending(ctx context.Context) ([]abi.SectorID, error) //perm:admin

SealingPipelineState(ctx context.Context) (*SealingPipelineState, error) //perm:admin

// WorkerConnect tells the node to connect to workers RPC
WorkerConnect(context.Context, string) error //perm:admin retry:true
WorkerStats(context.Context) (map[uuid.UUID]storiface.WorkerStats, error) //perm:admin
Expand Down
13 changes: 13 additions & 0 deletions api/proxy_gen.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions api/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -195,3 +195,10 @@ type RetrievalInfo struct {
TransferChannelID *datatransfer.ChannelID
DataTransfer *DataTransferChannel
}

type SealingPipelineState struct {
TaskJobsCount map[string]int
MaxWaitDealsSectors uint64
MaxSealingSectors uint64
MaxSealingSectorsForDeals uint64
}
Binary file modified build/openrpc/full.json.gz
Binary file not shown.
Binary file modified build/openrpc/miner.json.gz
Binary file not shown.
Binary file modified build/openrpc/worker.json.gz
Binary file not shown.
30 changes: 30 additions & 0 deletions cmd/lotus-miner/sealing.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
"text/tabwriter"
"time"

"github.com/davecgh/go-spew/spew"
"github.com/fatih/color"
"github.com/google/uuid"
"github.com/urfave/cli/v2"
Expand All @@ -29,6 +30,7 @@ var sealingCmd = &cli.Command{
sealingWorkersCmd,
sealingSchedDiagCmd,
sealingAbortCmd,
sealingStateCmd,
},
}

Expand Down Expand Up @@ -148,6 +150,34 @@ var sealingWorkersCmd = &cli.Command{
},
}

var sealingStateCmd = &cli.Command{
Name: "state",
Usage: "state of sealing pipeline",
Flags: []cli.Flag{},
Action: func(cctx *cli.Context) error {
if cctx.IsSet("color") {
color.NoColor = !cctx.Bool("color")
}

api, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
defer closer()

ctx := lcli.ReqContext(cctx)

res, err := api.SealingPipelineState(ctx)
if err != nil {
return xerrors.Errorf("getting worker jobs: %w", err)
}

spew.Dump(res)

return nil
},
}

var sealingJobsCmd = &cli.Command{
Name: "jobs",
Usage: "list running jobs",
Expand Down
20 changes: 20 additions & 0 deletions documentation/en/api-v0-methods-miner.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@
* [RuntimeSubsystems](#RuntimeSubsystems)
* [Sealing](#Sealing)
* [SealingAbort](#SealingAbort)
* [SealingPipelineState](#SealingPipelineState)
* [SealingSchedDiag](#SealingSchedDiag)
* [Sector](#Sector)
* [SectorAddPieceToAny](#SectorAddPieceToAny)
Expand Down Expand Up @@ -1710,6 +1711,25 @@ Inputs:

Response: `{}`

### SealingPipelineState


Perms: admin

Inputs: `null`

Response:
```json
{
"TasksStats": {
"name": 42
},
"MaxWaitDealsSectors": 42,
"MaxSealingSectors": 42,
"MaxSealingSectorsForDeals": 42
}
```

### SealingSchedDiag
SealingSchedDiag dumps internal sealing scheduler state

Expand Down
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ require (
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e
github.com/cockroachdb/pebble v0.0.0-20201001221639-879f3bfeef07
github.com/coreos/go-systemd/v22 v22.3.2
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/detailyang/go-fallocate v0.0.0-20180908115635-432fa640bd2e
github.com/dgraph-io/badger/v2 v2.2007.2
github.com/docker/go-units v0.4.0
Expand Down
1 change: 1 addition & 0 deletions node/builder_miner.go
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ func ConfigStorageMiner(c interface{}) Option {

Override(CheckFDLimit, modules.CheckFdLimit(build.MinerFDLimit)), // recommend at least 100k FD limit to miners

Override(new(*config.StorageMiner), cfg),
Override(new(api.MinerSubsystems), modules.ExtractEnabledMinerSubsystems(cfg.Subsystems)),
Override(new(stores.LocalStorage), From(new(repo.LockedRepo))),
Override(new(*stores.Local), modules.LocalStorage),
Expand Down
25 changes: 25 additions & 0 deletions node/impl/storminer.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ import (
"github.com/filecoin-project/lotus/chain/types"
"github.com/filecoin-project/lotus/markets/storageadapter"
"github.com/filecoin-project/lotus/miner"
"github.com/filecoin-project/lotus/node/config"
"github.com/filecoin-project/lotus/node/modules/dtypes"
"github.com/filecoin-project/lotus/storage"
"github.com/filecoin-project/lotus/storage/sectorblocks"
Expand All @@ -60,6 +61,8 @@ type StorageMinerAPI struct {

EnabledSubsystems api.MinerSubsystems

Config *config.StorageMiner
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least the Sealing part of the config supports auto-reloading (this is why we have the getConfig function in the sealing package) - but just putting this config here like that won't get you updated values after changing the config on disk

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(side note: we really need to unify how we support dynamic config, and make that nicer / more fully supported)


Full api.FullNode
LocalStore *stores.Local
RemoteStore *stores.Remote
Expand Down Expand Up @@ -123,6 +126,28 @@ func (sm *StorageMinerAPI) ServeRemote(perm bool) func(w http.ResponseWriter, r
}
}

func (sm *StorageMinerAPI) SealingPipelineState(ctx context.Context) (*api.SealingPipelineState, error) {
jobs, err := sm.WorkerJobs(ctx)
if err != nil {
return nil, xerrors.Errorf("getting worker jobs: %w", err)
}
Comment on lines +130 to +133
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WorkerJobs is already exposed no the RPC


jobsCount := make(map[string]int)

for _, wj := range jobs {
for _, j := range wj {
jobsCount[j.Task.Short()] = jobsCount[j.Task.Short()] + 1
}
}

return &api.SealingPipelineState{
TaskJobsCount: jobsCount,
MaxWaitDealsSectors: sm.Config.Sealing.MaxWaitDealsSectors,
MaxSealingSectors: sm.Config.Sealing.MaxSealingSectors,
MaxSealingSectorsForDeals: sm.Config.Sealing.MaxSealingSectorsForDeals,
}, nil
}

func (sm *StorageMinerAPI) WorkerStats(context.Context) (map[uuid.UUID]storiface.WorkerStats, error) {
return sm.StorageMgr.WorkerStats(), nil
}
Expand Down