Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disperser auth innabox #1052

Merged
merged 108 commits into from
Jan 10, 2025
Merged
Show file tree
Hide file tree
Changes from 106 commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
0936f5f
Enable TLS connections between the disperser and the DA nodes.
cody-littley Dec 4, 2024
d36424e
Incremental progress.
cody-littley Dec 4, 2024
342d500
Moar cowsay
cody-littley Dec 4, 2024
fd2ab69
formatting
cody-littley Dec 4, 2024
e2e3fef
Started working on unit test.
cody-littley Dec 4, 2024
4373095
Incremental progress.
cody-littley Dec 4, 2024
df38980
It works now, kind of
cody-littley Dec 4, 2024
d54c7f7
Incremental progress.
cody-littley Dec 5, 2024
083d67d
Start experimenting with ecdsa.
cody-littley Dec 6, 2024
0a5c91a
Authorize StoreChunks() requests.
cody-littley Dec 10, 2024
8e9c16b
Merge branch 'master' into disperser-auth
cody-littley Dec 10, 2024
f348550
Update docs.
cody-littley Dec 10, 2024
01b7ece
Incremental progress.
cody-littley Dec 11, 2024
e09a77c
Incremental progress.
cody-littley Dec 11, 2024
290c93c
Incremental progress.
cody-littley Dec 11, 2024
5ba4b7b
Incremental progress.
cody-littley Dec 11, 2024
e094371
Added test placeholder.
cody-littley Dec 11, 2024
db0885a
shuffle stuff around
cody-littley Dec 12, 2024
da2ca6c
Merge branch 'master' into disperser-auth
cody-littley Dec 12, 2024
b04fcfa
Unit tests for request signing.
cody-littley Dec 12, 2024
29e7e7d
Delete stale code.
cody-littley Dec 12, 2024
fb1ebfb
Get things kind of working
cody-littley Dec 12, 2024
6ca7c9b
Finished unit tests.
cody-littley Dec 13, 2024
3348883
Cleanup.
cody-littley Dec 13, 2024
e5966f0
Cleanup
cody-littley Dec 13, 2024
f28a5d4
Merge branch 'master' into disperser-auth
cody-littley Dec 16, 2024
f0c6f4e
Cleanup.
cody-littley Dec 16, 2024
6a00256
Added request signer.
cody-littley Dec 16, 2024
27d5b8a
Incremental progress.
cody-littley Dec 16, 2024
bbd82ec
Update flags.
cody-littley Dec 16, 2024
157c2ea
Started work on unit test, not yet working.
cody-littley Dec 16, 2024
d793501
Incremental progress in kludging something together
cody-littley Dec 16, 2024
b81e430
Incremental test iteration.
cody-littley Dec 16, 2024
7327db1
Added debug printing.
cody-littley Dec 17, 2024
8cfa971
IT'S ALIVE!
cody-littley Dec 17, 2024
306746e
Partial cleanup.
cody-littley Dec 17, 2024
b41d729
Move code to proper locations.
cody-littley Dec 17, 2024
d709520
Copy eigensdk-go implementation of kms parsing.
cody-littley Dec 17, 2024
2e44b02
Update kms.go
anupsv Dec 18, 2024
aebda81
Create kms_fuzz_test.go
anupsv Dec 18, 2024
260bf79
Update Makefile for fuzz tests
anupsv Dec 18, 2024
dbd0d24
Merge pull request #1 from anupsv/disperser-auth
cody-littley Dec 18, 2024
5dbb178
Merge branch 'master' into disperser-auth
cody-littley Dec 18, 2024
dcd3e06
Disable request signing if KMS key name is not provided.
cody-littley Dec 18, 2024
11cf45a
Merge branch 'master' into disperser-auth
cody-littley Dec 18, 2024
3580bed
Merge branch 'master' into disperser-auth
cody-littley Dec 18, 2024
f6b008f
Tie into bindings.
cody-littley Dec 18, 2024
1493ded
Incremental progress.
cody-littley Dec 18, 2024
feab7a1
Merge branch 'master' into disperser-auth
cody-littley Dec 19, 2024
52f2354
Add debug code.
cody-littley Dec 19, 2024
7c5cc93
Fix unit test.
cody-littley Dec 19, 2024
c83db79
Disable request signing for inabox e2e test.
cody-littley Dec 19, 2024
e5cb69c
Added flag for disabling signing.
cody-littley Dec 19, 2024
20216c4
Cleanup.
cody-littley Dec 19, 2024
3c325ef
Made suggested changes.
cody-littley Dec 20, 2024
d4b7051
Start messing around with external KMS keys.
cody-littley Dec 20, 2024
02ed38a
tweak docker build
cody-littley Dec 20, 2024
60ad5c9
Merge branch 'disperser-auth' into disperser-auth-innabox
cody-littley Dec 20, 2024
8560b8b
Remove test.
cody-littley Dec 20, 2024
613bd20
Generate kms key pair.
cody-littley Dec 20, 2024
4912547
Fix localstack url
cody-littley Dec 20, 2024
80c7e6d
Make requested changes.
cody-littley Dec 20, 2024
2b6cbb2
Made suggested changes.
cody-littley Dec 20, 2024
a9155a0
Merge branch 'master' into disperser-auth
cody-littley Jan 2, 2025
3a9792c
Merge branch 'disperser-auth' into disperser-auth-innabox
cody-littley Jan 2, 2025
a57140c
Add debug stack trace.
cody-littley Jan 2, 2025
11e2ffa
Tweak stack trace.
cody-littley Jan 2, 2025
ad89396
Tweak aws URL
cody-littley Jan 2, 2025
a2b602e
Manually start localstack.
cody-littley Jan 2, 2025
db018f0
Don't start localstack for churner test.
cody-littley Jan 2, 2025
d1ac4a8
Skip key generation when localstack is not running.
cody-littley Jan 2, 2025
822ab3d
Write code that sets disperser address.
cody-littley Jan 2, 2025
ee262bc
Enable StoreChunks() signing
cody-littley Jan 2, 2025
262e1a9
Change where the disperser address is written to contract.
cody-littley Jan 2, 2025
d32d0a8
Wait for disperser address transaction to be handled.
cody-littley Jan 3, 2025
1f42f9d
Actually submit transaction.
cody-littley Jan 3, 2025
7caac2c
Create a block every second.
cody-littley Jan 3, 2025
20fd4da
Use client with no confirmations.
cody-littley Jan 3, 2025
b09a43e
Revert anvil change
cody-littley Jan 3, 2025
6f9e242
Change where disperser address is updated.
cody-littley Jan 3, 2025
0a3e35b
Add stack trace.
cody-littley Jan 6, 2025
d92252d
Increase stack trace size.
cody-littley Jan 6, 2025
524b22a
Tweak stack trace.
cody-littley Jan 6, 2025
7971b13
Multiple log statements.
cody-littley Jan 6, 2025
e07b734
Update disperser address earlier.
cody-littley Jan 6, 2025
8452d09
Initiate logger earlier.
cody-littley Jan 6, 2025
987b26c
Made suggested change.
cody-littley Jan 6, 2025
525e37f
Reshuffle code.
cody-littley Jan 6, 2025
87a7610
Tweaks.
cody-littley Jan 6, 2025
3852dc7
Play with ordering.
cody-littley Jan 6, 2025
a305e93
Relocate code.
cody-littley Jan 6, 2025
568e971
Try moving disperser setup again.
cody-littley Jan 6, 2025
c77e596
move code
cody-littley Jan 6, 2025
7371497
Tweaks
cody-littley Jan 6, 2025
eca25b9
Use other eth client.
cody-littley Jan 6, 2025
26ec866
Made suggested changes.
cody-littley Jan 7, 2025
76efb7a
Lint
cody-littley Jan 7, 2025
afe242f
Debug log
cody-littley Jan 7, 2025
4419be1
Enable store chunks request signing.
cody-littley Jan 7, 2025
09b14ce
Generate key prior to writing env files.
cody-littley Jan 7, 2025
cd1bfc6
Merge branch 'master' into disperser-auth
cody-littley Jan 8, 2025
35caefb
Merge branch 'disperser-auth' into disperser-auth-innabox
cody-littley Jan 8, 2025
7355080
Fix merge conflict.
cody-littley Jan 8, 2025
6abb354
Merge branch 'master' into disperser-auth-innabox
cody-littley Jan 8, 2025
6003a89
Cleanup
cody-littley Jan 8, 2025
8cbdd60
Made suggested change
cody-littley Jan 9, 2025
cba0358
Made suggested changes.
cody-littley Jan 9, 2025
57abb29
Fix test issue
cody-littley Jan 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions core/eth/reader.go
Original file line number Diff line number Diff line change
Expand Up @@ -434,8 +434,8 @@ func (t *Reader) GetOperatorStakesForQuorums(ctx context.Context, quorums []core
Context: ctx,
}, t.bindings.RegCoordinatorAddr, quorumBytes, blockNumber)
if err != nil {
t.logger.Error("Failed to fetch operator state", "err", err)
return nil, err
t.logger.Errorf("Failed to fetch operator state: %s", err)
return nil, fmt.Errorf("failed to fetch operator state: %w", err)
}

state := make(core.OperatorStakes, len(state_))
Expand Down
39 changes: 39 additions & 0 deletions core/eth/writer.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ import (
"encoding/hex"
"encoding/json"
"fmt"
"github.com/Layr-Labs/eigenda/api"
contractEigenDADisperserRegistry "github.com/Layr-Labs/eigenda/contracts/bindings/EigenDADisperserRegistry"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use lowecase for alias

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed to dreg

"log"
"math/big"

"github.com/Layr-Labs/eigenda/api/grpc/churner"
Expand Down Expand Up @@ -329,3 +332,39 @@ func (t *Writer) ConfirmBatch(ctx context.Context, batchHeader *core.BatchHeader
}
return receipt, nil
}

// SetDisperserAddress sets the address of the disperser. Since there is currently only one disperser, this function
// can only be used to set the address of that disperser.
func (t *Writer) SetDisperserAddress(ctx context.Context, address gethcommon.Address) error {
registry := t.bindings.DisperserRegistry
if registry == nil {
log.Printf("disperser registry not deployed")
return errors.New("disperser registry not deployed")
}

log.Printf("Setting disperser %d address to %s", api.EigenLabsDisperserID, address.String())

options, err := t.ethClient.GetNoSendTransactOpts()
if err != nil {
t.logger.Error("Failed to generate transact opts", "err", err)
return err
}
options.Context = ctx

transaction, err := registry.SetDisperserInfo(
options,
api.EigenLabsDisperserID,
contractEigenDADisperserRegistry.DisperserInfo{
DisperserAddress: address,
})
if err != nil {
return fmt.Errorf("failed to create transaction for setting disperser address: %w", err)
}

err = t.ethClient.SendTransaction(ctx, transaction)
if err != nil {
return fmt.Errorf("failed to set disperser address: %w", err)
}

return nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to wait for txn to be mined?

Copy link
Contributor Author

@cody-littley cody-littley Jan 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a retry loop in the function that calls this function. It will now wait for up to a minute if the disperser address is not available on chain.

        // Read the disperser's public key from on-chain storage to verify it was written correctly
	
	retryTimeout := time.Now().Add(1 * time.Minute)
	ticker := time.NewTicker(1 * time.Second)
	
	for time.Now().Before(retryTimeout) {
		address, err := writer.GetDisperserAddress(context.Background(), 0)
		if err != nil {
			logger.Warnf("could not get disperser address: %v", err)
		} else {
			if address != env.DisperserAddress {
				return fmt.Errorf("expected disperser address %s, got %s", env.DisperserAddress, address)
			}
			return nil
		}
		
		<- ticker.C
	}

	return fmt.Errorf("timed out waiting for disperser address to be set")

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would EstimateGasPriceAndLimitAndSendTx in the ethclient work cleaner?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This caused errors. I initially tried to use this method, but later in the test run I'd get failures with the following error message:

{
   "time":"2025-01-03T16:28:01.407636692Z",
   "level":"ERROR",
   "source":{
      "function":"github.com/Layr-Labs/eigenda/core/eth.(*Reader).GetOperatorStakesForQuorums",
      "file":"/home/runner/work/eigenda/eigenda/core/eth/reader.go",
      "line":437
   },
   "msg":"Failed to fetch operator state",
   "component":"Reader",
   "err":"execution reverted: IndexRegistry._operatorCountAtBlockNumber: quorum did not exist at given block number"
}

I spent a lot of time on this and was never able to determine how these two things were related. But as soon as I switched to using SendTransaction, the problem went away.

}
18 changes: 10 additions & 8 deletions inabox/deploy/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -321,7 +321,10 @@ func (env *Config) generateEncoderV2Vars(ind int, grpcPort string) EncoderVars {
return v
}

func (env *Config) generateControllerVars(ind int, graphUrl string) ControllerVars {
func (env *Config) generateControllerVars(
ind int,
graphUrl string) ControllerVars {

v := ControllerVars{
CONTROLLER_DYNAMODB_TABLE_NAME: "test-BlobMetadata-v2",
CONTROLLER_BLS_OPERATOR_STATE_RETRIVER: env.EigenDA.OperatorStateRetreiver,
Expand All @@ -343,7 +346,8 @@ func (env *Config) generateControllerVars(ind int, graphUrl string) ControllerVa
CONTROLLER_AWS_ENDPOINT_URL: "",
CONTROLLER_ENCODER_ADDRESS: "0.0.0.0:34001",
CONTROLLER_FINALIZATION_BLOCK_DELAY: "0",
CONTROLLER_DISPERSER_STORE_CHUNKS_SIGNING_DISABLED: "true",
CONTROLLER_DISPERSER_STORE_CHUNKS_SIGNING_DISABLED: "false",
CONTROLLER_DISPERSER_KMS_KEY_ID: env.DisperserKMSKeyID,
}
env.applyDefaults(&v, "CONTROLLER", "controller", ind)

Expand Down Expand Up @@ -431,7 +435,7 @@ func (env *Config) generateOperatorVars(ind int, name, key, churnerUrl, logPath,
NODE_NUM_CONFIRMATIONS: "0",
NODE_ONCHAIN_METRICS_INTERVAL: "-1",
NODE_ENABLE_V2: "true",
NODE_DISABLE_DISPERSAL_AUTHENTICATION: "true",
NODE_DISABLE_DISPERSAL_AUTHENTICATION: "false",
}

env.applyDefaults(&v, "NODE", "opr", ind)
Expand Down Expand Up @@ -573,15 +577,13 @@ func (env *Config) GenerateAllVariables() {
// hardcode graphurl for now
graphUrl := "http://localhost:8000/subgraphs/name/Layr-Labs/eigenda-operator-state"

env.localstackEndpoint = "http://localhost:4570"
env.localstackRegion = "us-east-1"

// Create envs directory
createDirectory(env.Path + "/envs")
changeDirectory(env.rootPath + "/inabox")

// Gather keys
// keyData := readFile(gethPrivateKeys)
// keys := strings.Split(string(keyData), "\n")
// id := 1

// Create compose file
composeFile := env.Path + "/docker-compose.yml"
servicesMap := make(map[string]map[string]interface{})
Expand Down
14 changes: 12 additions & 2 deletions inabox/deploy/config_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package deploy

import (
"encoding/json"
"github.com/ethereum/go-ethereum/common"
"log"
"os"
"path/filepath"
Expand Down Expand Up @@ -181,10 +182,19 @@ type Config struct {
Retriever RetrieverVars
Controller ControllerVars
Relays []RelayVars

localstackEndpoint string
localstackRegion string

// DisperserAddress is the address of disperser 0 (aka the only disperser at the current time)
DisperserAddress common.Address

// DisperserKMSKeyID is the KMS key ID used to encrypt disperser data
DisperserKMSKeyID string
}

func (c Config) IsEigenDADeployed() bool {
return c.EigenDA.ServiceManager != ""
func (env *Config) IsEigenDADeployed() bool {
return env.EigenDA.ServiceManager != ""
}

func NewTestConfig(testName, rootPath string) (testEnv *Config) {
Expand Down
99 changes: 94 additions & 5 deletions inabox/deploy/deploy.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,23 @@ import (
"context"
"encoding/json"
"fmt"
"github.com/Layr-Labs/eigenda/common"
caws "github.com/Layr-Labs/eigenda/common/aws"
relayreg "github.com/Layr-Labs/eigenda/contracts/bindings/EigenDARelayRegistry"
eigendasrvmg "github.com/Layr-Labs/eigenda/contracts/bindings/EigenDAServiceManager"
thresholdreg "github.com/Layr-Labs/eigenda/contracts/bindings/EigenDAThresholdRegistry"
"github.com/Layr-Labs/eigenda/core/eth"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/kms"
"github.com/aws/aws-sdk-go-v2/service/kms/types"
"github.com/ethereum/go-ethereum/crypto"
"io"
"log"
"math/big"
"os"
"path/filepath"
"strconv"

"github.com/Layr-Labs/eigenda/common"
relayreg "github.com/Layr-Labs/eigenda/contracts/bindings/EigenDARelayRegistry"
eigendasrvmg "github.com/Layr-Labs/eigenda/contracts/bindings/EigenDAServiceManager"
thresholdreg "github.com/Layr-Labs/eigenda/contracts/bindings/EigenDAThresholdRegistry"
"strings"

"github.com/Layr-Labs/eigenda/core"
"github.com/ethereum/go-ethereum/accounts/abi/bind"
Expand Down Expand Up @@ -155,12 +161,95 @@ func (env *Config) DeployExperiment() {
env.deploySubgraphs(startBlock)
}

fmt.Println("Generating disperser keypair")
err = env.GenerateDisperserKeypair()
if err != nil {
log.Panicf("could not generate disperser keypair: %v", err)
}

fmt.Println("Generating variables")
env.GenerateAllVariables()

fmt.Println("Test environment has successfully deployed!")
}

// GenerateDisperserKeypair generates a disperser keypair using AWS KMS.
func (env *Config) GenerateDisperserKeypair() error {

// Generate a keypair in AWS KMS

keyManager := kms.New(kms.Options{
Region: env.localstackRegion,
BaseEndpoint: aws.String(env.localstackEndpoint),
})

createKeyOutput, err := keyManager.CreateKey(context.Background(), &kms.CreateKeyInput{
KeySpec: types.KeySpecEccSecgP256k1,
KeyUsage: types.KeyUsageTypeSignVerify,
})
if err != nil {
if strings.Contains(err.Error(), "connect: connection refused") {
log.Printf("Unable to reach local stack, skipping disperser keypair generation. Error: %v", err)
err = nil
}
return err
}

env.DisperserKMSKeyID = *createKeyOutput.KeyMetadata.KeyId

// Load the public key and convert it to an Ethereum address

key, err := caws.LoadPublicKeyKMS(context.Background(), keyManager, env.DisperserKMSKeyID)
if err != nil {
return fmt.Errorf("could not load public key: %v", err)
}

env.DisperserAddress = crypto.PubkeyToAddress(*key)
log.Printf("Generated disperser keypair: key ID: %s, address: %s",
env.DisperserKMSKeyID, env.DisperserAddress.Hex())

return nil
}

// RegisterDisperserKeypair registers the disperser's public key on-chain.
func (env *Config) RegisterDisperserKeypair(ethClient common.EthClient) error {

// Write the disperser's public key to on-chain storage

loggerConfig := common.DefaultLoggerConfig()
logger, err := common.NewLogger(loggerConfig)
if err != nil {
return fmt.Errorf("could not create logger: %v", err)
}

writer, err := eth.NewWriter(
logger,
ethClient,
env.Retriever.RETRIEVER_BLS_OPERATOR_STATE_RETRIVER,
env.Retriever.RETRIEVER_EIGENDA_SERVICE_MANAGER)
if err != nil {
return fmt.Errorf("could not create writer: %v", err)
}

err = writer.SetDisperserAddress(context.Background(), env.DisperserAddress)
if err != nil {
return fmt.Errorf("could not set disperser address: %v", err)
}

// Read the disperser's public key from on-chain storage to verify it was written correctly

address, err := writer.GetDisperserAddress(context.Background(), 0)
if err != nil {
return fmt.Errorf("could not get disperser address: %v", err)
}

if address != env.DisperserAddress {
return fmt.Errorf("expected disperser address %s, got %s", env.DisperserAddress, address)
}

return nil
}

func (env *Config) RegisterBlobVersionAndRelays(ethClient common.EthClient) map[uint32]string {
dasmAddr := gcommon.HexToAddress(env.EigenDA.ServiceManager)
contractEigenDAServiceManager, err := eigendasrvmg.NewContractEigenDAServiceManager(dasmAddr, ethClient)
Expand Down
13 changes: 10 additions & 3 deletions inabox/tests/integration_suite_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -109,13 +109,13 @@ var _ = BeforeSuite(func() {
testConfig.StartGraphNode()
}

fmt.Println("Deploying experiment")
testConfig.DeployExperiment()

loggerConfig := common.DefaultLoggerConfig()
logger, err = common.NewLogger(loggerConfig)
Expect(err).To(BeNil())

fmt.Println("Deploying experiment")
testConfig.DeployExperiment()

pk := testConfig.Pks.EcdsaMap["default"].PrivateKey
pk = strings.TrimPrefix(pk, "0x")
pk = strings.TrimPrefix(pk, "0X")
Expand All @@ -126,12 +126,19 @@ var _ = BeforeSuite(func() {
NumRetries: numRetries,
}, gcommon.Address{}, logger)
Expect(err).To(BeNil())

rpcClient, err = ethrpc.Dial(testConfig.Deployers[0].RPC)
Expect(err).To(BeNil())

fmt.Println("Registering blob versions and relays")
relays = testConfig.RegisterBlobVersionAndRelays(ethClient)

fmt.Println("Registering disperser keypair")
err = testConfig.RegisterDisperserKeypair(ethClient)
if err != nil {
panic(err)
}

fmt.Println("Starting binaries")
testConfig.StartBinaries()

Expand Down
2 changes: 1 addition & 1 deletion node/auth/request_signing.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ func VerifyStoreChunksRequest(key gethcommon.Address, request *grpc.StoreChunksR

signingPublicKey, err := crypto.SigToPub(requestHash, request.Signature)
if err != nil {
return fmt.Errorf("failed to recover public key from signature: %w", err)
return fmt.Errorf("failed to recover public key from signature %x: %w", request.Signature, err)
}

signingAddress := crypto.PubkeyToAddress(*signingPublicKey)
Expand Down
Loading