Marlowe History Architecture discussion

[jhbertra]

This thread is for exploratory brainstorming and fleshing out the requirements of the runtime history module. Here is what we have to work with at the moment:

• History communicates with Chain Sync via Filtered Chain Sync Protocol.
• History provides a synchronization API to notify clients of changes to contracts.
• History provides a query API for clients to pull history on demand.
• History can filter contracts in some way (e.g. by scanning for role tokens in a wallet).

Here is a possible architecture that could meet these needs:

History is (or is part of) a long-running background process. It maintains several threads of control:

1. A ContractFilter component connects to the Chain Sync via the Filtered Chain Sync Protocol. It receives a ContractFilter as an input. Its job is to traverse the chain to search for contracts that match the filter (for example, it could watch for role token deposits in a user's wallet). When it discovers a new contract, it spawns a ContractFollower component. It exposes an STM ContractEvents action, aggregating the events from the active ContractFollowers.
2. Each ContractFollower component connects to the Chain Sync via the Filtered Chain Sync Protocol. It receives a ContractId (TxOut reference that created the contract) as an input. Its job is to follow the events of the contract on chain. It exposes an STM ContractEvents action that retrieves unsaved events (similar to the STM Changes exposed by the NodeClient in the Chain Sync). It dies when the contract is closed after allowing the security parameter to elapse (so it can be sure it won't get rolled back).
3. A ContractStore component reads from the STM ContractEvents action exposed by the ContractFilter in a loop, writing unsaved contract events to a database. It exposes an STM (Map ContractId UTCTime) action, which it updates every time it saves new contract events. It also exposes a HistoryQuery a -> m a action for running queries.
4. A QueryServer component awaits queries and responds to them by running the HistoryQuery a -> m a action exposed by the ContractStore.
5. A SyncServer component implements the Server role in the Marlowe History Sync Protocol (see below). Clients connect to this server to synchronize contract history.

The Marlowe History Sync Protocol would function similarly to the Chain Sync Protocol. I won't get into the detailed design here, but in short, a client would initialize the connection by providing a contract filter to control which contracts the client sees. From the idle state, the client could choose to enter one of four modes:

1. Intersect - the client sends a list of intersection points to resume from where it last left off.
2. Fast Forward - similar to Next, but blocks not containing chain events are skipped.
3. Next - the client requests the next block. The server either rolls the client forward or forward. If rolling forward, the server provides the client with the header of the next block, any Marlowe events that block contains, and the current tip. If rolling backward, the server provides the client with the chain point it was rolled back to and the new tip.

[bwbush]

either rolls the client forward or forward

presumably "either rolls the client forward or backward"

[jhbertra]

No looking back

[bwbush]

Should we account for the edge case that one transaction creates two Marlowe contracts? ContractId = TxOutRef would be sufficient to separate them.

[jhbertra]

That's what I suggested ContractId = TxOutRef

[bwbush]

Will ContractStore have a rollback schema and approach similar or identical to that in Chain Sync?

[jhbertra]

Yes - all contract events will be tagged with the slot and block header hash they are associated with. In the event of a rollback, all events above the rollback point will be rolled back.

[bwbush]

We'll need to support the following query patterns:

1. Current on-chain Contract and State for a particular contract.
2. All history for a particular contract, perhaps equivalent to Contract and State interleaved with TransactionInput and TransactionOutput, also including TxOutRef.
3. List of active and/or past contracts at a particular payment or stake address.
4. Addresses at which role tokens for a contract reside.

What else?

See also #180 (comment).

[jhbertra]

An alternative approach is to make the core runtime only communicate ContractEvents and allow the clients to fold them into whatever state they need (e.g. a CLI "porcelain" command could do this to show contract state)

[jhbertra]

That way we have a future-proof design where we don't have to build support for all query patterns upfront explicitly - we instead have a single query pattern that is general enough to adapt to arbitrary needs in the future.

[jhbertra]

Contract filter considerations

Potential requirements for contract filtering

• Filter by contract ID (contract creation TxOut ref)
• Filter by role currency (minting policy ID)
• Filter by public key hash (or address)
• Filter by wallet
  • How to specify this? By address? By public key hash? Maybe this is best to delegate to a wallet provider.
  • What defines a "wallet" exactly?

Specifying Filters in the Runtime

Here are some options for how filters could be managed

1. Explicit add / remove APIs for adding and removing filters. Adding a filter would trigger the history of matching contracts to be fetched.
   • Pros: Very explicit, active filters managed manually (no question of cache invalidation). History is eagerly fetched.
   • Cons: Extra manual steps needed, Complexity involved in swapping out active filters at runtime
2. Filters could be added on demand and cached / cleaned up periodically.
   • Filter added when initializing history sync
   • Filter added when querying history.
   • How do we decide when to clean up a filter? Can filters be "pinned" to prevent unwanted cleanup?
   • Pros: Fewer manual steps needed/nicer API
   • Cons: History is lazily fetched so initial query latency would be worse (because history would need to be fetched), added complexity from cache invalidation strategy.
3. Filters provided on history startup (cannot change for lifetime of process, new filters added by starting a new history process).
   • Pros: By far the simplest approach
   • Cons: Lacks flexibility, complexity due to multiple history components with different filters.

Design considerations

Since traversing the chain is expensive, we may want to consider over fetching contract IDs and associating them with filter attributes. There could be a chain seek client which watches for all contract creation transactions and extracts info like: what is the role token currency for the contract? What are the pub key hashes / addresses mentioned in the contract? We could store this info in a way that is easy to query by the filter attributes, which would avoid having to re-traverse the chain from Genesis each time the filter changed.

[bwbush]

What defines a "wallet" exactly?

Cardano wallets should conform to the key-derivation procedure of CIP-1852. The sequence of all public keys can be derived from the root public key, but revealing this significantly weakens wallet privacy. (FYI, one can view this root public key in Daedalus under the "More"/"Settings" tab.)

However, if the Shelley address contains a staking key derived according to CIP-1852, then that reveals which addresses belong to a wallet. Additional CIPs may change this relationship in the future.

[bwbush]

A CIP-30 wallet will provide a list of all used keys along with the first 20 unused keys. For the purposes of Contract History, we should consider a "wallet" to just be a set of addresses and/or all of the addresses belonging to a stake key, perhaps something like the following would survive future CIPs:

type Wallet = These [AddressAny] [StakeCredential]

[jhbertra]

Data type ideas for the event-oriented representation of Marlowe contracts:

newtype ContractEvents = ContractEvents { unContractEvents :: Map BlockHeader (Map ContractId [ContractEvent]) }

newtype ContractId = ContractId { unContractId :: TxOutRef } -- The TxOut that created the contract.
newtype PayoutId = PayoutId { unPayoutId :: TxOutRef } -- The TxOut that transferred the payout.

data ContractEvent
  = Created
      { roleTokenPolicyId :: PolicyId 
      , marloweValidatorHash :: ValidatorHash
      , payoutValidatorHash :: ValidatorHash
      , contract :: Contract
      }
  | InputsApplied
      { inputs :: [Input]
      , outputs :: Outputs
      }
  | TimeoutElapsed
  | PayoutRedeemed
      { payoutId :: PayoutId
      }

data Party
  = Address Address
  | Role TokenName

newtype AccountId = AccountId Party

data Input
  = Deposit
      { fromParty :: Party
      , toAccount :: AccountId
      , assets :: Assets
      , continuation :: Maybe ContractContinuation
      }
  | Choice
      { choiceId :: ChoiceId
      , choiceValue :: ChoiceValue
      , continuation :: Maybe ContractContinuation
      }

data Outputs = Outputs
  { nextUTxO :: Maybe TxOutRef
  , payments :: [Payment]
  , warnings :: [TransactionWarning]
  }

data Payment = Payment
  { fromAccount :: AccountId
  , toRecipient :: Recipient
  , assets :: Assets
  }

data Recipient
  = Account AccountId
  | Party PaymentId Party

data ContractContinuation = ContractContinuation
  { hash :: ContractHash
  , contract :: Contract
  }

[jhbertra]

Is there any reason we should not just import most of these types from Language.Marlowe.Core.V1?

There's some missing contextual data that is relevant to Cardano in those types:

• Recipient vs Payee: Payee does not include the UTxO of the payout (nor should it, to remain blockchain-agnostic).
• Payment depends on recipient, also there is no direct correlate to this in Language.Marlowe.Core.V1
• TransactionInput includes a lot of data that does not need to be included in the events (e.g. the MarloweData / Contracts). These can be computed from the events.

Furthermore, I want to avoid pulling a dependency on plutus-apps into this project (though this could and should be factored out of marlowe-core

[jhbertra]

Input should have Notify too, even though its quite vacuous.

You are right, it should

[jhbertra]

I don't think we need TimeoutElapsed because that is equivalent to InputsApplied {inputs = []}. Additionally, TimeoutElapsed would have some outputs.

This wouldn't be triggered by an input being applied, but by the timeout actually elapsing. InputsApplied is emitted when script UTxO is consumed with a redeemer (i.e. it is associated with a transaction). TimeoutElapsed would be emitted when the node tip slot exceeds the timeout value of the current head of the contract. It's a completely optional history event, but it could still be useful to emit.

[bwbush]

I see your point and its conveniences, but it differs from how we've handled that history in the past and it breaks the correspondence between history events and ledger transactions. I'm wondering what @palas's thoughts are about this?

[jhbertra]

We can leave it out of the spec for now then - I like the correspondence between history events and transactions, as it gives them a natural ID (UTxO produced for creation, UTxO consumed for input application).

[jhbertra]

I have been thinking some more about this filtering problem, and I realized that perhaps we're giving Marlowe History a bit too much responsibility here. The problem of contract discovery is difficult to solve because different applications may have different criteria for which contracts they want to follow, and many of those criteria require domain knowledge that the runtime is not aware of (even the simple case of following all contracts related to a wallet is difficult because we need to have wallet awareness to do this).

Therefore, I propose that we solve this problem by yet again deferring some of the complexity. Instead of a general-purpose ContractFilter DSL, I suggest that Marlowe History exposes an API that allows contracts to be added or removed by ID (Contract ID = the TxOutRef of the TxO that created the contract). This allows discovery criteria to be "plugged in" to the runtime in an application-specific way. The discovery client would take responsibility for knowing which contracts it wants to follow, and it would follow them by adding or removing them from the runtime.

We could provide utilities that make this job easier. For example, we could define the notion of a "contract header" that can be used to identify and classify the contract, and provide an API that could page through all contract headers, possibly providing filtering at that point.

[bwbush]

single contract instead of a group of them

Yes!

[jhbertra]

Here's an example state diagram of the sync protocol:

stateDiagram-v2
  [*] --> Init
  Init --> Follow : FollowContract
  Follow --> Idle : ContractFound
  Follow --> Done : ContractNotFound
  Idle --> Next : RequestNext
  Idle --> Done : Done
  Next --> Retiring : Closed
  Retiring --> Done : Retired
  Retiring --> Idle : RollBackward
  Done --> [*]
  state Next {
    [*] --> CanWait : RequestNext
    CanWait --> MustReply : Wait
    CanWait --> [*] : InputsApplied
    CanWait --> [*] : PaymentRedeemed
    CanWait --> [*] : Closed
    CanWait --> [*] : RollBackward
    MustReply --> [*] : InputsApplied
    MustReply --> [*] : PaymentRedeemed
    MustReply --> [*] : Closed
    MustReply --> [*] : RollBackward
  }
  Next --> Idle : InputsApplied
  Next --> Idle : PaymentRedeemed
  Next --> Idle : RollBackward

Loading

Comments:

• The ADT version of the events in my previous post is represented as message types in the protocol. This also captures the fact that different events have different pre and post-conditions.
• The FollowContract message would specify the ContractID to follow.
• The ContractFound message would carry the contract creation information.
• The InputsApplied and Closed messages both represent an apply-inputs transaction and include the inputs that were applied. However, a Closed message indicates the contract has been closed (i.e. it does not carry a script UTxO, and it transitions the protocol to the Retiring state instead of back to the Idle state).
• The Retiring state represents the period of k blocks after the contract closes when the closure might still be rolled back, and so the protocol needs to remain alive.

There is a tradeoff here - it is less efficient than synchronizing a batch of contracts, but it is simpler to spin clients up and down as required. We could also make it more efficient at the transport layer by multiplexing multiple protocol sessions over a single transport layer channel.

[jhbertra]

Regarding where we use types from Language.Marlowe.V1.Core and where we use specialized types, I think that's something that will be more obvious in the specific contexts. For instance, I think we could use them in the sync protocol (we would need to write Binary instances for them, but that's easy enough). It may or may not be more convenient to use specialized types internally though, for example, in the Changes record or in the database code.

[paluh]

Furthermore, I want to avoid pulling a dependency on plutus-apps into this project (though this could and should be factored out of marlowe-core.

I believe that @paluh has already eliminated the dependency upon plutus-apps in the 1.35 branch.

I'm nearly there. Tomorrow I'm going to push this marlowe-core - marlowe-scripts split.

[jhbertra]

I also think that some of the semantics types use things like BuiltinByteString which we don't really want in the public API...

[jhbertra]

Marlowe History ERD V1:

erDiagram
  contract ||--o{ inputApplication : "advanced by"
  contract ||--o{ party : "participants of"
  inputApplication ||--o{ input : applies
  input ||--o{ depositAsset : deposits
  inputApplication ||--o{ payment : produces
  payment ||--o{ paymentAsset : transfers
  paymentAsset ||--o{ asset : "classified by"
  depositAsset ||--o{ asset : "classified by"
  party ||--o{ payment : receives
  party ||--o{ payment : sends
  payment ||--o| paymentRedemption : "redeemed by"
  contract {
    txOutRef contractId PK "NOT NULL"
    bigint slotNo "NOT NULL"
    bytea policyId "NOT NULL"
    jsonb contract "NOT NULL"
    bytea applicationValidatorHash "NOT NULL"
    bytea payoutValidatorHash "NOT NULL"
  }
  inputApplication {
    txOutRef inputApplicationId PK "NOT NULL"
    txOutRef contractId FK "NOT NULL"
    bigint slotNo "NOT NULL"
    txOutRef txOut
  }
  party {
    serial partyId PK "NOT NULL"
    txOutRef contractId FK "NOT NULL"
    bytea address "NOT NULL if role IS NULL and NULL if role IS NOT NULL"
    bytea role "NOT NULL if address IS NULL and NULL if address IS NOT NULL"
  }
  payment {
    serial paymentId PK "NOT NULL"
    int fromParty FK "NOT NULL"
    int toParty FK "NOT NULL"
    txOutRef inputApplicationId FK "NOT NULL"
    bigint lovelace "NOT NULL if txOut is NOT NULL"
    txOutRef txOut
  }
  asset {
    serial assetId PK "NOT NULL"
    bytea policyId "NOT NULL"
    bytea name "NOT NULL"
  }
  paymentAsset {
    int assetId FK "NOT NULL"
    int paymentId FK "NOT NULL"
    bigint quantity "NOT NULL"
  }
  paymentRedemption {
    bytea txOut PK "NOT NULL"
    bigint slotNo "NOT NULL"
    bytea txId "NOT NULL"
  }
  input {
    smallint inputNo PK "NOT NULL"
    txOutRef inputApplicationId FK "NOT NULL"
    jsonb continuationContract
    inputType tag "NOT NULL"
    int depositToAccount FK "NOT NULL if tag = deposit"
    int depositFromParty FK "NOT NULL if tag = deposit"
    bigint depositLovelace "NOT NULL if tag = deposit"
    bytea choiceName "NOT NULL if tag = choice"
    int choiceParty FK "NOT NULL if tag = choice"
    bigint choiceValue "NOT NULL if tag = choice"
  }
  depositAsset {
    int assetId FK "NOT NULL"
    smallint inputNo FK "NOT NULL"
    txOutRef inputApplicationId FK "NOT NULL"
    bigint quantity "NOT NULL"
  }

Loading

[palas]

Thanks for the detailed answers, it clarifies a lot.

Are they meaningful beyond validation, or do they impact the semantics of the contract as well? By this, I mean do they affect the happy path logic or are they just used to return error cases?

Yes, you could potentially contracts that do things like have a deposit depend on the length of the time interval of a transaction. Or ridiculous things like taking a different branch depending on whether the low bound of the time interval is odd or even.

The encoding for this is: if the txOut coiumn is NULL then it is a payment to an account (the funds are transferred only in the accounts in the State, so there is no UTxO for the payment). If the txOut column is not NULL, it is a payment to a party (either by role or by address).

That makes sense

We can certainly inline asset into the referenced tables, but bear in mind that this will increase the size of the database, because it will cause data to be replicated. It may also make indexes and statistics more complicated because of working with a denormalized schema (for example if we want to query by policy ID or token name).

It sound to me like a good tradeoff. It will probably make things much more efficient to query (less JOINs), and I am guessing having extra tables has overhead too. For example, it will need a set of indexes which may occupy a similar amount of space, and the queries become more expensive. If we want to query by policyId we can just add an index for that.

Inlining paymentRedemption is probably not a good idea, because a payment being made and it being redeemed are separate events that should be tracked separately (I'm trying to avoid mutable columns here)

Makes sense

As for folding depositAsset, this really comes down to how we want to represent multi-currency deposits. Do we want to have one IDeposit per currency type, or do we want to eventually support a single IDeposit that allows for a Map Token Quantity or similar? This is the way multi-currency support works in Cardano, but we don't necessarily need to do it the same way here.

Currently we don't support multicurency deposits, so same thinking as with paymentRedemption. It would also allow us to represent ada and tokens in a consistent (same reperesentation, no special case for ada) and in a way that is just as efficient way.

I agree, though is this supported yet? Currently, we don't require signed transactions - how are we planning on implementing this? I hesitate to encode it in the ERD before having a clear idea of how it works at the transaction level.

For public key parties they are authorised through signature, I think.

Payment redemption is all about which UTxOs to the payout validator have been redeemed. When we encounter a UTxO to the payout script (corresponding with a Pay account (Party (Role role)) token value contract), it will be recorded in the payment table with the TxId and TxIx of the UTxO in the txOut column. When we observe that UTxO as having been spent (i.e. it has been redeemed), we will record it in the paymentRedemption table. Therefore, to find unredeemed payouts:

SELECT payment.*
FROM payment
LEFT JOIN paymentRedemption USING (txOut)
WHERE payment.txOut IS NOT NULL AND paymentRedeption.txOut IS NULL

The benefit of this encoding (as opposed to having a payment.spendingTxId column for example) is that recording redeptions is an INSERT instead of an UPDATE (it does not mutate existing rows). Conversely, processing a rollback is a matter of DELETE FROM paymentRedeption WHERE slotNo > rollbackSlotNo. It also allows us to index paymentRedemption by slot number.

Makes sense

[bwbush]

Is slotNo sufficient? Wouldn't we need blockNo or blockHash somewhere? I'm thinking of the situation when a rollback occurs when the contract history component is offline. When it reconnects to the chain seeker, how would it detect that the transaction has been rolled back if the chain seeker has already rolled forward past the slotNo that the contract history has previously recorded? Or does contract history persist block-related information somewhere other than in this schema?

[jhbertra]

@bwbush you are right, we need blockNo and blockHash to prepare the intersection points for the chain seek.

[jhbertra]

Here is a new version:

erDiagram
  contract ||--o{ transaction : "advanced by"
  contract ||--o{ party : "participants of"
  transaction ||--o{ input : consumes
  input ||--o| deposit : contains
  input ||--o| choice : contains
  transaction ||--o{ payment : produces
  payment ||--o{ paymentAsset : transfers
  party ||--o{ payment : receives
  party ||--o{ payment : sends
  payment ||--o| paymentRedemption : "redeemed by"
  contract {
    txOutRef contractId PK "NOT NULL"
    bigint slotNo "NOT NULL"
    bigint blockNo "NOT NULL"
    bytea blockHash "NOT NULL"
    bytea policyId "NOT NULL"
    jsonb contract "NOT NULL"
    bytea applicationValidatorHash "NOT NULL"
    bytea payoutValidatorHash "NOT NULL"
  }
  transaction {
    txOutRef transactionId PK "NOT NULL"
    txOutRef contractId FK "NOT NULL"
    bigint slotNo "NOT NULL"
    bigint blockNo "NOT NULL"
    bytea blockHash "NOT NULL"
    timestamp intervalStart "NOT NULL"
    timestamp intervalEnd "NOT NULL"
    txOutRef txOut
  }
  party {
    serial partyId PK "NOT NULL"
    txOutRef contractId FK "NOT NULL"
    bytea address "NOT NULL if role IS NULL and NULL if role IS NOT NULL"
    bytea role "NOT NULL if address IS NULL and NULL if address IS NOT NULL"
  }
  payment {
    serial paymentId PK "NOT NULL"
    int fromParty FK "NOT NULL"
    int toParty FK "NOT NULL"
    txOutRef transactionId FK "NOT NULL"
    bigint lovelace "NOT NULL if txOut is NOT NULL"
    txOutRef txOut
  }
  paymentAsset {
    int assetId FK "NOT NULL"
    int paymentId FK "NOT NULL"
    bytea policyId "NOT NULL"
    bytea name "NOT NULL"
    bigint quantity "NOT NULL"
  }
  paymentRedemption {
    bytea txOut PK "NOT NULL"
    bigint slotNo "NOT NULL"
    bigint blockNo "NOT NULL"
    bytea blockHash "NOT NULL"
    bytea txId "NOT NULL"
  }
  input {
    serial inputId PK "NOT NULL"
    txOutRef transactionId FK "NOT NULL"
    smallint inputNo "NOT NULL"
    bytea continuationContractHash
    jsonb continuationContract
  }
  deposit {
    serial inputId FK "NOT NULL"
    int depositToAccount FK "NOT NULL"
    int depositFromParty FK "NOT NULL"
    bigint depositQuantity "NOT NULL"
    bytea depositAssetPolicyId "NOT NULL"
    bytea depositAssetName "NOT NULL"
  }
  choice {
    serial inputId FK "NOT NULL"
    bytea choiceName "NOT NULL"
    int choiceParty FK "NOT NULL"
    bigint choiceValue "NOT NULL"
  }

Loading

[bwbush]

Also, the current Marlowe semantics only pay one type of coin at a time, so payment.lovelace = 0 if a native token is paid. Maybe Ada should not be treated specially?

[jhbertra]

I vetted my initial state diagram of the Marlowe Sync Protocol by writing out the types and implementing the client and server peers. Here is a revised version based on changes that I made where the initial idea didn't stand up to scrutiny:

stateDiagram-v2
  [*] --> Init
  Init --> Follow : FollowContract
  Follow --> Idle : ContractFound
  Follow --> Done : ContractNotFound
  Idle --> Next : RequestNext
  Idle --> Done : Done
  Next --> Idle : InputsApplied
  Next --> Idle : Closed
  Next --> Idle : PaymentRedeemed
  Next --> Idle : RollBackward
  Done --> [*]
  state Idle {
    direction LR
    [*] --> Pending : RollBackward
    [*] --> Running : RollBackward
    [*] --> Running : ContractFound
    [*] --> Running : InputsApplied
    [*] --> Running : PaymentRedeemed
    [*] --> Closed : RollBackward
    [*] --> Closed : ContractFound
    [*] --> Closed : Closed
    [*] --> Closed : PaymentRedeemed
    Running --> [*] : Done
    Running --> [*] : RequestNext
    Closed --> [*] : Done
    Closed --> [*] : RequestNext
    Pending --> [*] : Done
  }
  state Next {
    direction LR
    [*] --> nextRunning : RequestNext
    [*] --> nextClosed : RequestNext
    nextRunning : Running
    nextClosed : Closed
    state nextRunning {
    direction LR
      [*] --> RunningCanWait : RequestNext
      RunningCanWait : CanWait
      RunningMustReply : MustReply
      RunningCanWait --> RunningMustReply : Wait
      RunningCanWait --> [*] : InputsApplied
      RunningCanWait --> [*] : PaymentRedeemed
      RunningCanWait --> [*] : Closed
      RunningCanWait --> [*] : RollBackward
      RunningMustReply --> [*] : InputsApplied
      RunningMustReply --> [*] : PaymentRedeemed
      RunningMustReply --> [*] : Closed
      RunningMustReply --> [*] : RollBackward
    }
    state nextClosed {
    direction LR
      [*] --> ClosedCanWait : RequestNext
      ClosedCanWait : CanWait
      ClosedMustReply : MustReply
      ClosedCanWait --> ClosedMustReply : Wait
      ClosedCanWait --> [*] : PaymentRedeemed
      ClosedCanWait --> [*] : RollBackward
      ClosedMustReply --> [*] : PaymentRedeemed
      ClosedMustReply --> [*] : RollBackward
    }
    nextRunning --> [*] : InputsApplied
    nextRunning --> [*] : PaymentRedeemed
    nextRunning --> [*] : Closed
    nextRunning --> [*] : RollBackward
    nextClosed --> [*] : PaymentRedeemed
    nextClosed --> [*] : RollBackward
  }

Loading

[jhbertra]

Here's a simplified version that collapses sub states:

stateDiagram-v2
  [*] --> Init
  Init --> Follow : FollowContract
  Follow --> Idle : ContractFound
  Follow --> Done : ContractNotFound
  Next --> Next : Wait
  Idle --> Next : RequestNext
  Idle --> Done : Done
  Next --> Idle : InputsApplied
  Next --> Idle : Closed
  Next --> Idle : PaymentRedeemed
  Next --> Idle : RollBackward
  Done --> [*]

Loading

[jhbertra]

RE: TimeoutElapsed:

I see your point and its conveniences, but it differs from how we've handled that history in the past and it breaks the correspondence between history events and ledger transactions. I'm wondering what @palas's thoughts are about this?

I thought we weren't going to represent those? I suppose there's no reason we can't do it in the protocol though - and it would be a useful source of synchronization between the node clock and the protocol client thread. I'm not sure if this needs to be represented in the ERD though.

[jhbertra]

Also - here's a use case for the AdvanceSlots chain-seek query ;)

-- assuming this generalization of FindConsumingTx is defined:
FindConsumingTxs :: Set TxOutRef -> Move (Map TxOutRef UTxOError) (Map TxOutRef Transaction)

-- Resolves when some of the specified UTxOs are resolved or when the next timeout elapses
runningContractQuery
  :: Move
      (These (Map TxOutRef UTxOError) Void)
      (These (Map TxOutRef Transaction) ())
runningContractQuery = FindConsumingTxs (Set.insert currentAppUTxO unspentPayoutUTxOs)
  `Fork` AdvanceSlots slotsUntilNextTimeout

[jhbertra]

Here is the diagram if we had TimeoutElapsed

stateDiagram-v2
  [*] --> Init
  Init --> Follow : FollowContract
  Follow --> Idle : ContractFound
  Follow --> Done : ContractNotFound
  Idle --> Next : RequestNext
  Idle --> Done : Done
  Next --> Idle : InputsApplied
  Next --> Idle : Closed
  Next --> Idle : TimeoutElapsed
  Next --> Idle : PaymentRedeemed
  Next --> Idle : RollBackward
  Done --> [*]
  state Idle {
    direction LR
    [*] --> Pending : RollBackward
    [*] --> Running : RollBackward
    [*] --> Running : ContractFound
    [*] --> Running : InputsApplied
    [*] --> Running : TimeoutElapsed
    [*] --> Running : PaymentRedeemed
    [*] --> Closed : RollBackward
    [*] --> Closed : ContractFound
    [*] --> Closed : Closed
    [*] --> Closed : PaymentRedeemed
    Running --> [*] : Done
    Running --> [*] : RequestNext
    Closed --> [*] : Done
    Closed --> [*] : RequestNext
    Pending --> [*] : Done
  }
  state Next {
    direction LR
    [*] --> nextRunning : RequestNext
    [*] --> nextClosed : RequestNext
    nextRunning : Running
    nextClosed : Closed
    state nextRunning {
    direction LR
      [*] --> RunningCanWait : RequestNext
      RunningCanWait : CanWait
      RunningMustReply : MustReply
      RunningCanWait --> RunningMustReply : Wait
      RunningCanWait --> [*] : InputsApplied
      RunningCanWait --> [*] : PaymentRedeemed
      RunningCanWait --> [*] : Closed
      RunningCanWait --> [*] : TimeoutElapsed
      RunningCanWait --> [*] : RollBackward
      RunningMustReply --> [*] : InputsApplied
      RunningMustReply --> [*] : PaymentRedeemed
      RunningMustReply --> [*] : Closed
      RunningMustReply --> [*] : TimeoutElapsed
      RunningMustReply --> [*] : RollBackward
    }
    state nextClosed {
    direction LR
      [*] --> ClosedCanWait : RequestNext
      ClosedCanWait : CanWait
      ClosedMustReply : MustReply
      ClosedCanWait --> ClosedMustReply : Wait
      ClosedCanWait --> [*] : PaymentRedeemed
      ClosedCanWait --> [*] : RollBackward
      ClosedMustReply --> [*] : PaymentRedeemed
      ClosedMustReply --> [*] : RollBackward
    }
    nextRunning --> [*] : InputsApplied
    nextRunning --> [*] : PaymentRedeemed
    nextRunning --> [*] : Closed
    nextRunning --> [*] : TimeoutElapsed
    nextRunning --> [*] : RollBackward
    nextClosed --> [*] : PaymentRedeemed
    nextClosed --> [*] : RollBackward
  }

Loading

[jhbertra]

And here is the ERD if we had timeouts in the schema:

erDiagram
  contract ||--o{ transaction : "advanced by"
  contract ||--o{ timeout : "constrained by"
  contract ||--o{ party : "participants of"
  transaction ||--o{ input : consumes
  input ||--o| deposit : contains
  input ||--o| choice : contains
  transaction ||--o{ payment : produces
  payment ||--o{ paymentAsset : transfers
  party ||--o{ payment : receives
  party ||--o{ payment : sends
  payment ||--o| paymentRedemption : "redeemed by"
  contract {
    txOutRef contractId PK "NOT NULL"
    bigint slotNo "NOT NULL"
    bigint blockNo "NOT NULL"
    bytea blockHash "NOT NULL"
    bytea policyId "NOT NULL"
    jsonb contract "NOT NULL"
    bytea applicationValidatorHash "NOT NULL"
    bytea payoutValidatorHash "NOT NULL"
  }
  transaction {
    txOutRef transactionId PK "NOT NULL"
    txOutRef contractId FK "NOT NULL"
    bigint slotNo "NOT NULL"
    bigint blockNo "NOT NULL"
    bytea blockHash "NOT NULL"
    timestamp intervalStart "NOT NULL"
    timestamp intervalEnd "NOT NULL"
    txOutRef txOut
  }
  timeout {
    txOutRef contractId FK "NOT NULL"
    bigint slotNo "NOT NULL"
    bigint blockNo "NOT NULL"
    bytea blockHash "NOT NULL"
  }
  party {
    serial partyId PK "NOT NULL"
    txOutRef contractId FK "NOT NULL"
    bytea address "NOT NULL if role IS NULL and NULL if role IS NOT NULL"
    bytea role "NOT NULL if address IS NULL and NULL if address IS NOT NULL"
  }
  payment {
    serial paymentId PK "NOT NULL"
    int fromParty FK "NOT NULL"
    int toParty FK "NOT NULL"
    txOutRef transactionId FK "NOT NULL"
    bigint lovelace "NOT NULL if txOut is NOT NULL"
    txOutRef txOut
  }
  paymentAsset {
    int assetId FK "NOT NULL"
    int paymentId FK "NOT NULL"
    bytea policyId "NOT NULL"
    bytea name "NOT NULL"
    bigint quantity "NOT NULL"
  }
  paymentRedemption {
    bytea txOut PK "NOT NULL"
    bigint slotNo "NOT NULL"
    bigint blockNo "NOT NULL"
    bytea blockHash "NOT NULL"
    bytea txId "NOT NULL"
  }
  input {
    serial inputId PK "NOT NULL"
    txOutRef transactionId FK "NOT NULL"
    smallint inputNo "NOT NULL"
    bytea continuationContractHash
    jsonb continuationContract
  }
  deposit {
    serial inputId FK "NOT NULL"
    int depositToAccount FK "NOT NULL"
    int depositFromParty FK "NOT NULL"
    bigint depositQuantity "NOT NULL"
    bytea depositAssetPolicyId "NOT NULL"
    bytea depositAssetName "NOT NULL"
  }
  choice {
    serial inputId FK "NOT NULL"
    bytea choiceName "NOT NULL"
    int choiceParty FK "NOT NULL"
    bigint choiceValue "NOT NULL"
  }

Loading

[jhbertra]

For completeness, here are a couple more general protocols which we will need here and in other components:

Query Protocol:

stateDiagram-v2
  [*] --> Init
  Init --> Next : Request(req delimiter err result)
  Next --> Done : Reject(err)
  Next --> Page : NextPage(result, Maybe delimiter)
  Page --> Next : RequestNext(delimiter)
  Page --> Done : Done
  Done --> [*]
  state Next {
    [*] --> CanReject : Request(req delimiter err result)
    [*] --> MustReply : RequestNext(delimiter)
    CanReject --> [*] : Reject(err)
    CanReject --> [*] : NextPage(result, Maybe delimiter)
    MustReply --> [*] : NextPage(result, Maybe delimiter)
  }

Loading

Notes:

• Requests determine result and error types (like in chain seek), in addition to a page delimiter type which is used to specify the next page to fetch.
• A request can be rejected with an error.
• Results are delivered in pages.
• Clients can request the next page, or exit early.
• Servers can send the next page. If it sends a Just nextPageDelimiter then there are more pages. If it sends Nothing then there are no more pages.
• For queries that only return a single page, they can specify Void as the delimiter type, allowing clients to trivially handle it.
• The server is expected to preserve the client's view of the data within the query. This implies that the pagination thread should operate within a transaction.

Async Command Protocol:

stateDiagram-v2
  [*] --> Init
  Init --> Cmd : Command(cmd)
  Init --> Cmd : Resume(id)
  Cmd --> Await : Await(id, progress)
  Await --> Poll : Poll
  Poll --> Await : Update(progress)
  Cmd --> Done : Fail(err)
  Cmd --> Done : Succeed(result)
  Poll --> Done : Fail(err)
  Poll --> Done : Succeed(result)
  Done --> [*]

Loading

Notes:

• cmd type associated with result and error type (like in chain seek), in addition to status type.
• server can respond with Fail, Succeed, or Await. Fail and Succeed end protocol, Await passes an id and a status. id is holomorphic to cmd (i.e. ∀ err result status (c ∈ cmd status err result). ∃ (i ∈ id status err result)). In other words, when you are told to wait for a cmd that is associated with a specific err, result, and status type, you will receive an id that is associated with the same err, result, and status types.
• Command message accepts a command, instructing the server to start processing it as a new command
• Resume message accepts an id of an already running command and asks for the status of the command
• In Await state, client can Poll the server for updates. Server can respond with Update indicating new progress, Fail indicating the command has failed, or Succeed indicating the command has succeeded.
• If a command does not spawn a long-running job (i.e. it will never send an Await), it can specify Void as its progress type.

Expectations:

• If server responds to Resume(id) with Succeed(result), all future calls to Resume(id) should send the same response.
• If server responds to Resume(id) with Fail(err), all future calls to Resume(id) should send the same response.
• If server responds to Poll with Succeed(result), all future calls to Resume(id) with the id obtained by Await(id, progress) should be responded to with the same Succeed(result).
• If server responds to Poll with Fail(err), all future calls to Resume(id) with the id obtained by Await(id, progress) should be responded to with the same Fail(err).

Implications:

• Server must maintain a history of commands to satisfy the expectations. This is a good thing for having an audit trail anyway.

[jhbertra]

I just revisited the Marlowe Sync Protocol and decided to make several simplifications. Instead of representing the stage of a contract's lifecycle in sub-states, there is only one Next state and the protocol allows for any type of history event to be sent at any time. This simplifies dealing with rollbacks, and intersections, and in general, makes the protocol a lot more simple.

stateDiagram-v2
  [*] --> Init
  Init --> Follow : FollowContract
  Init --> Intersect : Intersect
  Follow --> Idle : ContractFound
  Follow --> Done : ContractNotFound
  Idle --> Next : RequestNext
  Intersect --> Idle : IntersectFound
  Intersect --> Done : IntersectNotFound
  Idle --> Done : Done
  Next --> Idle : RollForward
  Next --> Idle : RollBackward
  Next --> Wait : Wait
  Wait --> Next : Poll
  Wait --> Idle : Cancel
  Done --> [*]

Loading