Driver fetches order's app-data

[squadgazzz]

Description

As part of the flashloans support, the driver needs be able to retrieve the order's full app data in order to send flashloans hints to solvers(#3216). This PR introduces an app-data retriever that works as follows:

• Check whether the app data already exists in the cache and returns it.
• Otherwise, it sends a corresponding request to the orderbook API to fetch the app-data. The request is sent using BoxRequestSharing, which helps to avoid request duplication.
• Since the full app data is an optional field in the Order struct, responses with 404 Not Found are also cached as None. There is a small amount of orders with empty full app data in the DB, but they still exist.
• In case of any app-data fetching error, the error gets logged.

Implementation details and considerations

• The LRU cache(moka) is used because, according to the mainnet DB, there is only 2% unique app_data among all the orders, so there is no need for a TTL cache since the cache is expected to be hit for the majority of orders.
• Based on the DB data, the average full app data size is ~800 bytes. The LRU cache has 2000 capacity, approximately equal to ~1.5MB of memory.
• Once app data is fetched, it needs to be stored in the domain::Order struct. To avoid creating new order structs, the AppData is converted into an enum, which gets updated accordingly.
• Already discussed with @MartinquaXD. The full app data is cached even though the flashloan part is only expected to be used further. This can be reconsidered either in this or future PRs.
• New metrics to better understand how much time is spent fetching the app_data. Also, added the same for the balance fetching and total preprocessing just for reference.

Rate limiting

The following is only valid for colocated solvers since the API is not rate-limited for the services within the same k8s cluster.

SQL query

WITH unique_app_data_per_auction AS (
  SELECT
      oe.auction_id,
      COUNT(DISTINCT o.app_data) AS unique_app_data_count
  FROM
      order_execution oe INNER JOIN orders o ON oe.order_uid = o.uid
  GROUP BY
      oe.auction_id
),
auction_count_by_unique_app_data AS (
  SELECT
      unique_app_data_count,
      COUNT(*) AS auction_count
  FROM
      unique_app_data_per_auction
  GROUP BY
      unique_app_data_count
)
SELECT
  unique_app_data_count as unique_app_data_per_auction,
  auction_count
FROM
  auction_count_by_unique_app_data
ORDER BY
  auction_count DESC;

This image represents the following data: The left column shows how many unique app data entries have a single auction, whereas the right shows how many auctions have the corresponding amount of unique app data.

The orderbook API RPS is 5. Only 0.05% of auctions have more than 5 unique app data entries. That means there is at most a 0.05% chance that a driver hits the RPS for a single auction, where the actual probability is even lower since some of the hashes will more likely be already cached. That is why the current implementation doesn't contain a rate-limiting mechanism.

Changes

• The app-data retriever.
• A new required driver cli argument orderbook-url.
• A mocked orderbook for driver tests that always returns 404. This should be improved in driver informs solver about flashloan hints #3216, where driver tests are expected to be added.
• A config that allows switching the feature on/off, as well as configuring the app-data cache size. The feature is disabled by default.

How to test

All the current tests pass. New driver tests can be added only once the driver starts sending the collected data to solvers(#3216).

Related Issues

Fixes #3215

[squadgazzz]

This is required to satisfy BoxRequestSharing constraints.

[mstrug]

Nice and thoughtful implementation. Only couple of comments.

[sunce86]

To build a system that is easily extensible and doesn't require every piece of metadata that protocol is supposed to understand to the auction we'll make it the responsibility of the driver to fetch the appdata, parse, and understand it.

@MartinquaXD what is the most important reason not to make app-data part of the auction (I don't fully understand the argument ☝️)?

AFAIS the current solution also has a number of cons:

1. Consistency - all order data is fetched from auction, but only app data is fetched directly from orderbook API
2. Complex solution that requires cache, with all the small assumptions that need to be watched and monitored (like the number of orders in db, number of unique app-datas, orderbook rps limit, all of which can change over time)
3. Requires all fully colocated drivers (running non-reference driver) to re-implement a similar cache to stay competitive on these orders

[MartinquaXD]

what is the most important reason not to make app-data part of the auction (I don't fully understand the argument ☝️)?

The main concern is that this simply doesn't scale. If we keep tagging more and more data onto the auction we just send tons of duplicated data around most of the time.

Consistency - all order data is fetched from auction, but only app data is fetched directly from orderbook API

That's true. Although eventually I'd love if we could use this approach for more information. Conceptually we could send only the order uids plus fee policies. In that world the driver could fetch the open orders once and subscribe to new orders via websockets. This seems like a reasonable approach given that multiple solvers already query the order endpoint for additional data we didn't want to put into the auction.

Requires all fully colocated drivers (running non-reference driver) to re-implement a similar cache to stay competitive on these orders

They would already have to implement support for the entire flashloan feature anyway. I don't think adding a bit of caching logic would be a huge concern for them.

IMO Another nice advantage of offloading this logic to the driver is fewer options to break compatibility. The less data with a stable format we send between autopilot and driver the less churn there would be. The only super important data (signed order data) has been stable for a very long time. Whereas the interface between autopilot and driver changed quite a lot already.