DUX-823 Prices v2 rework

[0xRobin]

Thank you for contributing to Spellbook 🪄

Please open the PR in draft and mark as ready when you want to request a review.

Description:

• new upstream models for the sqlmesh prices pipeline.

Goal is to:

• simplify the lineage and bring the vwap calculation into spellbook
• have only one model flowing into sqlmesh (that has sparse minute price data)

This should resolve any duplicates we're currently having and will allow us to switch tokens from different sources without having temporary gaps or duplicates.

Also resolves DUX-702 while we're at it.

---

quick links for more information:

• README.md
• spellbook docs
• CONTRIBUTING.md

[aalan3]

Had a few comments but lgtm.

[aalan3]

Why is this model needed?

[0xRobin]

It's not needed for the sqlmesh side, just thought it would be cheaper do do it directly on the sparse dataset then later on the filled one.

[aalan3]

Can't we do this filtering below in prices_v2_dex_minute_raw directly instead?

[0xRobin]

opted to keep prices_v2_dex_minute_raw a clean union model

[jeff-dude]

is the intention of this PR to remove the need for dex_prices_block altogether? hypothetical steps:

• merge this PR, build tables in prod
• update prices pipeline to use new output from here, simplify logic in pipeline downstream
• remove dex_prices_block from spellbook

if that is the case, where is the logic here for obtaining median price per token per block? did we decide to do away with that logic (that could change results a lot and require more testing)? we would need to perform this calculation prior to the vwap calculation.
https://github.com/duneanalytics/spellbook/blob/main/dbt_subprojects/dex/models/prices/dex_prices_block.sql#L169-L190

[0xRobin]

is the intention of this PR to remove the need for dex_prices_block altogether?

That's indeed the end goal.
With the thought also being that we need to do this in parallel to the current running setup to not disturb that.

Do you know why we needed the median block price before doing the vwap?
I don't think median block with weighted average minute on top makes a ton of sense, but I might be missing something.

Is it outliers in dex.trades, where we don't want them to influence the vwap price?
Maybe just doing the median aggregation on the minute level is a good approach to get the same result.

[jeff-dude]

Do you know why we needed the median block price before doing the vwap? I don't think median block with weighted average minute on top makes a ton of sense, but I might be missing something.

Is it outliers in dex.trades, where we don't want them to influence the vwap price? Maybe just doing the median aggregation on the minute level is a good approach to get the same result.

yeah, we use the median price per block to remove outliers from DEX activity such as MEV/sandwiches/other approaches i'm no expert in 😅 we had hit a lot of outliers across various tokens and this helped get around most. we still have some outlier prices that get through, which will require more in-depth filtering/logic if fixing.

it's documented a bit here:
https://docs.dune.com/data-catalog/curated/prices/overview#methodology

i think we should stick to block for now, and can revisit minute level later to see how it compares. i think the decision of block vs. minute was due to grouping of tx's that could produce those outlier numbers (mev/sandwich/etc)

[0xRobin]

Ran some tests, and results look good.
We can safely change to a minute base median methodology instead of block median + minute vwap.

[jeff-dude]

Ran some tests, and results look good. We can safely change to a minute base median methodology instead of block median + minute vwap.

nice 🙏 as we mentioned, let's get the team on the same page before finalizing plan here