feat: lutra query runner

[aljazerzen]

Initial take on #3825. Inspired by prql-query.

Adds a query runner named Lutra that is basically a wrapper for prqlc compile, sqlite3 run and print(pl.DataFrame).

Connection parameters are defined within PRQL source with @lutra.sqlite annotation.

Uses connectorx for executing the query and converting to Apache Arrow. Uses polars to print the dataframe.

Next step is to create Python bindings so the resulting dataframe can be used with other existing tooling.

[hulxv]

Before a few months, I built a tool called prqlite (I think you took a look at it). Is there any difference between it and Lutra?

[eitsupi]

Oops, this seems to depend on a very old version of polars.......

[aljazerzen]

Before a few months, I built a tool called prqlite (I think you took a look at it). Is there any difference between it and Lutra?

There are. prqlite is primarily a TUI and it does that well. My intention with Lutra is to be able to call it from Python (and other languages) while providing it with only PRQL source.

[aljazerzen]

Oops, this seems to depend on a very old version of polars

Yeah. It also won't compile for wasm32-unknown-unknown without a serious effort, as we'd have to compile polars and rusqlite too. That does not seem like a battle worth taking right now.

[hulxv]

My intention with Lutra is to be able to call it from Python (and other languages) while providing it with only PRQL source.

Then, Lutra is something like a library or do I misunderstand it?

[aljazerzen]

Then, Lutra is something like a library or do I misunderstand it?

Yes, but also with an option to invoke it from the command line.

[max-sixty]

Before a few months, I built a tool called prqlite (I think you took a look at it). Is there any difference between it and Lutra?

@hulxv this is cool! Nice work. I don't think we knew about this...

(the goals seem to be slightly different to Lutra fwiw)

[max-sixty]

(FYI if you're in the mood for new tools, I really like rargs rather than xargs for this sort of thing. I almost never use xargs now)

[max-sixty]

...even if I still maintain that pre-commit is better for these sorts of lints! 😄

[max-sixty]

FYI if you want to put nextest & {{.packages}} into the base task totally fine w me. Then we have a single task and can pass packages.

(but also zero stress if you prefer to have them separate)

[aljazerzen]

I like to have a separate task that builds and tests only the thing that I'm currently developing.

What do you mean "pass packages"? To have the base task take packages as argument?

[max-sixty]

What do you mean "pass packages"? To have the base task take packages as argument?

Currently this task has a {{.packages}} arg, so it's possible to pass a subset of packages

I like to have a separate task that builds and tests only the thing that I'm currently developing.

Very much agree! TBC, this task is almost identical to the task in the parent task, but passes -p lutra to .packages. So they could be combined.

(I can make a PR later, maybe that's an easier way of explaining)

[max-sixty]

How do we want to think about coupling the target and the query?

I would think we want to use some form of composition, where we say "associate this chinook.db file with this module chinook" somewhere, but we don't couple their definitions, so it's possible to change.

At a more conceptual level, it's could be similar to applying a functor in OCaml (or somewhat implementing a trait in Rust) — specifying a mapping of conceptual queries/functions to concrete queries/functions. Another reference is dbt's approach for templating the source of a query, so it's possible to run the same queries on a production vs dev vs personal database / schema.

I don't have a really good idea for how to implement this, and obv fine to experiment with things. But I do think that a really good design would be to make decouple the query and the target.

[aljazerzen]

Yeah, I was thinking about not having actual connection params within the PRQL source, similar to how dbt does it.

I've opted for this approach because it is way more convenient and it does not actually rule out composition - the queries are not defined within the database module. This means that one could swap out the database module and run the queries on a different database.

This could be done by defining the database module in a separate file, which is not committed to VCS or is overridden in production.

[max-sixty]

Yes, we can think about this.

I think there's some nice design that's possible, where we map the relations to concrete tables. dbt do a nice job at the moment even if it needs a decent amount of jinja.

I'll meditate on it...

[max-sixty]

Exciting!

Maybe we put a readme of a couple of lines saying what it is and that it's still experimental?

[max-sixty]

FYI can merge despite the failures — one is a cargo-audit which is fine and we can ignore; the other is the drop in test coverage

[aljazerzen]

I'd like to replace connector-x with connector-arrow before merging. ConnectorX has stale dependencies and does a lot of things that are not needed in for lutra.