Load data using pyarrow types

[camposandro]

Adds the use_pyarrow_types argument to the read_hipscat interface. This allows us to pass the dtype_backend to the calls that read the parquet leaf files, as well as metadata, loading the data and respective schema with pyarrow types.

Using the pyarrow backend (which is now the default!) we can load strings directly into a pandas dataframe with the pyarrow string type, avoiding the creation of slow tasks to convert python strings into pyarrow strings.

This change required me to also address #303, allowing catalogs to be created with pyarrow types in from_dataframe.

Closes #279 and #303.

[github-actions]

Before [5482cf5] <v0.2.2>	After [85a46f2]	Ratio	Benchmark (Parameter)
36.8±0.9ms	52.5±0.6ms	1.43	benchmarks.time_kdtree_crossmatch
12.5±0.2ms	16.6±0.7ms	1.33	benchmarks.time_polygon_search
6.45±0.3ms	7.09±0.3ms	1.1	benchmarks.time_box_filter_on_partition
494±2ms	494±4ms	1	benchmarks.time_create_midsize_catalog
3.24±0.02s	3.21±0.01s	0.99	benchmarks.time_create_large_catalog

Click here to view all benchmarks.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.08%. Comparing base (5482cf5) to head (5cd240f).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #306      +/-   ##
==========================================
+ Coverage   99.06%   99.08%   +0.02%     
==========================================
  Files          41       41              
  Lines        1277     1306      +29     
==========================================
+ Hits         1265     1294      +29     
  Misses         12       12              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hombit]

Does it also fix #89?

[camposandro]

Does it also fix #89?

Yes, the string conversion tasks should not be required for catalogs loaded with pyarrow types.

I created a notebook that loads a chunk of ZTF sources with the "band" information, of string-type (we saw several bottlenecks happening with this column before). Using a single worker and the latest version of Dask (2024.5.0) these were the compute times I obtained:

Previously (w/ pyarrow string conversion): 181.35 s
Previously (w/o pyarrow string conversion): 88.31 s
Currently (w/ pyarrow typed catalogs): 83.81 s