Add metadata to Gaia and ZTF catalogs

[troyraen]

Feature request

We are developing a prototype web service comprised of an LSDB query server backend and a Firefly frontend. For the purposes of demonstration, it would be nice to have the following metadata:

• Column descriptions
• Column units
• Specification of "default" columns (used to determine which columns are selected by default)

added to the following catalogs found at https://data.lsdb.io/:

• Gaia DR3 (GAIA_SOURCE)
• ZTF DR14 (objects)
• ZTF DR14 (sources)

We will use the metadata to populate the respective columns in the bottom panel of the screenshot below.

The default_columns property of TableProperties looks like the intended place to specify those. I don't see an obvious place for column descriptions and units at the moment. Ultimately, we'd like the metadata in the VOTable-in-Parquet format that is currently under development. In the meantime, we ought to be able to pull it from wherever you think is easiest/best to put it. We're using the hats library to get at the metadata, though we're not limited to that.

Before submitting
Please check the following:

• I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
• I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
• If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

[delucchi-cmu]

Can you please list the expected default columns for the three catalogs specified?

[troyraen]

There's no specific expectation but here's my suggestions.

Gaia

• solution_id
• designation
• source_id
• ref_epoch
• ra
• ra_error
• dec
• dec_error
• parallax
• parallax_error
• pm
• pmra
• pmra_error
• pmdec
• pmdec_error
• phot_g_n_obs
• phot_g_mean_flux
• phot_g_mean_flux_error
• phot_g_mean_mab
• phot_bp_n_obs
• phot_bp_mean_flux
• phot_bp_mean_flux_error
• phot_bp_mean_mag
• phot_rp_n_obs
• phot_rp_mean_flux
• phot_rp_mean_flux_error
• phot_rp_mean_mag

ZTF Objects

• oid
• ra
• dec
• field
• ccdid
• qid
• filtercode
• ngoodobs
• ngoodobsrel
• nobs
• nobsrel
• refMag
• refMagErr
• astrometricRMS
• chiSQ
• magRMS
• maxMag
• medianAbsDev
• medianMag
• medMagErr
• minMag

ZTF Sources

• There are very few columns so might as well do all of them.

[delucchi-cmu]

The fields for ZTF DR14 object don't match the ones you've indicated, and there are only 12 fields.

Added the 27 fields as default columns for GAIA. This was done in the properties file with a line like:

hats_cols_default=solution_id designation source_id ref_epoch ra ra_error dec dec_error parallax parallax_error pm pmra pmra_error pmdec pmdec_error phot_g_n_obs phot_g_mean_flux phot_g_mean_flux_error phot_g_mean_mag phot_bp_n_obs phot_bp_mean_flux phot_bp_mean_flux_error phot_bp_mean_mag phot_rp_n_obs phot_rp_mean_flux phot_rp_mean_flux_error phot_rp_mean_mag

Note that original request had phot_g_mean_mab, but has been corrected in the actual file's contents.

>>> lsdb.read_hats("https://data.lsdb.io/hats/gaia_dr3/gaia").hc_structure.catalog_info.default_columns
['solution_id',
 'designation',
 'source_id',
 'ref_epoch',
 'ra',
 'ra_error',
 'dec',
 'dec_error',
 'parallax',
 'parallax_error',
 'pm',
 'pmra',
 'pmra_error',
 'pmdec',
 'pmdec_error',
 'phot_g_n_obs',
 'phot_g_mean_flux',
 'phot_g_mean_flux_error',
 'phot_g_mean_mag',
 'phot_bp_n_obs',
 'phot_bp_mean_flux',
 'phot_bp_mean_flux_error',
 'phot_bp_mean_mag',
 'phot_rp_n_obs',
 'phot_rp_mean_flux',
 'phot_rp_mean_flux_error',
 'phot_rp_mean_mag']

[troyraen]

Thank you. Might as well use all the columns in ZTF DR14 Objects since there are so few.