Feature Request: incremental changes to catalog

[dlmcelroy]

I saw an intriguing talk at the Nov. 2024 IVOA meeting on HATS. I have a use case it might be suited to, although not designed for, but it involves the ability to incrementally update the catalog on a recurring basis:

1. add new records to the catalog (routine)
2. remove a record from the catalog (very rare)
3. update metadata associated with a record in the database, possible triggering re-partitioning

I can see how the index creation support might be used to facilitate option 3, or you could do it as a removal and addition. Removal, otherwise, is rare to the point of non-existent. But the first use case would be done on a near weekly basis.

It's possible that I could partition our static content into a single catalog, put the dynamic content in a living catalog and update the dynamic content by doing a merge with a small catalog of the new material - the volume of data would be relatively low, for now anyway - but this seems a bit brute force, and I'm not sure if it would address removals or updates.

[delucchi-cmu]

Yes! This has been on our radar for a while, but I don't think it appears anywhere on our roadmap.

Because of limitations on appending parquet files, our solution would likely be very close the brute force one described above, but would have the benefit of handling the finicky book-keeping for our users.