export for QA chatbot on repology DB

[ghost]

Hi guys,

Hope you are all well !

I want to create a project based on repology database and create a telegram chatbot for searching package or authors with a QA approach. I already did it with the arxiv abstracts (1.7M entries) with a telegram bot.

Refs.

• https://github.com/deepset-ai/haystack
• https://github.com/paper2code/telegram-qa-with-arxiv-papers

Is it possible to dump the whole repology database with that script ?
Do you think that side project is interesting ?

Nb. It is really simple to do such task with haystack, just have to batch a set of documents json or txt ^^

Cheers,
X

[AMDmi3]

telegram chatbot for searching package or authors with a QA approach

Not quite sure what the usecase for this one would be, but why not. I've had distant plans on telegram bot which would inform users on package updates though.

Customized dumps are possible, I just need a desired format. Also note that dumps of whole Repology database are already available (https://dumps.repology.org/), but the format may not be convenient since these are PostgreSQL database dumps.

[ghost]

We can add this feature as I am building a multi-featured bot. No problem.

Format would be probably json but I wanted to ask your help on that point.
How can do a json dump with this export script ?

[AMDmi3]

This script is intended to run on Repology server and regularly produce dumps in consumer-specific format, it needs access to repology SQL database.