[BUG] Torii is not returning indexing error even though it hasn't fully indexed data

[steebchen]

In the beginning, it returns:

{
  "errors": [
    {
      "message": "World does not have any indexed data yet."
    }
  ]
}

which is good. However, at some point, the message disappears, even though the data is not fully indexed yet. Some data is returned but it takes another plenty of minutes. Maybe this might be due to the data not fully being written do the db yet?

[Larkooo]

We show that error when there are 0 models indexed yet. The error disappears when at least 1 model has been indexed. So it isnt supposed for the world to be fully indexed, because anyway indexing is not finite. I can get rid of the error though ig, as it is a bit confusing

[steebchen]

Makes sense. Is there a way to know when the initial indexing is done? That would be very helpful when scaling up a cluster of torii replicas

[glihm]

Makes sense. Is there a way to know when the initial indexing is done? That would be very helpful when scaling up a cluster of torii replicas

This should be something we can have yeah, when torii is indexing the pending block and not historical blocks. We already have in the database this info with the contracts heads @Larkooo right? Maybe in an other PR we could expose this info in a similar manner to the syncing endpoint of the starknet API where we can query the syncing state of a node.

[steebchen]

#2827 just removes the message but doesn't implement the full solution we've talked above right?

[glihm]

#2827 just removes the message but doesn't implement the full solution we've talked above right?

Ah right, it removed the error, but doesn't expose any syncing state. But maybe the syncing state should be queried differently, with a specific query on the tables I mentioned.

Since we can't return a valid data for a query with other data that is not expecting in return.

Currently people are mostly using SQL to know the Torii syncing state:

SELECT * FROM contracts;

This gives the head for each contract:

Do you think we need an other endpoint to give the head of each contracts specifically? Or since a SQL query can be crafted for that, it's ok like so?

[steebchen]

I definitely need some way of checking the state "is this node fully synced" or not to be able to provide proper upscaling of replicas. So the ideal solution would be to have an endpoint /ready or something which returns 200 only when that is the case. I can also query the SQL though, it's not much different for me. The only question I have is what do I compare the HEAD values to; where to get the comparison from?

[glihm]

I definitely need some way of checking the state "is this node fully synced" or not to be able to provide proper upscaling of replicas. So the ideal solution would be to have an endpoint /ready or something which returns 200 only when that is the case. I can also query the SQL though, it's not much different for me. The only question I have is what do I compare the HEAD values to; where to get the comparison from?

Ah, I see.
In this case, you need the RPC used by torii to know the head of the chain and if torii is at the head.
But the simplest way would be a dedicated endpoint with a status retuned then.

@Larkooo does this sounds good to you too?

[Larkooo]

I think for now you could just do something like

SELECT last_pending_block_tx from contracts

which will mean the contract has been synced from a pending block. If it switches back to range fetching, that will get NULLed

so in combination with the sql endpoint. you could do a fetch to
http://localhost:8080/sql?q=SELECT%20last_pending_block_tx%20from%20contracts which will give you back a JSON object with last_pending_block_tx. If its null then the state is currently syncing, otherwise it has been synced from a pending block.

@steebchen @glihm

[glihm]

Sounds great to me @Larkooo, @steebchen if you have any issue with this solution, please re-open here or a new issue to follow up on that.

[steebchen]

Confirmed it works, thanks!