Paging not working as expected

[nocollier]

This script currently loops over all items in the response and depends on the user setting --limit high enough to include all items in the index. This is because I found that pages to not be functional. This is what I should have done:

client = pystac_client.Client.open(ref_endpoint_url)
for page in client.search().pages():
    for item in page:
        # check item

But this only ever gives me a single page. There are 2 relevant parameters to search:

• max_items – The maximum number of items to return from the search, even if there are more matching results
• limit – A recommendation to the service as to the number of items to return per page of results.

From my testing, limit works but forces a single page. So if there are additional records, you don't get them. I am unsure if this is a python client problem or something else but it means that you need to set limit to all of the records of the index you want to check.

[nocollier]

This will print a single page with a number of items equal to what you set limit.

import pystac_client

client = pystac_client.Client.open("https://api.stac.esgf-west.org/")
p = 1
for page in client.search(collections="CMIP6", max_items=1000, limit=3).pages():
    print(f"Page {p} has {len(page)} items.")
    p += 1

[nocollier]

@sturoscy-personal This appears to be a problem with our (west's) endpoint. I have to change the collection to lower case, but running the same code on the CEDA endpoint returns the proper pagination.

import pystac_client

client = pystac_client.Client.open("https://api.stac.ceda.ac.uk/")
results = client.search(collections="cmip6", max_items=10, limit=3)
pages = results.pages()
p = 1
for page in pages:
    print(f"Page {p} has {len(page)} items.")
    p += 1

returns

Page 1 has 3 items.
Page 2 has 3 items.
Page 3 has 3 items.
Page 4 has 1 items.