[Bug]: Snapshot 123 is not an ancestor of 456 when running DELETE query on Trino (parallel writes)

[paul-bormans-pcgw]

What happened

Every so many seconds we append new data to a Table using pyIceberg (using Arrow tables).

On the background we run some Table maintenance operations; specifically removing old data is important (since we have limited space available). We do this using Trino:

DELETE FROM pack WHERE epoch_timestamp_tz <= timestamp '2024-11-29 13:15' AND timestampns < 1.7328861183132278e+18

Now this query takes a few minutes to complete but in the end it fails to create a new snapshot:

java.lang.IllegalArgumentException: Snapshot 3398968223100354057 is not an ancestor of 5174469298448934675
	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:430)
	at org.apache.iceberg.BaseRowDelta.validate(BaseRowDelta.java:113)
	at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:235)
	at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:386)

I'm sure it's related to the parallel write operations, because when I stop the ingest activities (Trino is then the only remaining writer); the query succeeds.

I know with use of Nessie you do not have table history (can't see older snapshots) since Nessie maintains these are catalog level instead; i'm wondering if this has anything to do with my finding?

Or are there additional iceberg configuration to set?
Can I extend the query to claim exclusive write access to the Table?

How to reproduce it

1. Table is created in pyIceberg:

catalog.create_table(
                identifier=f"{self.namespace}.{table_name}",
                schema=pt.schema,
                properties={
                    "gc.enabled": True,
})
Also partition specs are added:

update.add_field(
                    source_column_name="epoch_timestamp_tz",
                    transform=HourTransform(),
                    partition_field_name="epoch_hours",
                )

2. append Arrow tables every second
3. run trino query which take a few minutes to complete.

Nessie server type (docker/uber-jar/built from source) and version

docker: quay.io/projectnessie/nessie:0.100.1

Client type (Ex: UI/Spark/pynessie ...) and version

PyIceberg + Trino

Additional information

No response

[dimas-b]

@paul-bormans-pcgw : since PyIceberg is involved, I assume the catalog interactions were done via the Iceberg REST API. Could you please confirm this (especially with Trino)?

[paul-bormans-pcgw]

Hello @dimas-b,

You are correct; iceberg connector configuration is as follows:

connector.name=iceberg
fs.native-s3.enabled=true
iceberg.catalog.type=rest
iceberg.rest-catalog.security=NONE
iceberg.rest-catalog.uri=http\://nessie\:19120/iceberg/
iceberg.rest-catalog.vended-credentials-enabled=false
s3.endpoint=http\://minio\:9000/
s3.path-style-access=true
s3.region=us-east-1
s3.aws-access-key=minioadmin
s3.aws-secret-key=minioadmin

This is btw the Nessie generated config:

NESSIE_BASE_URL="http://127.0.0.1:19120/"
curl "${NESSIE_BASE_URL}/iceberg-ext/v1/client-template/trino?format=static"

Paul

[snazy]

Looks related/similar to #9969

[paul-bormans-pcgw]

Please confirm, you mean that I should run this DELETE query using catalog type="nessie"? (and not over type="rest")?
Also I'm pretty sure behind this first issue with ancestor not found, there is an issue with concurrent writers (that however is not a Nessie problem i think but an Iceberg configuration thing...).
Paul