Skip to content

Commit

Permalink
DBZ-7987 Updated FAQ entry for oversized BSON documents in MongoDB
Browse files Browse the repository at this point in the history
  • Loading branch information
jcechace authored and jpechane committed Jun 27, 2024
1 parent 7e4cac3 commit 2a1d695
Showing 1 changed file with 2 additions and 12 deletions.
14 changes: 2 additions & 12 deletions documentation/faq.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -383,15 +383,5 @@ The database timezone must be specified explicitly in this situation, using the
This may be the case when Debezium (or rather the MongoDB change stream cursor) encounters a change event document with total size exceeding the https://www.mongodb.com/docs/manual/reference/limits/#mongodb-limit-BSON-Document-Size[BSON document size limit of 16 megabytes].
Note that depending on the used `capture.mode` this issue can still manifest even when the actual value of the *stored* documents is significantly lower.


Starting with Debezium version 2.3.4 you can use the `cursor.pipeline` and `cursor.pipeline.order` properties to circumvent this error

First set the `cursor.pipeline.order` to `user_first`, then set `cursor.pipeline` to add the following aggregation stage.


```
[{ "$match": { "$and": [{"$expr": { "$lte": [{"$bsonSize": "$fullDocument"}, 8000]}}, {"$expr": { "$lte": [{"$bsonSize": "$fullDocumentBeforeChange"}, 8000]}}]} }]
```

In the example above all events for documents with original or updated size exceeeding 8000 bytes will be safely filtered out.

To mitigate this issue, refer to the documentation of link:/documentation/reference/stable/connectors/mongodb.html#mongodb-property-cursor-oversize-handling-mode[cursor.oversize.handling.mode] and
link:/documentation/reference/stable/connectors/mongodb.html#mongodb-property-cursor-oversize-skip-threshold[cursor.oversize.skip.threshold] properties.

0 comments on commit 2a1d695

Please sign in to comment.