"TypeError: Object of type FrozenEventV3 is not JSON serializable" prevents sync from working, causing an indefinite hang until the reverse proxy times out.

[TheArcaneBrony]

Description

Some errors are not causing sync to return an internal server error, instead silently leaving the connection in an open state until the reverse proxy timed out (reproduced with 1h sync limit).

Steps to reproduce

unknown, i had concurrently mass joined rooms when the issue appeared, though CPU load had dropped back to idle levels.

Homeserver

rory.gay

Synapse Version

1.99.0

Installation Method

Docker (matrixdotorg/synapse)

Database

PostgreSQL, single server, no portdb, no backups

Workers

Single process

Platform

NixOS, QEMU VM, 16GB of ram, 14vCPUs

Configuration

Presence: yes
Experimental features:

experimental_features = {
  "org.matrix.msc3026.busy_presence" = true;
  "fi.mau.msc2815" = true;
  "org.matrix.msc3881" = true;
  "org.matrix.msc3874" = true;
  "org.matrix.msc3912" = true;
};

Message retention: redaction_retention_period = null; (aka unset, iirc?)
Modules: no

Full configuration: https://cgit.rory.gay/Rory-Open-Architecture.git/tree/host/Rory-nginx/services/matrix/synapse.nix?h=2b68eeed357e6b83103de561637f465c4f6b363c

Relevant log output

twisted: [sentinel] Unhandled error in Deferred:
twisted: [sentinel] 
Traceback (most recent call last):
  File "/nix/store/gmygp534ydqbs1iw63090m9vixkcydrq-python3-3.11.7-env/lib/python3.11/site-packages/twisted/internet/defer.py", line 1996, in _inlineCallbacks
    result = context.run(
  File "/nix/store/gmygp534ydqbs1iw63090m9vixkcydrq-python3-3.11.7-env/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/nix/store/fr7dnga88i7c49s2xkhkdz254wszvvcm-matrix-synapse-1.99.0/lib/python3.11/site-packages/synapse/http/server.py", line 870, in _async_write_json_to_request_in_thread
    json_str = await defer_to_thread(request.reactor, encode, span)
  File "/nix/store/gmygp534ydqbs1iw63090m9vixkcydrq-python3-3.11.7-env/lib/python3.11/site-packages/twisted/python/threadpool.py", line 269, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/nix/store/gmygp534ydqbs1iw63090m9vixkcydrq-python3-3.11.7-env/lib/python3.11/site-packages/twisted/python/threadpool.py", line 285, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/nix/store/gmygp534ydqbs1iw63090m9vixkcydrq-python3-3.11.7-env/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/nix/store/gmygp534ydqbs1iw63090m9vixkcydrq-python3-3.11.7-env/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/nix/store/fr7dnga88i7c49s2xkhkdz254wszvvcm-matrix-synapse-1.99.0/lib/python3.11/site-packages/synapse/logging/context.py", line 975, in g
    return f(*args, **kwargs)
  File "/nix/store/fr7dnga88i7c49s2xkhkdz254wszvvcm-matrix-synapse-1.99.0/lib/python3.11/site-packages/synapse/http/server.py", line 863, in encode
    res = json_encoder(json_object)
  File "/nix/store/fr7dnga88i7c49s2xkhkdz254wszvvcm-matrix-synapse-1.99.0/lib/python3.11/site-packages/synapse/http/server.py", line 750, in _encode_json_bytes
    return json_encoder.encode(json_object).encode("utf-8")
  File "/nix/store/w4fvvhkzb0ssv0fw5j34pw09f0qw84w8-python3-3.11.7/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/nix/store/w4fvvhkzb0ssv0fw5j34pw09f0qw84w8-python3-3.11.7/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
  File "/nix/store/fr7dnga88i7c49s2xkhkdz254wszvvcm-matrix-synapse-1.99.0/lib/python3.11/site-packages/synapse/util/__init__.py", line 63, in _handle_immutabledict
    raise TypeError(
TypeError: Object of type FrozenEventV3 is not JSON serializable

Anything else that would be useful to know?

Was also dealing with this message in logs:

synapse.storage.util.partial_state_events_tracker: [GET-37433] Awaiting un-partial-stating of events ['$8UwTGZaBQZ-mYtJvPCfyOKsHnUZT5SOMFWS4gBgo4-Y']

Issue was resolved by purging rooms that popped up with this message using the admin API.