Release v2.7.0

Changelog

Notice for JetStream Users

See important note if using LeafNode regarding domains.

Go Version

• 1.17.6: Both release executables and Docker images are built with this Go release.

Added

• Configuration:
  • Ability to configure account limits (max_connections, max_subscriptions, max_payload, max_leafnodes) in server configuration file (#2755)
• JetStream:
  • Overflow placement for streams. A stream can now be placed in the closest cluster from the origin request if it can be placed there (#2771, #2779)
  • Support for ephemeral Pull consumers (client libraries will need to be updated to allow those) (#2776)
  • New consumer configuration options (#2776):
    • For Pull Consumers:
      • MaxRequestBatch to limit the batch size any client can request
      • MaxRequestExpires to limit the expiration any client can request
    • For ephemeral consumers :
      • InactiveThreshold duration that instructs the server to cleanup ephemeral consumers that are inactive for that long
  • Ability to configure max_file_store and max_memory_store in the jetstream{} block as strings with the following suffixes K, M, G and T, for instance: max_file_store: "256M". Thanks to @hooksie1 for the contribution (#2777)
  • Support for the JWT field MaxBytesRequired, which defines a per-account maximum bytes for assets (#2779)
• MQTT:
  • Support for websocket protocol. MQTT clients must connect to the opened websocket port and add /mqtt to the URL path. Thanks to @Etran-H for the suggestion (#2735)
• TLS:
  • Ability to rate-limit the clients connections by adding the connection_rate_limit: <number of connections per seconds> in the tls{} top-level block. Thanks to @julius-welink for the contribution (#2573)

Improved

• JetStream:
  • MemStore is improved in the presence of a lot of interior delete messages, which could happen for instance with the use of KV store (#2752)
  • Pull consumers behavior: more resilient, better at waiting request management, and the ability to determine and watch interest cross account (#2776)
  • In clustering mode, with streams with replication factor greater than 1, a server will now use much less memory. For instance: 1,000 streams, with 10 consumers each would use about 15GB of memory, with this change memory usage would go down to about 1.5GB (#2775)
• Monitoring:
  • Make the HTTP(s) monitoring endpoints available before starting JetStream to be able to monitor whether it is available during startup and recovery of streams. This could help with health probes (#2782)
• Websocket:
  • Added client IP from X-Forwarded-For header. Thanks to @byazrail for the suggestion (#2734, #2769)

Changed

• JetStream:
  • JetStream API traffic will now be denied across leaf nodes, unless the system account is shared and the availability domain is identical (#2693)

Updated

• Some dependencies (#2727)

Fixed

• JetStream:
  • Possible panic "could not decode consumer snapshot". Thanks to @raoptimus for the report (#2738)
  • Added missing entries to stream and consumer list requests (#2739)
  • Interest across gateways that would prevent a push consumer from receiving messages. Thanks to @ZMing316 for the report (#2750)
  • Stream could "stall" after printing warnings such as "AppendEntry failed to be placed on internal channel" (#2751)
  • Stream (with WorkQueue retention policy) could have its first/last sequences reset to 0 after a non-clean server restart, causing pull subscriptions to fail getting newly published messages (#2753)
  • Stall of consumers after getting a "partial cache" error message in the server log. Thanks to @rino-pupkin, @raoptimus, @broken-ufa and @abishai for the report (#2761)
  • Large number of ephemeral consumers could exhaust Go runtime's maximum threads. Thanks to @rh2048, @OpenGuidou for the report (#2764)
  • Restarting a cluster with lots of streams/consumers could cause routes to break with "write deadline" in some situations (#2780)
  • Changes to inter-process communications could solve some issues caused by inability to send messages to internal Go channels (#2775, #2780)
  • A consumer's number of pending messages (or "Unprocessed Messages" in NATS cli) could be wrong when the consumer was created with inflight messages. Stepping down of the consumer leader would solve the discrepancy (#2776)
  • Possible file store deadlock in when trying to rebuild some state (#2781)
• Monitoring:
  • Possible panic when inspecting the /jsz endpoint. Thanks to @rh2048 for the report (#2743)
• TLS:
  • When setting verify_and_map to true, if a connection connects with a certificate with an ID matching an existing user, but that user's allowed_connection_types is specified and does not have the type of the connection trying to connect, the server would panic (#2747)
• Check for no_auth_user should be done only when no authentication at all is provided by the connection (#2744)
• System account issue where the wrong structure was updated (#2757)
• Broken link to monitoring documentation. Thanks to @kfabryczny for the contribution (#2766)

Complete Changes

v2.6.6...v2.7.0

Release v2.6.6

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.10: Both release executables and Docker images are built with this Go release.

Changed

• Profiler:
  • Moved the start of the profiler earlier in the start sequence so it is possible to get some insight should a start take a long time (for instance recovering JetStream datastore) (#2723)
• Added LEAFNODE_WS in allowed_connection_types to allow or prevent Leafnode over websocket. Previously, if allowed_connection_types was specified and contained LEAFNODE, this user could be used to create a Leafnode connection over websocket. The existing WEBSOCKET connection type is used for regular client applications (#2707)

Fixed

• JetStream:
  • Slowness with large interior deletes (#2711)
  • Possible panic when removing a source from a stream. Thanks to @kylebernhardy for the report (#2712)
  • Reject invalid subjects in stream configuration, and attempt to fix them on recovery (#2717, #2721)
  • Prevent stream update to add subjects to mirror and remove them on recovery (#2718, #2721)
  • Application may receive corrupted headers when receiving from a consumer configured with "MetaOnly" option (#2719)
  • Aligned internal timeout to 4 seconds for stream and consumer list requests (#2722)
  • Incomplete error when consumer list in cluster mode failed (#2725)
• Leafnode:
  • When using allowed_connection_types, it was not possible to restrict a Leafnode connection over websocket (#2707)
  • Reaching the account connections limit left an outgoing Leafnode connection in a bad state (#2715)
• Monitoring:
  • TLS configuration changes would not take effect after a configuration reload (#2714)
• Documentation:
  • Links in the README. Thanks to @Shoothzj for the contribution (#2724)

Complete Changes

v2.6.5...v2.6.6

Release v2.6.5

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.10: Both release executables and Docker images are built with this Go release.

Added

• Server option AlwaysEnableNonce to force the server to send a nonce in the INFO block, regardless if the client connects with a NKey or not. Since its primarily useful to embedded scenarios there is no corresponding option in configuration file (#2696, #2699)

Fixed

• JetStream:
  • The stream list API request was not using the optional "subject" filter (#2695)
  • Followers may keep trying to reset streams, due to either snapshots not containing any actionable data or when stream had lot of failures by design (such as KV abstraction) (#2702)
  • Memstore: setting max messages per subject to 1 would not be properly enforced (#2704)
• OCSP:
  • Allow intermediates, or designated responders, to sign responses on behalf of the CA (#2692)
• A slow consumer could cause the publisher to block. Thanks to @cpiotr, @GuanYin108 and @lukedodd for the reports (#2684)
• Possible panic due to latency tracking when more than one replies are sent (#2688)

Complete Changes

v2.6.4...v2.6.5

Release v2.6.4

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.10: Both release executables and Docker images are built with this Go release.

Improved

• JetStream:
  • Allow system account to respond with "JetStream not enabled", which makes interactions with NATS cli better (#2671)
  • Allow certain consumer attributes to be updated. Thanks to @andreib1 for the suggestion (#2674)
• Websocket:
  • Add client IP in websocket upgrade error messages (#2660)

Fixed

• JetStream:
  • Messages not expiring properly after server restart (#2665)
  • Conditional failures for stream messages could cause stream resets (#2668)
  • Duplicate stream create returned wrong response type (#2669)

Complete Changes

v2.6.3...v2.6.4

Release v2.6.3

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.9: Both release executables and Docker images are built with this Go release.

Fixed

• Gateways:
  • Each server within a cluster gossips its gateway URL, however, when two servers create route to each other and one route is dropped as a duplicate, a server may lose the peer's gateway URL and therefore would not be able to gossip it to inbound gateway connections (#2653)
• JetStream:
  • Purging would, in some cases, not honor the amount of messages to keep after purge (#2623)
  • Panic caused by the Unlock of unlocked RWMutex, which would happen through non client connections when the limit of JetStream API in-flight requests was reached. Thanks to @minight for the report (#2645)
  • Cluster becomes inconsistent with constant "catchup for stream stalled" warnings. Thanks to @aksdb for the report (#2648)
  • Ensure that stream's sealed, deny delete, deny purge and allow rollup booleans in stream config are always set (for languages that may not set to false when decoding JSON when absent) and set the deny delete and purge to true and allow rollup to false when a stream is sealed (#2650)
  • Redelivered state of consumers was not properly decoded from file (#2652)
• LeafNode:
  • Apparent memory leak on "Authorization Violation" errors. Thanks to @DavidSimner for the contribution (#2637)
• Monitoring:
  • The /varz endpoint may still show gateway URLs of servers that have been removed from the remote cluster (#2647)
  • The /varz endpoint would not show the configured URLs of a remote gateway block if that remote gateway was the one of the server being inspected (#2653)
• OCSP:
  • Only last of LeafNode and Gateway remotes were being updated. Thanks to @guodongli-google for the report (#2632)
• TLS:
  • If a connection is closed before the specified TLS timeout, the connection object would be retained in the go runtime due to the TLS timeout timer not being stopped, which could be seen as a memory leak, until that timer fired (#2638)
• Redact URLs before logging. Thanks to @fowlerp-qlik for the report (#2643)

Complete Changes

v2.6.2...v2.6.3

Release v2.6.2

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.9: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Ability to seal a stream through a stream update (#2584)
  • Ability to do roll ups (#2585, #2593, #2600, #2601)
  • Allow consumers to request only headers to be delivered with HeadersOnly consumer configuration (#2596)
  • New stream configuration properties to limit delete/purge/rollups (#2595)

Updated

• JWT library dependency to v2.1.0

Improved

• MQTT:
  • The client ID will now be part of the client log statements. Thanks to @imranrazakhan for the report (#2598)

Fixed

• JetStream:
  • Adjust cluster size on cold start in mixed mode (#2579)
  • Issue with the "get message" audit event (#2583)
  • Some data races (#2590, #2619)
  • More robust checking for decoding append entries (#2606)
  • Issues with RAFT around snapshots and recovery (#2615)
  • Starting a consumer with "deliver last per subject" with a filtered subject same as the stream's subject would possibly return wrong NumPending value (#2616)
  • Purging filtered stream resets all stream consumer offsets. Thanks to @minight for the report (#2616, #2617)
  • Honor max memory and max file settings of 0. Thanks to @biskit for the report (#2618)
• Websocket:
  • Issue with compression and Safari browsers using the experimental feature NSURLSession WebSocket. Thanks to @byazrail for the report (#2613)
• Handling of message headers in some system requests (#2580)
• Go doc fixes and some code cleanup. Thanks to @dtest11 for the contributions (#2610, #2614)

Complete Changes

v2.6.1...v2.6.2

Release v2.6.1

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.8: Both release executables and Docker images are built with this Go release.

Fixed

• JetStream:
  • Deadlock when using stream mirrors with source with non limit retention, which could manifest with increase memory usage (#2561)
  • Possible panic due when getting a slice of a message block cache buffer (#2565)
  • Internal "direct consumers" used for mirroring were incorrectly subject to the "max consumers" limit (#2564)
  • Storage limit of 0 (which means disabled) was not always enforced (#2563)
  • Handling when exceeding account resources (#2569)
  • Internal connections were counted toward the account's max connections (client or leafnode) (#2568)

Complete Changes

v2.6.0...v2.6.1

Release v2.6.0

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.8: Both release executables and Docker images are built with this Go release.

Added

• Monitoring:
  • JetStream's reserved memory and memory used from accounts with reservations in /jsz and /varz endpoints (#2539, #2555)
• Hardened systemd service. Thanks to @c0deaddict for the contribution (#2318)

Changed

• JetStream:
  • The server store message headers with a uint 16, which limits to 64KB, however, it was not rejecting incoming messages with headers bigger than that but then was failing its recovery (#2526)
  • The server was accepting consumer creation requests with flow control enabled but without heartbeats, which could lead to the consumer stalling if the library missed the flow control message. The server will now reject such consumer creation request (#2533)
  • Trying to add a stream with a configuration that matched exactly the one of an existing stream would report an error in cluster mode, not in standalone mode. The cluster mode will now behave as standalone mode and treat the stream creation as idempotent (#2535)
  • The server will now reject negative duplicate window values (#2546)
• The server will now fail to start if a token is specified in cluster{authorization{}} or gateway{authorization{}}. Routes and Gateways have never supported token authentication, however, it was silently ignored. Thanks to @antong for the report (#2518)

Fixed

• JetStream:
  • Filestore compaction issue that could to message lookup to fail, possibly stalling consumers (#2522)
  • Reject messages with headers greater than 64KB (#2526)
  • Condition where assets could be created in multiple leafnode domains (#2529)
  • Reject consumer creation requests with a flow control but no heartbeat (#2533)
  • Stream creation request was idempotent in standalone mode, but not in clustered mode (#2535)
  • In clustered mode, getting consumer information or a just created consumer could sometimes return a "consumer not found" error (#2537)
  • Make large pull subscribe batch requests expire more efficiently (#2538)
  • Stabilize filestore implementation. Thanks to @izarraga for the report (#2545)
  • Reject duplicate window with negative value (#2546)
  • Memory store memory leak when running in cluster mode. Thanks to @anjmao for the report (#2550)
  • Some issues with RAFT WAL repair (#2549)
  • A consumer on mirror stream would fail to be recovered if its stream was recovered before the mirrored stream. Thanks to @tbeets for the report (#2554)
• Websocket:
  • Possible panic when decoding CLOSE frames. Thanks to @byazrail for the report (#2519)
• Memory leak when using Queue Subscriptions. Thanks to @andrey-shalamov for the report and fix! (#2517)
• Fail configuration that would have token specified in cluster{} or gateway{}'s authorization{} blocks. Authentication token were never supported there but the misconfiguration was silently ignored. Thanks to @antong for the report (#2518)

Complete Changes

v2.6.0...v2.5.0

Release v2.5.0

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See note in the Improved and Changed section.

Go Version

• 1.16.8: Both release executables and Docker images are built with this Go release.

Added

• MQTT/Monitoring:
  • MQTTClient in the /connz connections report and system events CONNECT and DISCONNECT. Ability to select on mqtt_client. Thanks to @carr123 and @imranrazakhan for the suggestions (#2507)

Improved

• MQTT:
  • Sessions are now all stored inside a single stream, as opposed to individual streams, reducing resources usage (#2501)

Changed

• JetStream:
  • Using Nats-Expected-Last-Subject-Sequence header with a value of 0 now means that the server will reject the store command if there were messages on this subject (#2506)
• MQTT:
  • Due to improvement described above, when an MQTT client connects for the first time after an upgrade to this server version, the server will migrate all individual $MQTT_sess_<xxxx> streams to a new $MQTT_sess stream for the user's account (#2501)

Fixed

• JetStream:
  • Possible deadlock due to lock inversion (#2479)
  • Possible consumer stall. Thanks to @carr123 for the report (#2480)
  • Don't send 408 status when pull request expires. This has no visible impact for users, but library implementers may want to be aware of it (#2482)
  • During peer removal, try to remap any stream or consumer assets (#2493)
  • Issues with remapping of stream and/or consumer assets during peer removal (#2493)
  • Issue that could lead to perceived message loss (#2490)
  • Message cleanup for interest stream and AckNone consumers in clustered mode (#2499)
  • Suppress duplicates on JS deny all for system account (#2502)
  • Consumers stopped receiving messages. Thanks to @anjmao, @izarraga and @tigrato for the report (#2505)
• Handle SIGTERM on Windows platform (#2481)
• Account resolver TLS connection may fail with x509: certificate signed by unknown authority errors. Thanks to @Ryner51, @ronaldslc for the report (#2483)

Complete Changes

v2.4.0...v2.5.0

Release v2.4.0

Changelog

Notice for JetStream Users

With the latest release of the NATS server we have fixed bugs around queue subscriptions and have restricted undesired behavior that could be confusing or introduce data loss by unintended/undefined behavior of client applications. If you are using queue subscriptions on a JetStream Push Consumer or have created multiple push subscriptions on the same consumer, you may be affected and need to upgrade your client version along with the server version. We’ve detailed the behavior with different client versions below.

With a NATS Server prior to v2.4.0 and client libraries prior to these versions: NATS C client v3.1.0, Go client v1.12.0, Java client 2.12.0-SNAPSHOT, NATS.js v2.2.0, NATS.ws v1.3.0, NATS.deno v1.2.0, NATS .NET 0.14.0-pre2:

• It was possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is not correct since each instance will receive the same copy of a message and acknowledgment is therefore meaningless since the first instance to acknowledge the message will prevent other instances to control if/when a message should be acknowledged.
• Similar to the first issue, it was possible to create many different queue groups for one single JetStream consumer.
• For queue subscriptions, if no consumer nor durable name was provided, the libraries would create ephemeral JetStream consumers, which meant that each member of the same group would receive the same message than the other members, which was not the expected behavior. Users assumed that 2 members subscribing to “foo” with the queue group named “bar” would load-balance the consumption of messages from the stream/consumer.
• It was possible to create a queue subscription on a JetStream consumer configured with heartbeat and/or flow control. This does not make sense because by definition, queue members would receive some (randomly distributed) messages, so the library would think that heartbeat are missed, and flow control would also be disrupted.

If above client libraries are not updated to the latest but the NATS server is upgraded to v2.4.0:

• It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. Since the check is performed by the library (with the help of a new field called PushBound in the consumer information object set by the server), this mis-behavior is still possible.
• Queue subscriptions will not receive any message. This is because the server now has a new field DeliverGroup in the consumer configuration, which won’t be set for existing JetStream consumers and by the older libraries, and detects interest (and starts delivering) only when a subscription on the deliver subject for a queue subscription matching the “deliver group” name is found. Since the JetStream consumer is thought to be a non-deliver-group consumer, the opposite happens: the server detects an core NATS queue subscription on the “deliver subject”, therefore does not trigger delivery on the JetStream consumer’s “deliver subject”.

The 2 other issues are still present because those checks are done in the updated libraries.

If the above client libraries are update to the latest version, but the NATS Server is still to version prior to v2.4.0 (that is, up to v2.3.4):

• It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is because the JetStream consumer’s information retrieved by the library will not have the PushBound boolean set by the server, therefore will not be able to alert the user that they are trying to create multiple subscription instances for the same JetStream consumer.
• Queue subscriptions will fail because the consumer information returned will not contain the DeliverGroup field. The error will be likely to the effect that the user tries to create a queue subscription to a non-queue JetStream consumer. Note that if the application creates a queue subscription for a non-yet created JetStream consumer, then this call will succeed, however, adding new members or restarting the application with the now existing JetStream consumer will fail.
• Creating queue subscriptions without a named consumer/durable will now result in the library using the queue name as the durable name.
• Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message.

For completeness, using the latest client libraries and NATS Server v2.4.0:

• Trying to start multiple non-queue subscriptions instances for the same JetStream consumer will now return an error to the effect that the user is trying to create a “duplicate subscription”. That is, there is already an active subscription on that JetStream consumer.
  It is now only possible to create a queue group for a JetStream consumer created for that group. The DeliverGroup field will be set by the library or need to be provided when creating the consumer externally.
• Trying to create a queue subscription without a durable nor consumer name results in the library creating/using the queue group as the JetStream consumer’s durable name.
• Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message.

Note that if the server v2.4.0 recovers existing JetStream consumers that were created prior to v2.4.0 (and with older libraries), none of them will have a DeliverGroup, so none of them can be used for queue subscriptions. They will have to be recreated.

Go Version

• 1.16.7: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Domain to the content of a PubAck protocol (#2432, #2434)
  • PushBound boolean in ConsumerInfo to indicate that a push consumer is already bound to an active subscription (#2438)
  • DeliverGroup string in ConsumerConfig to specify which deliver group (or queue group name) the consumer is created for (#2438)
  • Warning log statement in situations where catchup for a stream resulted in an error (#2444)
• Monitoring:
  • Ability for normal accounts to access scoped connz information (#2437)
• Operator option resolver_pinned_accounts to ensure user are signed by certain accounts (#2461)

Changed

• JetStream:
  • ConsumerInfo's Delivered and AckFloor are now SequenceInfo instead of SequencePair. SequenceInfo contains Last which represents the last active time (in UTC) (#2462)
  • Delivery of messages for consumers will now be subject to proper group information. Older clients may fail to receive messages on a queue subscription if the library has not been updated to specify the DeliverGroup (#2438)

Improved

• Make error message actionable when adding operator and leaf nodes. Thanks to @alsuren for the contribution (#2449)
• JetStream:
  • File utilization when using a Jetstream stream as a KeyValue store (#2456)
  • Encryption at rest with message expiration on server restart (#2467)
  • Enable global account on non JetStream servers in mixed mode (#2473)

Fixed

• JetStream:
  • Stream delete can fail for non empty directory (#2418)
  • Possible panic for concurrent stream remove and consumer create (#2419)
  • Simplified flow control and avoid stalls due to message loss (#2425)
  • Consumer info max_msgs_per_subject defaults to 0, but should be -1 (#2426)
  • Creating a consumer with a max_waiting value and a deliver subject was returning the wrong error message (#2427)
  • Assign default to max_ack_pending when AckExplicit or AckAll (#2428)
  • Subscriptions for internal clients for JetStream consumers were not properly removed, resulting in possible subscriptions leak/high memory usage (#2439)
  • Expiration of messages during a server restart could lead to clients reporting errors after reconnect when trying to send new messages (#2452)
  • Added additional checks for failures during filestore encryption (#2453)
  • Processing of a publish ACK could cause a server panic (#2460)
  • Consumer's number of pending messages for multiple matches and merging (#2464)
  • Deadlock due to lock inversion when creating a RAFT group. Thanks to @lokiwins for the report (#2471)
• Leafnode:
  • User authorization issue when JetStream is involved. Thanks to @wchajl for the report (#2430)
  • A remote websocket connection with wss:// scheme but no tls{} configuration block would be attempted as non TLS connection, resulting on an "invalid websocket connection" in the log of the server attempting to create the remote connection (#2442)
  • Wrong permission checks prevented messages flow. Thanks to @rbboulton for the report (#2455, #2470)
  • Daisy chained subject propagation issue. Thanks to @KimonHoffmann for the report (#2468)
• Possible subscription leak with the use of "AutoUnsubscribe" (#2421)
• Prevent JWT claim updates from removing system imports (#2450, #2451)
• Error print when adding back existing system imports (#2466)
• Build on OpenBSD-6.9. Thanks to @miraculli for the contribution (#2472)

Complete Changes

v2.3.4...v2.4.0