Release v2.9.21

Changelog

Go Version

• 1.19.12

Dependencies

• github.com/klauspost/compress v1.16.7
• github.com/nats-io/nats.go v1.28.0
• go.uber.org/automaxprocs v1.5.3
• golang.org/x/crypto v0.11.0
• golang.org/x/sys v0.10.0

Added

OCSP

• Add fetch, cache, and verification of client CA's OCSP Response for NATS, WebSocket, and MQTT client mTLS connections (#4362, backported from 2.10)
• Add bi-directional fetch, cache, and verification of CA OCSP Response for LEAF connections (#4362, backported from 2.10)

See ADR-38 OCSP Peer Verification

General

• Add UTC log timestamp option (#4331, backported from 2.10)

Improved

JetStream

• Don't error to server logs if message was deleted for consumer (#4328)
• Improve publish performance for zero-interest subjects (#4359) Thanks to @antlad for reporting the issue!
• Sync and reset message rejected count to ensure replicas don’t incorrectly discard messages (#4365, #4366)

Fixed

General

• Leaking memory on usage of getHash() (#4329) Thanks to @VuongUranus for reporting the issue!
• Server reload with highly active accounts and service imports could cause panic or dataloss (#4327)
• Fix detection of an unusable configuration file (#4358)
  • NOTE: as a side effect of this fix, the server will no longer startup with an empty config file
• Fix a few system service imports going missing after configuration reload (#4360)

OCSP

• Fix local-determination of issuer CA at startup (#4362)
• Remove constraint that all (super)cluster node peers must be issued by the same CA (#4362)

Embedded

• Don't require TLS for in-process client connection (#4323)

JetStream

• Fix serializability guarantee for concurrent publish when using expected-last-subject-sequence (#4319)
• Report correct consumer count in paged list response (#4339)
• Fix not validating single token filtered consumer (#4338)
• Fix stream recovery of message block with sequence gaps (#4344)
• Fix panic when re-calculating first sequence of SimpleState info (#4346)
• Fix stream store accounting drift (#4357)

Complete Changes

v2.9.20...v2.9.21

Release v2.9.20

Changelog

Go Version

• 1.19.11

Added

Windows

• Backport 2.10 support for native Windows certificate store (#4268)

Improved

Accounts

• Allow advisories to be exported/imported across accounts (#4302)

JetStream

• Optimize consumer create time on streams with a large number of blocks (#4269)

Fixed

Gateways

• Protect possible data race when reloading accounts (#4274)

Leafnodes

• Prevent zombie subscriptions which could lead to silent data loss when using queue subscriptions (#4299)

WebSocket

• Prevent reporting tls_required when tls_available is not set (#4264)

JetStream

• Prevent corrupting streams actively being restored during health check (#4277) Thank you @vitush93 for the report!
• Prevent encrypted data attempting to be decrypted with an empty key (#4301)

MQTT

• Ensure republished messages from streams are received by MQTT subscriptions (#4303)

Complete Changes

v2.9.19...v2.9.20

Release v2.9.19

Changelog

Go Version

• 1.19.10

Improved

JetStream

• Improve resource utilization when creating mirrors on very high-sequence streams (#4249)

Fixed

WebSocket

• Ensure INFO properties are populated based on the WebSocket listener when enabled (#4255) Thanks to @Envek for reporting the issue!

Complete Changes

v2.9.18...v2.9.19

Release v2.9.18

Changelog

Go Version

• 1.19.10

Dependency Updates

• golang.org/x/crypto v0.9.0 (#4236)
• golang.org/x/sys v0.8.0 (#4236)
• github.com/nats-io/nats.go v1.27.0 (#4239)

Improved

Monitoring

• Optimize /statsz locking and sending in standalone mode (#4235)

JetStream

• Apply ack floor check only for interest-based streams (#4206)
• Improved efficiency and reduced CPU usage of the consumer ack floor check, particularly when the stream first sequence is a large number (#4226)
• Improve clean-up phase of R1 consumers on server restart for name reuse (#4216)
• Optimize “last message lookups” by subject (KV get operations) for small messages (#4232) Thanks to @jjthiessen for reporting the issue!
• Only enable JetStream account updates in clustered mode (#4233) Thanks to @tpihl for reporting the issue!

Fixed

General

• Fix a variety of potential panic scenarios (#4214) Thanks to @Zamony and @artemseleznev for the contribution!

Leadnode

• Daisy chained leafnodes could have unreliable interest propagation (#4207)
• Properly distribute requests to queue groups across leafnodes (#4231)

JetStream

• Killed server on restart could render encrypted stream unrecoverable (#4210) Thanks to @BhatheyBalaji for the report!
• Fix a few data races detected internal testing (#4211)
• Process extended purge operations correctly when being replayed (#4212, #4213) Thanks to @MauriceVanVeen for the report and contribution!

Complete Changes

v2.9.17...v2.9.18

Release v2.9.17

Changelog

Go Version

• 1.19.9

Dependency Updates

• github.com/klauspost/compress v1.16.5 (#4088)

Improved

Core

• Additional optimizations to outbound queues, reducing memory footprint (#4084, #4093, #4139)
• Use faster flate compression library for WebSocket transport compression (#4087)

Leafnodes

• Optimize subscription interest propagation for large leafnode fleet (#4117, #4135)

Monitoring

• Support sorting by RTT for /connz (#4157)

Resolver

• Improve signaling for missing account lookups (#4151)

JetStream

• Optimized determining if a stream snapshot is required (#4074)
• Run periodic check for consumer “ack floor” drift on leader (#4086)
• Optimize leadership transfer during a stream migration (#4104)
• Improve how clustered consumer state is hydrated on startup (#4107)
• Add operation type to panic messages for improved debugging (#4108)
• Improve health check to repair stalled assets periodically (#4116, #4172)
• Remove unnecessary filestore lock to improve I/O performance (#4123)
• Various Raft leadership improvements (#4126, #4142, #4143, #4145)
• Improve accuracy of account usage (#4131)
• Clean up old Raft groups when streams are reset (#4177)

Fixed

General

• Fix various names in comments (#4099) Thanks to @cuishuang for the contribution!
• Fix various typos in comments (#4169) Thanks to @savion1024 for the contribution!
• Update tests to reflect the server.Start() call no longer blocks (#4111) Thanks to @lheiskan for reporting the issue!
• Fix race condition in config reload with gateway sublist check (#4127)
• Track all remote servers in a NATS system with different domains (#4159)

Core

• Fix premature closing in WebSocket transport due to outbound queue changes (#4084)
• Fix subscription interest for config-based accounts during config reload (#4130)
• Use monotonic time for measuring durations internally (#4132, #4154, #4163)

Monitoring

• Service import reporting for /accountz when mapping to local subjects (#4158)

JetStream

• Fix formatting of Raft debug log (#4090)
• Prevent failure of /healthz in single server mode on failed snapshot restore (#4100)
• Ensure a stream Raft node has fully stopped and resources freed (#4118)
• Fix case where R1 streams are orphaned and can’t scale up (#4146)
• Protect against out of bounds access on usage updates (#4164)
• Fix state rebuild where the first block is truncated and missing index info (#4166)
• Avoid stale KV reads on server restarted for replicated stores (#4171) Thanks to @yixinin for reporting the issue!
• Prevent deadlock with usage report for accounts (#4176)

Complete Changes

v2.9.16...v2.9.17

Release v2.9.16

Changelog

Go Version

• 1.19.8

Dependency Updates

• github.com/klauspost/compress v1.16.4
• github.com/nats-io/jwt/v2 v2.4.1
• github.com/nats-io/nkeys v0.4.4
• golang.org/x/crypto v0.8.0
• golang.org/x/sys v0.7.0

Added

Build

• Nightly build of the “main” branch as a Docker image: synadia/nats-server:nightly-main (#3961, #3962, #3963, #3972, #4019, #4063)
• Version control SHA in the Goreleaser build of the server (#3993). Thanks for the report @jzhoucliqr!
  Monitoring
• Add server name and route remote server name to /routez (#4054)

Resolver

• Add “hard_delete” option for stored JWTs (#3783). Thanks for the contribution @JulienVdG!

Improved

JetStream

• Storage and Raft layer improvements to decrease p99 latency variance and memory pressure in high load environments (#3956, #3952, #3965, #3981, #3999, #4018, #4020, #4021, #4022, #4026, #4027, #4028, #4029, #4030, #4038, #4045, #4050, #4053)
• Don’t show Raft warning for node that is closed (#3968)
• Use pooled buffer for flushing encrypted message blocks (#3975)
• Remove snapshotting of cores and maxprocs (#3979)
• Improvements to interest-based streams to optimize when messages are deleted (#4006, #4007, #4012)
• Better handling of concurrent stream and consumer creation of the same name (#4013)
• Finer-grain locking during asset checking to reduce contention in the /healthz endpoint (#4031)
• Encrypted file stores will now limit block sizes to improve performance (#4046)
• Improve performance on storing messages with varying subjects and limits are imposed (#4048, #4057). Thanks for the report @kung-foo!

Fixed

Subjects

• Ensure subjects containing percent (%) are escaped (#4040)

Accounts

• Fix data race when setting up service import subscriptions (#4068)

Leaf

• Fix leaf client connection failing on OCSP setups (#3964)
• Fix case when allow/deny permissions on leaf connection could block legitimate interest (#4032)

Cluster

• Route disconnect not detected by ping/pong (#4016). Thanks for the contribution @sandykellagher!

JetStream

• Pull consumer not sending timeout error to clients for expired requests (#3942)
• Prevent meta leader deadlock during deletion of orphaned streams during server startup (#3945)
• Clear ack’ed messages when scaling workqueue or interest-based streams (#3960) Thanks for the report @Kaarel!
• Remove messages from interest-based stream on consumer snapshot (#3970)
• Fix potential panic in message block buffer pool (#3978)
• Fixed an issue with consumer states growing and causing instability (#3980)
• Improve handling of out-of-storage condition (#3985)
• Address memory leak of unreachable Raft groups when JetStream becomes disabled (#3986)
• Prevent Raft leader from being placed on server in lame-duck mode (#4002)
• Remove potential race condition on sysRequest (#4017)
• Fix FirstSeq not being updated with filestore when purging subject (#4041). Thanks for the contribution @MauriceVanVeen!
• Fix Raft log debug reloading (#4047)
• Ensure consumer recovers fully on restart before being eligible for leader (#4049)
• Fix incorrect check between stream source/mirror with external streams (#4052)
• Fix various conditions during Raft group recovery (#4056, #4058)

Complete Changes

v2.9.15...v2.9.16

Release v2.9.15

Changelog

Go Version

• 1.19.6: Both the release executables and Docker images are built with this Go release

Added

• Monitoring
  • Add raft query parameter to /jsz to include group info (#3915)
  • Update /leafz to include leaf node remove server name and “spoke” flag (#3923, #3925)

Changed

• Lower default value of jetstream.max_outstanding_catchup to prevent slow consumers between routes (#3922)
  • Note: The new value is now 64MB from 128MB. This is better optimized for 1 Gbit links, however if your links are 10 Gbit or higher, this value can be set back to 128MB if slow consumers were not previously observed.

Improved

• Refactor intra-process queue to reduce allocations (#3894)
• JetStream
  • Better system stability and recovery from corrupt metadata due to hard forced restarts (#3934)
  • Optimize on-disk, per-subject info update (#3867)
  • Limit concurrent blocking IO to improve stability under heavy IO loads (#3867)
  • Improve message expiry calculation for large numbers of messages (#3867)
  • Optimize when and how consumer num pending is calculated, significantly speeding up consumer info requests (#3877)
  • Improve parallel consumer creation to prevent dropped messages (#3880)
  • Properly warn on consumer state update state failures (#3892)
  • Performance of consumer creation for certain configurations (#3901)
  • Send current snapshot to followers when becoming meta-leader (#3904)
  • Ensure preferred peer during stepdown is healthy (#3905)
  • Optimized various store calls on stream state (#3910)
  • Various performance and stability under heavy IO loads (#3922) (Thank you @matkam and @davidzhao for the report and the test harness!)

Fixed

• Fix stack overflow panic in reverse entry check when inbox ends with wildcard (#3862)
• Check if client connection name was already set when storing, preventing recursive memory growth (#3886)
• Fix check for count of wildcard tokens in “partition” subject transform (#3887) (Thank you @MauriceVanVeen for the contribution!)
• Fix panic if service export is nil (#3917) (Thank you @MauriceVanVeen for the report!)
• JetStream
  • Ensure per-subject info is updated when doing stream compact (#3860)
  • Ensure account usage is updated in the filestore when extended version purge occurs (#3876)
  • Prevent consumer deletes on restart, with non-fatal errors (#3881)
  • Do not warn if consumer replicas is zero since it will be inherited from the stream (#3882)
  • Named push consumers with inactive thresholds deleted when still active (#3884)
  • Prevent spurious “Error storing entry to WAL” log messages (#3893, #3891)
  • Clean up consumer redelivery state on stream purge (#3892)
  • Clean up consumer ack pending if below stream ack floor (#3892)
  • Update ack floors on messages being expired (#3892)
  • Fix lost ack pending consumer state on partial stream purge (#3892)
  • Snapshot and compact the consumer RAFT WAL, even when state changes do not occur, to prevent excessive disk usage (#3898)
  • Fix KV accounting errors under heavy concurrent usage (#3900)
  • Ensure new replicas respect MaxAge when a stream is scaled up (#3861)
  • Snapshots would not compact after being applied (#3907)
  • Fix filtered pending state calculation (#3910)
  • Recover from a failed truncate on raft WAL (#3911)
  • Fix JWT claims update if headers are passed (#3918)
• MQTT
  • Prevent use of wildcard characters with topics beginning with $, per the MQTT spec violation 4.7.2-1 (#3926) (Thank you @dominikh for the report!)

Dependency Updates

• klauspost/compress - v1.16.0
• nats-io/nats.go - v1.24.0
• golang.org/x/crypto - v0.6.0
• golang.org/x/sys - v0.5.0

Complete Changes

v2.9.14...v2.9.15

Release v2.9.14

Changelog

Go Version

• 1.19.5: Both the release executables and Docker images are built with this Go release

Fixed

• JetStream
  • Fix circumstance when an empty snapshot could be written (#3844)
  • Fix possible panic and deadlock during a consumer filter subject update (#3845)
  • Fix consumer snapshot logic (#3846)

Complete Changes

v2.9.12...v2.9.14

Release v2.9.12

Changelog

NOTE: regressions were found in this release. Please skip this and go directly to the v2.9.14 release.

Go Version

• 1.19.5: Both the release executables and Docker images are built with this Go release

Added

• OS/Arch
  • Add support for dragonfly BSD (#3804)

Improved

• JetStream
  • Use highwayhash to optimize difference tracking for stream, consumer, and cluster snapshots (#3780)
  • Add small tolerance in lag for stream and consumer health checks (#3794)
  • Various optimizations related to snapshots and memory usage (#3828, #3831, #3837) Thanks to @MauriceVanVeen for the collaboration on this issue.

Fixed

• JetStream
  • Update numCores and maxProcs if altered by container limits (#3796)
  • Fix filtered state for all subjects when the first sequences are deleted (#3801)
  • Updating a stream to direct gets would fail direct gets (#3806)
  • Force consumer replicas to match for interest-based policy streams (#3817)
  • Assign signal subscription to consumer when created (#3808)
  • Properly process updates on a stream on restart (#3818)
  • Select consumer peer(s) from active/online peers only on creation (#3821)
  • Sourced streams that do not overlap subjects were incorrectly reported as a cycle (#3825)
  • Fix for isGroupLeaderless when JS not available due to shutdown (#3830)
  • Deadlock on data loss when holding mb lock (#3832)
  • Fix consumer not getting messages after filter update (#3829)
  • Fix current consumers not getting messages after purge (#3838) Thanks to @pcsegal for the report!

Updated Dependencies

• github.com/klauspost/compress - v1.15.15
• github.com/nats-io/nats.go - v1.23.0
• golang.org/x/time - v0.3.0

Complete Changes

v2.9.11...v2.9.12

Release v2.9.11

Changelog

Go Version

• 1.19.4: Both the release executables and Docker images are built with this Go release

Improved

• Remove duplicate subscription point to reduce memory usage (#3768)

Updated

• Update golang.org/x/crypto dependency to v0.5.0 (#3761). Thanks to @friedrichwilken for the contribution.

Fixed

• JetStream
  • Allow recovery of stream meta file if not properly flushed prior to restart (#3752)
  • Ensure max bytes are honored when message block is not properly flushed (#3757). Thanks to @containerpope for the report.
  • Ensure new consumers assigned to an offline server are properly restored from the metaleader's snapshot (#3760)
  • Set and clear observer state for servers with disconnected leaf nodes (#3763)
• Leaf node
  • Fix duplicate messages when leaf node binds an account with service imports (#3751). Thanks to @asambres-form3 and @maxarndt for the original reports.
• Monitoring
  • Add back existing HealthzOptions.JSEnabled field to restore backwards compatibility (#3744)

Complete Changes

v2.9.10...v2.9.11