Release notes from KeyDB

KeyDB v6.3.4

2023-10-30T20:26:08Z

KeyDB 6.3.4 Release Notes:

Add keydb_modstatsd, a module for providing keydb stats to a local statsd server. Can be found under src/modules/keydb_modstatsd
Fixed FreeBSD compile(Thanks to @bra-fsn)
Fixed a bug in s3 bucket config that blocked startup(Thanks to Alexandre Chichmanian)
Fixed a bug causing crash if keys command is called after a blocking command (Fixes issues #659, #605, #532)
Added proper error checking of replica configs, now KeyDB will throw an error if replica-of config is passed before active-replica or multimaster configs
Fixed double free bug in lazy free
Added "overload-protect-percent" config, when enabled this will load shed clients whenever CPU usage exceeds configured value
Added "availability-zone" config, this can be passed any string which will then be reported by info command
Updated eviction logic to account for total system memory availability using sysinfo
Fixed a bug where repl-backlog-size config was modified in keydb.conf with the runtime value during config rewrite
Fixed a possible deadlock when running CLIENT KILL with large number of clients
Fixed a bug where KeyDB would overcommit memory during fork BG save
Fixed a bug with disk repl_backlog causing double free
Fixed integer overflow issue in the temp rdb file naming (Thanks to @karthyuom)
Fixed compile issue with GCC 13.1.1 (Thanks to @michieldwitte)
Removed expireset and restored redis expire behaviour
Fixed a bug causing forked processes to hang, specifically affecting RDB and AOF(Fixes issues #675, #619)
Added "CLUSTER REPLICATE NO ONE" to turn a replica into an empty primary
Added RDB-less full sync, can be enabled with config 'enable-keydb-fastsync'

FLASH(Beta):

Added "flash-disable-key-cache" config to disable key cache which stores every key(but not value) in memory
Moved cluster slot to key map to being stored by storage provider, rather than in memory
Moved expires to be stored by storage provider, rather than in memory
Enabled expiry/eviction from storage provider, previously only data stored in memory could be expired/evicted(Fixes #645 along with the 2 previous updates)
Fixed a bug where swapdb result was not recovered after keydb restarts in FLASH mode (Thanks to @karthyuom)
Fixed double free bug when fast sync was canceled early
Fixed a bug where a temp rdb file with zero bytes is generated in flash mode (Thanks to @karthyuom)
Fixed a bug where flash CF options are being reset to default after flushall (Thanks to @karthyuom)(Fixes issue #717)
Updated debug reload command to work with flash

Dockerfile:

Added support for KEYDB_PASSWORD env variable(Thanks to @einar-pexip)
Compile with flash in Dockerfile(Thanks to @der-eismann)
Fix broken redis-cli symlink (Thanks to @pimvandenbroek)
Use tini for alpine docker(Thanks to @rofafor)

KeyDB v6.3.3

2023-05-03T05:11:58Z

Version 6.3.3 Release Notes

To help accelerate our development efforts for KeyDB, this will be the last release containing support for Centos 7, Ubuntu 16.04, Ubuntu 18.04, Debian 9 and 32-bit builds. For more information click here.

This release contains fixes for 17 issues along with improvements to the KeyDB FLASH feature:

Fixed race condition with expireset access (Issue #597)
Fixed keys command with lua, and added keydb as an alternative to redis as lua variable to access db (Issue #562 and #594)
Fixed hang in aof child (Issue #554)
Fixed leaking fds from RDB save (Issue #453 and #584)
Added config for S3 RDB load/save (Issue #584)
Enabled active defrag during forkless background save to improve average memory efficiency (Issue #460)
Fixed crash with fork background save during replication (Issue #567)
Fixed integer overflow in rand family of commands(Issue #631, #632, #633)
Fixed bad value in hincrbyfloat(Issue #634)
Fixed OOM hang in rand family of commands(Issue #635)
Added config to limit count of return values in rand family of commands(Issue #636)

Updates to KeyDB FLASH (Beta):

RocksDB has been updated to v7.9.2
Fixed missing slot_to_key map in FLASH cluster mode (Issue #574)
Added keyspace notifications for keys loaded from FLASH at startup
Fixed race condition in prefetchKeysAsync with FLASH enabled (Issue #571)

KeyDB Version 6.3.2

2023-01-22T20:55:51Z

KeyDB v6.3.2 Release

This release contains Beta level support for KeyDB FLASH, new ASYNC commands, latency improvements and a number of bug fixes.

KeyDB FLASH Support

KeyDB FLASH is included as a Beta feature with this release. Enabling this feature avoids the need to store all data in memory, allowing you to store more data at a lower cost. KeyDB will persist data to the storage medium it is written to avoiding the need for AOF/RDB files. KeyDB uses RocksDB as the persistent storage provider and can be enabled with config storage-provider flash /path/to/flash/output. Read more at https://docs.keydb.dev/docs/flash/

New ASYNC Commands

In addition to GET/MGET support released with v6.3.0, ASYNC support has been added for the following commands: HGET, HMGET, HKEYS, HVALS, HGETALL, HSCAN and can be enabled with config enable-async-commands yes

Jammy & Bookworm Support

Packaging support for Ubuntu 22.04 (Jammy) and Debian 12 (Bookworm) has been included with this release and will be maintained moving forwards. For details on installation please refer to https://docs.keydb.dev/docs/ppa-deb

Other Improvements & Bug Fixes

Added new soft shutdown feature, can be enabled with config "soft-shutdown yes".
If soft shutdown is enabled, instead of shutting down right away, the server will wait until all clients have disconnected, and will reject all new connection attempts.
Fixed memory leak with tls certificates when tls allowlist is enabled
Fixed bug in rdb load with flash enabled to ensure all dbs are safe to load (previously only checked db[0])
Fixed race conditions in rdb load and replication
Fixed memory access of rdb file after it should have been deleted
Fixed integer overflow bug in flash(Issue #486)
Improve TLS latency by queueing new commands before executing instead of after
Removed O(n) count of memory usage from info command(replaced by O(1) estimate)
Improved latency of clearing large number of flash DBs(Thanks to Paul Chen for this fix)(Issue #516)
replaced sprintf with snprintf to avoid potential security bugs
Fixed bug where a failed move due to key already existing in move target would result in the key being removed from move source(Thanks to Paul Chen for this fix)(Issue #497)
Fixed usage of deprecated OpenSSL api in OpenSSL v>3.0.1(Issue #392)
Imported security fixes from Redis (CVE-2023-22458 CVE-2022-35977)
Other fixed issues: #480 #477 #454 #452 #303 #425 #492 #541

KeyDB v6.3.1

2022-06-09T01:34:47Z

KeyDB v6.3.1 Release

This point release contains fixes to bugs related to expires, active-rep, and rdb saving.

The following Issues have been resolved in this release: #419, #422, #428
PRs associated with this release: #426, #429, #431, #433

Please see main commit notes below. For full details please reference PRs

Fix reference counting failure in the dict
Fix assertion in async rehash
Prevent crash on exit by avoiding destructors on exit
Prevent accidental rehash initiation when an async rehash is aborted
Respond to cancelled rehashes earlier to avoid wasted CPU
Add memory fence barriers to the replication backlog
Silence TSAN errors on ustime and mstime
Resolve incorrect locking during repl backlog resize
Make async commands "opt-in" as it can change the percieved ordering across clients
Fix reference counting failure in the dict

KeyDB Version 6.3.0

2022-05-19T18:57:22Z

KeyDB v6.3.0 Release

KeyDB version 6.3.0 is the first open source release to include previously KeyDB Enterprise Features! This release is the culmination of many years of work to move away from KeyDB’s global lock and enable better scaling across cores.

Active Replication PSYNC

Active Replication has been a feature of many KeyDB releases, but always required a full sync while connecting to a new peer. In this release KeyDB now supports partial synchronization to enable fast cluster healing. In addition KeyDB is able to update its replication offset based upon the knowledge of peer nodes, meaning a full sync from one node is enough to permit partial syncs to other nodes within a mesh.

This change greatly reduces the time to add a new node to an active replication cluster. In our testing we’ve seen the time reduced from well over 10 minutes to a minute and a half for a 12 node mesh cluster.

Async Commands

Async commands are commands which can execute without the global lock. This feature must be enabled first in your configuration file by setting “enable-async-commands yes”. When enabling async commands the consistency is modified slightly, in particular writes from other clients may take a configured amount of time to become visible to other clients. This time is configured with the max-snapshot-slip configuration parameter.

The ordering rules with async commands are as follows:
Writes are always totally ordered among clients
Writes are always immediately visible by the client that performed the write

For most applications the slight modification to the consistency model of Active Replication will not be visible. If your application relies on ordering between different clients this feature is disabled by default. In addition a new command called “lfence” is provided for clients that do rely on consistency across clients but are willing to insert fences to assert total ordering at the correct times.

The following commands support async operation:
GET
MGET

Additional read-only commands will get added over time. If a command is of specific interest to you please post an issue to help us prioritize your use case.

Async Rehash

KeyDB relies upon a spinlock to synchronize threads. In prior versions of KeyDB the CPU time consumed while waiting for the lock to be acquired was wasted. KeyDB now has the ability to perform rehash during this time using otherwise wasted CPU time for a more useful purpose. In many cases this can almost completely hide the overhead of rehashing. No configuration is required to enable this functionality, it operates by default.

In Process Background Saving

Historically Redis has used the “fork” system call to create a new process which performed the background save. This made the code easier to follow as the kernel handled the hard task of copy-on-write of memory while background saving was in progress. However the downside to this approach is that it is not possible to accurately estimate the total amount of memory required including background saving.

KeyDB now uses a new “snapshot” system to create snapshots of the database at a specific period and copy-on-write new changes within the process. As a result the maxmemory setting is now a global setting and will include memory used for background save.

For backwards compatibility the semantics of maxmemory are modified slightly during a background save. While a background save is in progress KeyDB will permit memory consumption to exceed maxmemory by up to 20%. This is to simulate the old behavior of background save memory consumption not being counted towards the maxmemory setting while permitting easy calculation of a total upper bound for memory usage.

IStorage Interface

This release does not yet include KeyDB’s FLASH feature based upon RocksDB. However the release does include KeyDB’s IStorage interface which is the backbone of our persistent storage feature. By implementing this interface for your custom storage solution you will enable all KeyDB features.

Thanks To Our Contributors:

Thank you to everyone who contributed to KeyDB with bug reports, design and testing:

Ben Schermel
Firaenix: Bug Report #393 - Crash in sorted sets with long names
Kerog: #401 Error after include *.confg
Gvsafronov: Fedora 35 Compile Failure
Talkabout: Always providing helpful and detailed bug reports, not limited to #378
Inakisoriamrf: #383 crash
Alebcay: #384 macOS build break
Server2245: #379 Install issue on RHEL7
Antiarchitect: #380, and many others
Tchernomax: #352 missing support for systemd
Heng Kuang: Detailed bug reports and design suggestions
Kajaruban Surendran: Detailed performance analysis of async rehash and other features
Paul Chen: Your enthusiasm and willingness to help debug issues has been deeply appreciated.

Special thanks to Huawei Canada Research Team for your detailed and in-depth bug reports and feature requests and design participation.

Thanks to the EQ Alpha Team that made this release possible: Malavan Sotheeswaran, Vivek Saini, Christian Legge, Peter Liang, Ben Schermel, John Sully

KeyDB v6.2.2

2022-01-19T20:21:31Z

Changes:

Acquire lock in module.cpp to fix module test break
Fix usage of nanosleep() on Apple hardware
Fix networking deadlock (#183)
Remove redundant zfree() in keydb-benchmark
Added a read/write lock to fix forking issues

Fixed Issues:
#378 #384

KeyDB v6.2.1

2021-11-19T17:13:43Z

This release of KeyDB is at parity with Redis 6.2.6. In addition, the following changes were included:

Systemd support for release packages
Fixes to buffer overflows in the BITOP SHIFT and CRON commands
Server times are now computed on a seperate thread to improve performance
Now enforces syslog identity and facility as soon as possible
Removed erroneous use of LOG_... flags when using syslog
Fixed erroneous #endif leading to build errors on some platforms
Fixed the incorrect counting of client connections

Issues resolved:
#355, #370

KeyDB v6.2.0

2021-08-11T23:24:38Z

Changes:

Removed unused command line options(-a, -d, -P, -r, -q, --csv, -l, -I, -e, --precision, --cluster, --enable-tracking)
--ms option changed to --time, added --clients, --host, --port, and --threads options
Added keydb-diagnostic-tool
Improved test reliability
Fixed memory leak in keydb-benchmark
Fixed a number of race conditions
Fixed OSX build issues
Feature parity with Redis 6.2.3

Fixed Issues:
#323 #325 #276

KeyDB 6.0.18

2021-08-09T18:19:10Z

Changes:

Fixed memory leak in MVCC
Improvements to internal timers
Reduced redundant synchronization
Improved test reliability
Added config option for loading callback interval keys
Added ascii logo image!
Added config to disable message of the day
Improved multithreaded replication performance
Feature parity with redis 6.0.10

Fixed Issues:
#300 #285 #273 #260 #257 #238

KeyDB 6.0.16

2020-09-28T20:02:05Z

This release has the following enhancements:

Reduced memory consumption when Active Replication is not used
Reduced CPU consumption
Improved TLS performance
replica-quorum config for multi-master.
Intended to be used with "replica-serve-stale-data no", if at least N replicas are connected we will consider our master link up and serve data.
Subkey expires performance improvement for big sets

The following issues were fixed:
#234 #236 #233 #207 #229 #231