Skip to content

v4.1.0
Compare
Choose a tag to compare
@github-actions github-actions released this 01 Nov 17:54
· 765 commits to master since this release
v4.1.0
4f5a6db

⭐ New APIs

In-memory Backend

You can now open ArcticDB with an in-memory backend,

from arcticdb import Arctic
ac = Arctic("mem://")
ac.create_library("test")
assert ac.list_libraries() == ["test"]
# Create libraries as normal. Each `Arctic` object manages its own in-memory storage, so the lifetime
# of your libraries and data is the same as the lifetime of the `Arctic` instance that owns them.

ac2 = Arctic("mem://")
assert ac2.list_libraries() == []  # ac2 is backed by different memory to ac so the "test" library is not returned

Query Builder

We now support a new "count" aggregator. You can invoke it with:

q = QueryBuilder()
q = q.groupby("grouping_column").agg({"a": "count"})

⚠️ Breaking and API Changes

LMDB Backend

This release includes a fix for issue #850 : Ensure that LMDB libraries are readable after being moved to a different location. The fix means that LMDB libraries created with arcticdb>=4.1.0 will not be readable by older clients and those clients must update.

This is because the fix stops us from serializing the LMDB library path (instead we always prefer the one in the Arctic URI), but older clients still expect to see the LMDB path serialized. Older clients reading a new LMDB library will in fact ignore the path passed in to the Arctic constructor and instead read the current working directory.

When you exceed the LMDB map size, we now raise a custom exception arcticdb.exceptions.LmdbMapFullError that explains how to re-open LMDB with a larger map size, whereas previously we raised a less helpful arcticdb.exceptions.InternalException.

🚀 Features

  • Support count aggregator with groupby (#948)
  • Warning for LMDB when two Arctic instances open over the same storage (#1000)
  • Small LMDB Fixes: 2GiB map size for Windows, Validation before delete (#918)
  • Custom exception when LMDB map is full (#1006)
  • Memory backed API (#860)
  • Allow ampersand in symbol names (#900)
  • Add querybuilder notebook demo into the docs (#875)
  • Extended testing against real cloud storages (#789)
  • ASV Benchmarking published here (#962 #970)
  • Preparatory work for RocksDB backend (#945)

🐛 Fixes

  • Fix LZ4 decoding error issues that occurred with a mix of empty and non-empty columns (#964)
  • Performance improvement for read_batch when called with many symbols (4-5x improvement) (#870)
  • Convert semimap to switch which has resolved some segmentation fault issues (#912)
  • Upgrade cUrl to 8.4.0 (#977)
  • Cache open libraries in the LibraryManager (#990)
  • Enhancement 914: Improve error messaging when string column encoding fails due to the presence of a non-string object (#933)
  • Fix storage lock mutex implementation (#966)
  • Make azure sdk stick to winhttp if possible (#851)
  • Extra update checks (#539)
  • maint: Indicate the non-support of PyArrow (#882)
  • Add pymongo to the list of install dependencies (#891)
  • conda-build: Run python tests for macos-latest (#873)
  • fix: Change comparison in test_hypothesis_{sum,mean}_agg (#931)
Uncategorized
  • Fixes IFNDR issue due to mismatching inline/non-inline functions depending on the translation unit (fixes #943) (#949)
  • docs: Post 4.0.0 release documentation (#967)
  • Remove Black and pre-commit setup (#972)
  • Improve string writing performance (#969)
  • Skip docs in build.yml (#984)
  • Move api docs from sphinx to mkdocs (#897)
  • Document support for Mac on intel (#982)
  • Update PAT for publish to master (#988)
  • Fix ccache's non-existence exiting the workflow (#996)
  • Refactor get_descriptions lib methods to be more consistent (#994)
  • Fail docs build on sphix failure (#883)
  • docs: Better document publishing release candidates on conda-forge (#901)
  • conda-build: Unpin some dependencies (#888)
  • docs: Reword conda-forge section mentioning libevent-2.1.10 (#905)
  • Update readme to reflect supported/beta status of Windows PyPi/MacOS conda-forge builds (#907)
  • Skip flaky Mac test (#924)
  • docs: Improve section "Building using mamba and conda-forge" (#917)
  • fix interface in ManualClockVersionStore getter (#925)
  • docs: Add high-level documentation of abstractions (#628)
  • Update storage compatibility (#916)
  • Update copyright notice (#939)
  • Use new get_library argument in ArcticDB_demo_lmdb.ipynb (#930)

The wheels are on Pypi. Below are for debugging:

Assets 3
Loading