improvement: add some SQLite pragma statement settings to improve per…

…formance (#5150)

# Description

Add some additional pragma statement that should improve performance
with SQLite.

This configuration is the same one used by Rails and added in
rails/rails#49349

**Type of change**

- Improvement (change adding some improvement to an existing
functionality)

**How Has This Been Tested**

- [x] Manually tested and added some new tests checking that pragma
statements are correctly configured in the database.

**Checklist**

- I added relevant documentation
- follows the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

argilla-server/CHANGELOG.md

@@ -22,6 +22,10 @@ These are the section headers that we use:
 
 ## [2.0.0rc1](https://github.com/argilla-io/argilla/compare/v1.29.0...v2.0.0rc1)
 
+### Added
+
+- Added some new performance tuning settings for SQLite database. ([#5150](https://github.com/argilla-io/argilla/pull/5150))
+
 ### Changed
 
 - Change `responses` table to delete rows on cascade when a user is deleted. ([#5126](https://github.com/argilla-io/argilla/pull/5126))
@@ -34,6 +38,7 @@ These are the section headers that we use:
 
 - Fixed error when updating records in bulk with wrong `external_id` but correct record `id`. ([#5014](https://github.com/argilla-io/argilla/pull/5014))
 - Fixed error when searching all record response values. ([#5003](https://github.com/argilla-io/argilla/pull/5003))
+- Fixed SQLite connection settings not working correctly due to a outdated conditional. ([#5149](https://github.com/argilla-io/argilla/pull/5149))
 
 ## [1.29.0](https://github.com/argilla-io/argilla/compare/v1.28.0...v1.29.0)
 

argilla-server/src/argilla_server/database.py

@@ -47,7 +47,34 @@
 def set_sqlite_pragma(dbapi_connection, connection_record):
     if isinstance(dbapi_connection, AsyncAdapt_aiosqlite_connection):
         cursor = dbapi_connection.cursor()
+
+        # Enforce foreign key constraints
+        # https://www.sqlite.org/pragma.html#pragma_foreign_keys
+        # https://www.sqlite.org/foreignkeys.html
         cursor.execute("PRAGMA foreign_keys = ON")
+
+        # Journal mode WAL allows for greater concurrency (many readers + one writer)
+        # https://www.sqlite.org/pragma.html#pragma_journal_mode
+        cursor.execute("PRAGMA journal_mode = WAL")
+
+        # Set more relaxed level of database durability
+        # 2 = "FULL" (sync on every write), 1 = "NORMAL" (sync every 1000 written pages) and 0 = "NONE"
+        # https://www.sqlite.org/pragma.html#pragma_synchronous
+        cursor.execute("PRAGMA synchronous = NORMAL")
+
+        # Set the global memory map so all processes can share some data
+        # https://www.sqlite.org/pragma.html#pragma_mmap_size
+        # https://www.sqlite.org/mmap.html
+        cursor.execute("PRAGMA mmap_size = 134217728")  # 128 megabytes
+
+        # Impose a limit on the WAL file to prevent unlimited growth
+        # https://www.sqlite.org/pragma.html#pragma_journal_size_limit
+        cursor.execute("PRAGMA journal_size_limit = 67108864")  # 64 megabytes
+
+        # Set the local connection cache to 2000 pages
+        # https://www.sqlite.org/pragma.html#pragma_cache_size
+        cursor.execute("PRAGMA cache_size = 2000")
+
         cursor.close()
 
 

argilla-server/tests/unit/test_database.py

@@ -26,3 +26,8 @@ async def test_sqlite_pragma_settings(self, db: AsyncSession):
             return
 
         assert (await db.execute(text("PRAGMA foreign_keys"))).scalar() == 1
+        assert (await db.execute(text("PRAGMA journal_mode"))).scalar() == "wal"
+        assert (await db.execute(text("PRAGMA synchronous"))).scalar() == 1
+        assert (await db.execute(text("PRAGMA journal_size_limit"))).scalar() == 67108864
+        assert (await db.execute(text("PRAGMA mmap_size"))).scalar() == 134217728
+        assert (await db.execute(text("PRAGMA cache_size"))).scalar() == 2000