V1.2.0

Add: Radio browser : data structures and stations local database cache (based on sqlite database).
Improvement: Memory optimization and performance acceleration for cache options: catFile, catMemStr.

README.md

@@ -32,7 +32,9 @@ YTuner supports :
 * Optional SSL support for YTuner HTTPS web request.
 * Radio stations logo images conversion/resize on the fly with couple of supported image formats (JPG,PNG,GIF,TIFF-optional) 
 * Radio stations logo images optional cache.
-* Radio browser UUIDs, data structures and stations cache (based on files or RAM storage) with optional auto refresh.
+* Radio-browser extensive caching with many options and auto refresh
+  * UUIDs, data structures and stations cache (based on files or RAM storage).
+  * Local DB based full cache of Radio-browser resources.
 * Radio browser advanced filtering and sorting (single config for many AVRs or dedicated configs for each AVR (if you own more then one))
 
 YTuner also has build in :
@@ -60,7 +62,7 @@ Now, the list of supported **and tested** devices below is short, but I hope it
   * Denon AVR-X3300W (Tested by [citronalco](https://github.com/citronalco). Thank you.)
 
 ## Installation
-YTuner is a standalone application and in most cases it does not require additional services, frameworks, packages, virtual machines, libraries or tools to run properly (except optional OpenSSL libraries).
+YTuner is a standalone application and in most cases it does not require additional services, frameworks, packages, virtual machines, libraries or tools to run properly (except optional OpenSSL and/or SQLite3 libraries).
 You can download from [Releases](https://github.com/coffeegreg/YTuner/releases) a file specific to your operating system and CPU architecture or build YTuner from source (look at [Build](README.md#build) section).
 
 After download (or build) save and extract files into prepared directory with granted read/write/execute privileges.
@@ -79,31 +81,41 @@ Now, you should have directory with some of the following subdirectories and fil
  |-- cache (subdir for cache files)
    |-- rbuuids.txt (Radio browser UUIDs cache file)
    |-- ...... (other cache files)
+ |-- db (subdir for databse cache file)
+   |-- rb.db (Radio browser database cache file)
  |-- ytuner (or ytuner.exe for Windows)
  |-- ytuner.ini (YTuner important config file)
 ```
  Do not forget to add execute privileges to `ytuner` on linux/*nix systems with a command like `chmod +x ytuner`.  
 
 
-### OpenSSL
+### OpenSSL (optional)
 If you want to use SSL to support YTuner HTTPS web request you have to get OpenSSL libraries.
 - Most linux/*nix systems install OpenSSL by default. Otherwise, use your favorite package manager to get OpenSSL libraries or download them from [Github](https://github.com/openssl/openssl) or visit [OpenSSL Wiki](https://wiki.openssl.org/index.php/Binaries) for binary distributions source.   
 - Windows users can download them from [Github](https://github.com/openssl/openssl) (follow [NOTES-WINDOWS.md](https://github.com/openssl/openssl/blob/master/NOTES-WINDOWS.md) instructions) or visit [OpenSSL Wiki](https://wiki.openssl.org/index.php/Binaries) for binary distributions source.
 Make sure to get/build the correct version of the OpenSSL libraries with the correct bit length for your OS. 32-bit libraries are needed if you chose to use the 32-bit version of YTuner or 64-bit for the AMD64/x86_64 version of YTuner.
 Finally, you should have 2 files:
   * OpenSSL 1.0.2 and earlier:
      + `ssleay32.dll` (or `libssl32.dll`) and `libeay32.dll`
-  * OpenSSL 1.1.0 and above:
+  * OpenSSL 1.1.x:
      + 64-bit: `libssl-1_1-x64.dll` and `libcrypto-1_1-x64.dll`
      + 32-bit: `libssl-1_1.dll` and `libcrypto-1_1.dll`
-  * OpenSSL 3.0.0 and above:
+  * OpenSSL 3.x.x:
      + 64-bit: `libssl-3-x64.dll` and `libcrypto-3-x64.dll`
      + 32-bit: `libssl-3.dll` and `libcrypto-3.dll`
 
 and place them in your `ytuner` directory or anywhere in your system `PATH`.
 Make sure your system has valid CA certificates.
 >Tip: The YTuner should work with LibreSSL libraries as well.
 
+### SQLite3 (optional)
+If you want to forget about potential connection problems with `Radio-browser.info` while using YTuner and listening to your favorite stations, use one of the options `[catDB, catMemDB, catPermMemDB]` of the `RBCacheType` parameter in the ytuner.ini file to download the full contents of the `Radio-browser.info` resources once and store it in your local SQLite3 database.
+Of course, only data that is useful for YTuner and AVR devices is downloaded and stored locally.
+Due to the use of the very popular SQLite database, YTuner will need to use the library provided by the SQLite development team.
+>! Important ! : Minimal version of SQLite library is 3.33.0 (2020-08-14)
+
+> If you faced problems with the SQLite library, read [this](doc/SQLITE.md) description.
+
 ## Configuration
 
 Your YTuner machine and AVR(s) have to have internet access. Make sure your firewall is properly configured if necessary.
@@ -181,27 +193,12 @@ If you credentials meet all requirements mentioned above just go to your ytuner
 ```
 $ sudo ./ytuner
 ```
-If you `MessageInfoLevel` parameter from `ytuner.ini` has a value greater then 0 and Radio-browser.info with stations list local file support are enabled, you should see something like this :
-```
-2023-04-25 20:36:38 : Inf : Starting services...
-2023-04-25 20:36:38 : Inf : Getting Radio-browser.info UUIDs...
-2023-04-25 20:36:38 : Inf : Successfully loaded 10 my stations.
-2023-04-25 20:36:38 : Inf : DNS server listening on: 192.168.1.2:53.
-2023-04-25 20:36:38 : Inf : Web server listening on: 192.168.1.2:80.
-2023-04-25 20:36:42 : Inf : Successfully downloaded 37542 RB UUIDs.
-2023-04-25 20:36:42 : Inf : Successfully saved 37542 RB UUIDs to cache file.
-```
-Now you can see Radio-browser.info cache file `rbuuids.txt` in your ytuner directory.  
-If you have enabled the radio station icon cache `IconCache=1` and started browsing radio stations with your AVR, you can see the `cache` subdirectory and the files inside.  
-If you AVR support bookmarks and you add some station(s) you should see bookmark file(s).  
-It is possible to add or remove stations from one AVR and use this bookmark as a catalogue in another AVR (even without bookmark support).
-
 ## Build
 You can use [CodeTyphon Studio](https://www.pilotlogic.com) or [Lazarus Free Pascal RAD IDE](https://www.lazarus-ide.org/) to build YTuner.  
 Use the latest versions of these IDE. Relevant project files are included.
 
 ### Dependencies
-YTuner uses [Indy - Internet Direct](https://github.com/IndySockets/Indy) library to build its own binary files. Of course, YTuner binaries no longer need any additional libraries beyond the optional OpenSSL.
+YTuner uses [Indy - Internet Direct](https://github.com/IndySockets/Indy) library to build its own binary files. Of course, YTuner binaries no longer need any additional libraries beyond the optional OpenSSL and/or SQLite3.
 >Important: Use the latest version of Indy library to build YTuner.
 
 ## Summary

cfg/ytuner.ini

@@ -1,6 +1,6 @@
 [Configuration]
 ; Please do not change this value !
-INIVersion=1.1.0
+INIVersion=1.2.0
 
 ;Force the use of the selected IP address of one of the existing interfaces. 
 ;Leave this parameter with "default" value when you have one IP address only or you are not sure what to do.
@@ -10,7 +10,7 @@ IPAddress=default
 ;Use SSL. 1-Yes 0-No ; Default = 1
 UseSSL=1
 
-;Force HTTP code for redirect. It may be useful for PLAY endpoint. Default = 302
+;Force HTTP code for redirect. It can be useful for PLAY endpoint. Default = 302
 RedirectHTTPCode=302
 
 ;Log message level. 0-None, 1-Info, 2-Warning, 3-Error, 4-Debug ; Default = 0
@@ -22,11 +22,11 @@ IconSize=200
 ;Cache icon files. 1-Yes 0-No ; Default = 1
 IconCache=1
 
-;Sixteen characters string with your token. Allowed characters: 0-9 and A-F; Default=0123456789ABCDEF
-;Use with care. Changing this parameter also changes the name of the AVR bookmark file (unless you are using a common bookmark.xml file).
+;Sixteen characters string with your token. Allowed characters: 0-9 and A-F/a-f; Default=0123456789ABCDEF
+;Use with care. Changing this parameter also changes the name of the AVR bookmark file(s) (unless you are using a common bookmark.xml file) and not-common filtering/configuration options ini file(s).
 MyToken=0123456789ABCDEF
 
-;One common configuration ini file (avr.ini) for all AVR devices. Otherwise, each AVR will own its own configuration ini file. 1-Yes 0-No ; Default = 0
+;One common configuration ini file (avr.ini) for all AVR devices. Otherwise, each AVR will own its own filtering/configuration ini file. 1-Yes 0-No ; Default = 0
 CommonAVRini=1
 
 ;Cache folder location. Empty or "default" = Same as YTuner app location.
@@ -37,6 +37,23 @@ CacheFolderLocation=default
 ;Config folder named "config" contains bookmark files and AVRs config files.
 ConfigFolderLocation=default
 
+;DB folder location. Empty or "default" = Same as YTuner app location.
+;DB folder named "db" contains db files.
+DBFolderLocation=default
+
+;SQLite3 library file (with full path). Empty or "default" = YTuner will try to find DB library file.
+;Applicable when RBCacheType is one of: catDB, catMemDB, catPermMemDB. 
+;Windows users can get latest sqlite3.dll driver from https://www.sqlite.org/download.html . Please make sure to download the correct 64-bit or 32-bit version for your system.
+;Linux / Unix (Solaris, BSD) / macOS users should have such a driver already installed by default. Please make sure you specify the path to the correct 64-bit or 32-bit library file..
+;Typicaly:
+;Linux64 : /usr/lib64/libsqlite3.so, /usr/lib64/libsqlite3.so.0, /usr/lib/x86_64-linux-gnu/libsqlite3.so, /usr/lib/x86_64-linux-gnu/libsqlite3.so.0, /usr/lib/aarch64-linux-gnu/libsqlite3.so, /usr/lib/aarch64-linux-gnu/libsqlite3.so.0
+;Linux32 : /usr/lib/libsqlite3.so, /usr/lib/libsqlite3.so.0, /usr/lib/i386-linux-gnu/libsqlite3.so, /usr/lib/i386-linux-gnu/libsqlite3.so.0, /usr/lib/arm-linux-gnueabi/libsqlite3.so, /usr/lib/arm-linux-gnueabi/libsqlite3.so.0, , /usr/lib/arm-linux-gnueabihf/libsqlite3.so, /usr/lib/arm-linux-gnueabihf/libsqlite3.so.0
+;Solaris64 : /usr/lib/64/libsqlite3.so, /usr/lib/64/libsqlite3.so.0
+;Solaris32 : /usr/lib/libsqlite3.so, /usr/lib/libsqlite3.so.0
+;macOS : /opt/local/lib/libsqlite3.dylib, /usr/lib/libsqlite3.dylib, /opt/local/lib/libsqlite3.0.dylib, /usr/lib/libsqlite3.0.dylib
+;BSD : /usr/local/lib/libsqlite3.so, /usr/lib/libsqlite3.so, /usr/local/lib/libsqlite3.so.0, /usr/lib/libsqlite3.so.0
+DBLibFile=default
+
 [MyStations]
 ;Enables support for the stations list local file. 1-Yes 0-No ; Default = 1
 Enable=1
@@ -53,33 +70,48 @@ Enable=1
 
 ;Radio-browser.info API URL. Default = http://all.api.radio-browser.info ; 
 ;Warning! Using an IP address instead of a domain name results in faster performance, however, such a domain IP address may change 
-;and result in loss of connection to radio-browser.info. It is always worth checking the current list of servers at: https://api.radio-browser.info/net.html.
+;and result in loss of connection to radio-browser.info. It is always worth checking the current list of servers at: http://all.api.radio-browser.info/json/servers
 RBAPIURL=http://all.api.radio-browser.info
 
 ;Radio-browser.info most popular stations limit. Default = 100
 ;Warning! Due to the filtering functionality (since YTuner version 1.1.0) performed by the client side, this value means the number of stations got from radiobrowser-info and later optionally filtered out by YTuner. 
 ;For this reason, the station list displayed by AVR may be significantly different from this value or even be empty.
 ;Determine this value for yourself based on your filtering scope.
+;Applicable for [catNone,ctFile,ctMemStr] RBCacheType only.
 RBPopularAndSearchStationsLimit=1000
 
 ;Radio-browser.info minimum number of stations per category. Default = 3
+;Applicable for [catNone,ctFile,ctMemStr] RBCacheType only.
 RBMinStationsPerCategory=3
 
 ;Radio-browser.info UUIDs file cache TTL (hours) ; Empty or value <= 0 always means a valid UUID file cache (if exists) and never refreshes its content. ; Default = 24
-RBUUIDsCacheTTL=24
+;Applicable for [catNone,ctFile,ctMemStr] RBCacheType only.
+RBUUIDsCacheTTL=0
 
-;Radio-browser.info UUIDs file cache auto refresh. Only applicable when RBUUIDsCacheTTL>0. 1-Yes 0-No ; Default = 0
+;Radio-browser.info UUIDs file cache auto refresh. Applicable when RBUUIDsCacheTTL>0 only. 1-Yes 0-No ; Default = 0
+;Applicable for [catNone,ctFile,ctMemStr] RBCacheType only.
 RBUUIDsCacheAutoRefresh=0
 
-;Use one value of set [catNone,ctFile,ctMemStr] or leave blank.
-;[blank] or catNone = Do not use cache. Slow! 
-;catFile = Store cache in files. Most suitable if the amount of free memory is very small. Fast!
-;catMemStr = Store cache in memory. Fastest but may need more bytes of free memory.
-;Test the best option for you. Default = catFile;
-RBCacheType=catFile
+;Use one value of set [catNone,catFile,catMemStr,catDB,catMemDB,catPermMemDB] or leave blank.
+;catNone or blank = Do not use cache. Slow! 
+;catFile      = Store previously browsed categories and stations in cache files. Most suitable if the amount of free memory is very small.
+;catMemStr    = Store previously browsed categories and stations in memory cache. Fastest, but may need more bytes of free memory.
+;catDB        = Store ALL useful information about categories and stations of Radio-browser.info in SQLite DB file.
+;               The initial DB file creation process may take a few minutes, but will remain persistent after you restart YTuner.
+;               After this operation, you will no longer need access to redio-browser.info until you decide to refresh the downloaded data (when RBCacheTTL>0).
+;               Keep RBCacheTTL = 0 to prevent the DB file from being rebuilt after the RBCacheTTL time expires.
+;catMemDB     = Same as catDB, but the entire database is placed in memory. This solution is NOT permanent and once YTuner is down all data will be lost.
+;catPermMemDB = Combination of catMemDB+catDB; The entire database is placed in memory, but it is also saved to a file and remains persistent. The best option if you have valid sqlite3 driver.
+;Windows users can get sqlite3.dll driver from https://www.sqlite.org/download.html . Linux / Unix (Solaris, BSD) / macOS users should have such a driver already installed in their OS. 
+;Pay attention to "DBLibFolderLocation" parameter in "Configuration" section of this ini file.
+;catMemDB and catPermMemDB require additional ~30MB of free RAM to load entire database. Refreshing the database involves starting a concurrent thread that downloads the current content of the radio-browser.info resources and creates a second database while the current one is still running.
+;After the process is completed, the newly created database becomes the current database and the old one is deleted. Due to this, an additional ~30MB of free memory is needed for the duration of this process.
+;Additionally, data in JSON format downloaded from Radio-browser.info takes over 50MB.
+;Test the best option for you. Default = catPermMemDB;
+RBCacheType=catPermMemDB
 
 ;Radio-browser.info stations cache TTL (hours). Empty or value <= 0 always means a valid cache (if exists) and never refreshes its contents. ; Default = 24
-RBCacheTTL=24
+RBCacheTTL=0
 
 [Bookmark]
 ;Enables bookmarks. Your AVR must support bookmark to save stations to bookmark. 1-Yes 0-No ; Default = 1

doc/SQLITE.md

@@ -0,0 +1,90 @@
+## SQLite library
+* Windows users
+
+You can use attached `sqlite3.dll` library or get latest version from https://www.sqlite.org/download.html.
+* Linux / Unix (Solaris, BSD) / macOS users 
+
+You should have such a library already installed by default. If you keep your OS up to date you should have a newer version of this library than required (3.33.0).
+However, if for some reason you cannot upgrade the package containing the SQLite library on your OS, you should download the source code from https://www.sqlite.org/download.html (E.g. https://www.sqlite.org/2024/sqlite-autoconf-3450000.tar.gz) and build your own latest SQLite3 library.
+If you are familiar with building C code, you may want to skip reading further. If not, don't worry and read on.
+1) Get source code with command like :
+```
+$ wget https://www.sqlite.org/2024/sqlite-autoconf-3450000.tar.gz
+```
+2) Extract the tarball with :
+```
+$ tar -xvf sqlite-autoconf-3450000.tar.gz && cd sqlite-autoconf-3450000
+```
+3) Now, if you own 64-bit OS and want to build 64 bit SQLite library you can use:  
+```
+$ ./configure --prefix=your_64bit_library_target_directory
+```
+Change the `your_64bit_library_target_directory` to a directory of your choice, which YTuner will then be able to use, or select a directory appropriate for the libraries in your OS, e.g.:
+* Debian / Ubuntu / other Debian based Linux :
+
+`/usr/lib/x86_64-linux-gnu` or `/usr/lib/aarch64-linux-gnu` depends on your CPU arch. Check `gcc -dumpmachine` to find out.
+
+* CentOS / RHEL / Fedora / Oracle Linux :
+
+`/usr/lib64`
+
+* macOS :
+
+`/usr/lib` or `/usr/local/lib` or `/opt/local/lib`
+
+>Tip: Please note that from macOS version Big Sur 11.0.1, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. 
+
+* BSD :
+
+`/usr/local/lib`
+
+* Solaris :
+
+`/usr/lib/64`
+
+> Tip: You can uninstall obsolete (< 3.33.0) package with sqlite3 library.
+
+4) Run :
+```
+$ make
+$ sudo make install
+```
+5) If the entire process went correctly, you should find a new `lib/` subdirectory in the indicated directory, containing your new SQLite 64 bit libraries:
+* Linux/Unix: `libsqlite3.so.0.8.6` and/or some links like `libsqlite3.so` and/or `libsqlite3.so.0` and/or   
+* macOS: `libsqlite3.0.dylib` and/or some links like `libsqlite3.0.dylib`
+
+If for some reason you also need the 32 bit version (if your OS supports 32 bit applications), follow the above steps again but in step 3 use:
+
+```
+$ ./configure --prefix=your_32bit_library_target_directory --host=your_32bit_host_param "CFLAGS=-m32" "CXXFLAGS=-m32" "LDFLAGS=-m32"
+```
+Change the `your_32bit_library_target_directory` to a directory of your choice, which YTuner will then be able to use, or select a directory appropriate for the libraries in your OS, e.g.:
+* Debian / Ubuntu / other Debian based Linux :
+
+`/usr/lib/i386-linux-gnu` or `/usr/lib/arm-linux-gnueabi` or `/usr/lib/arm-linux-gnueabihf` depends on your CPU arch. Check `gcc -m32 -Q --help=target | fgrep -- -march` to find out.
+
+* CentOS / Fedora / Oracle Linux :
+
+`/usr/lib`
+
+* Solaris :
+
+`/usr/lib`
+
+Change the `your_32bit_host_param` to:
+
+* Debian / Ubuntu / other Debian based Linux :
+
+`i386-linux-gnu` or `arm-linux-gnu` or any others depends on your CPU arch.
+
+* CentOS / Fedora / Oracle Linux :
+
+`i386-redhat-linux` or `armv7hl-redhat-linux-gnueabi` or any others depends on your CPU arch.
+
+* Solaris :
+
+`i386-pc-solaris2.11` or any others depends on your CPU arch.
+
+> ! Importanat ! : Please make sure you get right 64-bit or 32-bit library file suitable for your YTuner 64-bit or 32-bit version.
+
+If you found any errors or inaccuracies in this description, please let me know.

res/sql.rc

@@ -0,0 +1,17 @@
+RBDB_SQL_CREATE_IDX	RCDATA "RBDB_SQL_CREATE_IDX.zip"
+RBDB_SQL_CREATE_REIDX	RCDATA "RBDB_SQL_CREATE_REIDX.zip"
+RBDB_SQL_CREATE_TABLES	RCDATA "RBDB_SQL_CREATE_TABLES.zip"
+RBDB_SQL_CREATE_VIEW_AVR	RCDATA "RBDB_SQL_CREATE_VIEW_AVR.zip"
+RBDB_SQL_DROP_VIEW_AVR	RCDATA "RBDB_SQL_DROP_VIEW_AVR.zip"
+RBDB_SQL_FIX_DATA	RCDATA "RBDB_SQL_FIX_DATA.zip"
+RBDB_SQL_INSERT_CODECS	RCDATA "RBDB_SQL_INSERT_CODECS.zip"
+RBDB_SQL_INSERT_COUNTRIES	RCDATA "RBDB_SQL_INSERT_COUNTRIES.zip"
+RBDB_SQL_INSERT_LANGUAGES	RCDATA "RBDB_SQL_INSERT_LANGUAGES.zip"
+RBDB_SQL_INSERT_STATIONS	RCDATA "RBDB_SQL_INSERT_STATIONS.zip"
+RBDB_SQL_INSERT_TAGS	RCDATA "RBDB_SQL_INSERT_TAGS.zip"
+RBDB_SQL_LIST_VIEWS	RCDATA "RBDB_SQL_LIST_VIEWS.zip"
+RBDB_SQL_SELECT_CATEGORY	RCDATA "RBDB_SQL_SELECT_CATEGORY.zip"
+RBDB_SQL_SELECT_ROOT	RCDATA "RBDB_SQL_SELECT_ROOT.zip"
+RBDB_SQL_SELECT_STATION	RCDATA "RBDB_SQL_SELECT_STATION.zip"
+RBDB_SQL_SELECT_STATIONS	RCDATA "RBDB_SQL_SELECT_STATIONS.zip"
+RBDB_SQL_VERSION	RCDATA "RBDB_SQL_VERSION.zip"

script/zip-sql.bat

@@ -0,0 +1 @@
+FOR %%i IN (..\src\sql\*.sql) DO 7z.exe a "..\res\%%~ni.zip" "%%i"