Relink not possible for Hyper Backup on Amazon S3 Glacier

[TorbenSchreiter]

Hi there,

the one remaining issue I have after moving all packages to M.2s is with Hyper Backup. I have a regular backup job configured to Amazon S3 (actually Glacier). After the package move it showed up red with a "Relink now" button. Whenever I press it, some process starts, runs for ~15-20 minutes with a progress bar actually step-by-step completing and afterwards the screen looks exactly as before (see screenshot).

EDIT. The Hyper Backup log says:

Error | 2024/10/21 18:08:39 | SYSTEM | [Amazon S3][glhv3-backup-s3-01 - Amazon S3] Failed to run relink task.
Error | 2024/10/21 18:08:39 | SYSTEM | [Amazon S3][glhv3-backup-s3-01 - Amazon S3] Moving backed up files from Amazon S3 to Amazon Glacier is not supported. Please restore files from Amazon Glacier via the AWS Management console.
Information | 2024/10/21 16:57:36 | SYSTEM | [Amazon S3][glhv3-backup-s3-01 - Amazon S3] Relink task started.

Is this an issue related to the the pkg move at all or shall I contact Synology support about it? What would I tell the support team what has happened? Some of the to-be-backed-up shared folders are now on a different volume. Is this the potential cause?

The source volume of the pkg move is, by now, deleted.

Cheers and thanks.

[007revad]

Some of the to-be-backed-up shared folders are now on a different volume. Is this the potential cause?

I would say, yes.

The source volume of the pkg move is, by now, deleted.

This will be a problem. DSM's @database folder would have been deleted.

[007revad]

It is possible that DSM only creates the web_packages folder when you install a package that needs it. The packages that I know create a folder in the web_packages folder are:

container manager
docker
idrive
joomla
kodexplorer
mediawiki
phpmyadmin
vtigercrm
wordpress

[TorbenSchreiter]

Thanks, I had Docker installed previously and will reinstall at some point in the future.

Ok to delete/rename /volume1/@database?

[007revad]

I'd rename it to /volume1/@database_bak just in case. If everyone is okay after a reboot, and DSM did not create a new /volume1/@database you can then delete /volume1/@database_bak.

[TorbenSchreiter]

Strange: after reboot it has recreated /volume1/@database even though the symlink is pointing to /volume2. Looks like it might actually still be using the one on /volume1.

user123@GLHV3:/volume1$ ls -l
[..]
drwxr-xr-x  1 root         root          32 Nov  9 20:27 @database
drwxr-xr-x  1 root         root         120 Oct 25 10:06 @database_bak
[..]
user123@GLHV3:/volume1$ readlink /var/services/pgsql
/volume2/@database/pgsql

[007revad]

/var/services/pgsql only relates to the pgsql folder (and contents) in the @database folder. As long as /var/services/pgsql is pointing to
/volume2/@database/pgsql then the postgress sql database DSM is using is on volume2.

DSM recreating the /volume1/@database folder is okay. DSM saves some other files in the @database folder on the lowest numbered volume, like:

/volume1/@database/autoupdate/syno-abuser-blocklist/ipblocklist
/volume1/@database/synolog/.SMBXFERDB
/volume1/@database/synologan/alert.sqlite

When I deleted volume1 DSM recreated those same files in /volume2/@database even though my postgress sql database DSM is on volume4.

[TorbenSchreiter]

the one remaining issue I have after moving all packages to M.2s is with Hyper Backup. I have a regular backup job configured to Amazon S3 (actually Glacier). After the package move it showed up red with a "Relink now" button. Whenever I press it, some process starts, runs for ~15-20 minutes with a progress bar actually step-by-step completing and afterwards the screen looks exactly as before (see screenshot).

EDIT. The Hyper Backup log says:

Error | 2024/10/21 18:08:39 | SYSTEM | [Amazon S3][glhv3-backup-s3-01 - Amazon S3] Failed to run relink task.
Error | 2024/10/21 18:08:39 | SYSTEM | [Amazon S3][glhv3-backup-s3-01 - Amazon S3] Moving backed up files from Amazon S3 to Amazon Glacier is not supported. Please restore files from Amazon Glacier via the AWS Management console.
Information | 2024/10/21 16:57:36 | SYSTEM | [Amazon S3][glhv3-backup-s3-01 - Amazon S3] Relink task started.

I still had not fully solved the issue with the Hyper Backup S3 relink related to the files in storage are actually not standard S3 buckets, but Glacier Deep Archive Access tier.

I had reached out to Synology Support, but their response was not very helpful as they just responded that Glacier is not supported.

Looking for other solutions, I found these findings generally helpful in understanding what part of the HyperBackup folder structure needs to be retrievable (i.e. S3 and not Glacier) and what parts can stay in Glacier Deep Archive:
https://community.synology.com/enu/forum/1/post/135696

As my aim is not to rotate or change the existing backup archive (I am fine with incremental backup as the files rarely change and the relevant value is that new files are being backed up), this part seems most relevant:

Changing the storage class of all objects under Pool// to "Glacier" seemed to have no immediate ill effects on backups. I was able to successfuly run backup integrity checks, and even browse historical backup versions. Trying to download backed up files, did not work, as expected.

So, what I tried first (naively), was to change for all files/folders outside /pool/ via AWS console just the storage class from Glacier Deep Archive to STANDARD_IA (i.e. /., /Config/, /Control/, /Guard/). That job completed, however, did not have the desired effect (relink still failed despite files in AWS console being shown as STANDARD_IA). This is probably the reason: https://repost.aws/questions/QUWpuCbMiBTciiCScrjOT3EQ/move-objects-from-glacier-to-s3-standard-class

So, the second attempt that I tried today is to change the storage class back to DEEP_ARCHIVE, which completed already. I had to do this as the objects that were previously in Deep Archive but were changed in Storage Class to STANDARD_IA before, had the "Initiate restore" button greyed out. After changing the storage class back to Deep Archive, the option reappeared and I started all of the retrieval requests according to this:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/restoring-objects.html#restore-archived-objects

This will now take ~48 hours. I will revert back once that is done and I had time to look at it again.

[TorbenSchreiter]

Still no success. Here is what I tried so far:

After I had restore manually from AWS Console before, and after the wait of ~48 hours for restore to successfully complete from Deep Archive, I ran this via AWS CloudShell (https://console.aws.amazon.com/cloudshell/home):

aws s3 cp s3://<S3_Bucket_Name>/<HyperBackup-Jobname>.hbk/ s3://<S3_Bucket_Name>/<HyperBackup-Jobname>.hbk/ --exclude 'Pool/*' --recursive --storage-class STANDARD_IA --force-glacier-transfer

(Be aware these operations may bear cost w/ AWS.)

After that, everything except for contents of /Pool/* was back in STANDARD_IA and at least downloadable via AWS Console. However, the HyperBackup relink operation still fails for the same reason. So, it probably needs to retrieve/download something from within /Pool/... (ouch)