feat: add object tagging and object overwrite setting

TAGGING.md

@@ -0,0 +1,83 @@
+# Tagging
+Tagging allows extra metadata about your files to be send to the external object store. These sources are defined in code, and currently cannot be configured on/off from the UI.
+
+Currently, this is only implemented for the S3 file system client. [See the S3 docs for more information](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html).
+
+**Tagging vs metadata**
+
+Note object tags are different from object metadata.
+
+Object metadata is immutable, and attached to the object on upload. With metadata, if you wish to update it (for example during a migration, or the sources changed), you have to copy the object with the new metadata, and delete the old object. This is problematic, since deletion is optional in objectfs.
+
+Object tags are more suitable, since their permissions can be managed separately (e.g. a client can be allowed to modify tags, but not delete objects).
+
+## File system setup
+### S3
+You must grant `s3:GetObjectTagging` and `s3:PutObjectTagging` permission.
+
+## Sources
+The following sources are implemented currently:
+### Environment
+What environment the file was uploaded in. Configure the environment using `$CFG->objectfs_environment_name`
+
+### Mimetype
+What mimetype the file is stored as under the `mdl_files` table.
+
+## Multiple environments pointing to single bucket
+It is possible you are using objectfs with multiple environments (e.g. prod, staging) that both point to the same bucket. Since files are referenced by contenthash, it generally does not matter where they come from, so this isn't a problem. However, for tagging to work as expected, you must ensure object override is turned off for every environment except production. This stops non-production sites overwriting tags on objects set by production.
+
+This means that staging is unable to overwrite objects (thus overwriting tags) for files uploaded elsewhere. However, files uploaded elsewhere can be overwitten (and overwrite tags) by production. 
+
+```mermaid
+graph LR
+    subgraph S3
+        Object("`**Object**
+        contenthash: xyz
+        tags: env=prod`")
+    end
+    subgraph Prod
+        UploadObjectProd["`**Upload object**
+        contenthash: xyz
+        tags: env=prod`"] --> Object
+    end
+    subgraph Staging
+        UploadObjectStaging["`**Upload object**
+        contenthash: xyz
+        tags: env=staging`"]
+    end
+    Blocked["Blocked - does not have permissions\nto overwrite existing objects"]
+    UploadObjectStaging --- Blocked
+    Blocked -.-> Object
+
+    style Object fill:#ffffff00,stroke:#ffa812
+    style S3 fill:#ffffff00,stroke:#ffa812
+    style Prod fill:#ffffff00,stroke:#26ff4a
+    style UploadObjectProd fill:#ffffff00,stroke:#26ff4a
+    style Staging fill:#ffffff00,stroke:#978aff
+    style UploadObjectStaging fill:#ffffff00,stroke:#978aff
+    style Blocked fill:#ffffff00,stroke:#ff0000
+```
+
+## Migration
+If the way a tag was calculated has changed, or new tags are added (or removed) or this feature was turned on for the first time (or turned on after being off), you must do the following: 
+- Manually run `trigger_update_object_tags` scheduled task from the UI, which queues a `update_object_tags` adhoc task that will process all objects marked as needing sync (default is true)
+or
+- Call the CLI to execute a `update_object_tags` adhoc task manually.
+
+## Reporting
+The objectFS status report now generates a summary of object tags. Note - The tags reported here are what are stored in the local DB, not the external store. Theoretically these the local DB and the external store are in sync, but this might not always be the case (for example if migration has not been run for a file, or file was uploaded to the s3 bucket another way such as a different environment).
+
+## For developers
+
+### Adding a new source
+Note the rules about sources:
+- Identifier must be < 32 chars long.
+- Value must be < 32 chars long.
+
+While external providers allow longer key/values, we intentionally limit it to reserve space for future use. These limits may change in the future as the feature matures.
+
+To add a new source:
+- Implement `tag_source`
+- Add to the `tag_manager` class
+- As part of an upgrade step, mark all objects `tagsyncstatus` to needing sync (using `tag_manager` class, or manually in the DB)
+- As part of an upgrade step, queue a `update_object_tags` adhoc task to process the tag migration.

classes/check/tagging_status.php

@@ -0,0 +1,62 @@
+<?php
+// This file is part of Moodle - http://moodle.org/
+//
+// Moodle is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// Moodle is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with Moodle.  If not, see <http://www.gnu.org/licenses/>.
+
+namespace tool_objectfs\check;
+
+use core\check\check;
+use core\check\result;
+use tool_objectfs\local\tag\tag_manager;
+
+/**
+ * Tagging status check
+ *
+ * @package   tool_objectfs
+ * @author    Matthew Hilton <[email protected]>
+ * @copyright Catalyst IT
+ * @license   http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later
+ */
+class tagging_status extends check {
+    /**
+     * Link to ObjectFS settings page.
+     *
+     * @return \action_link|null
+     */
+    public function get_action_link(): ?\action_link {
+        $url = new \moodle_url('/admin/category.php', ['category' => 'tool_objectfs']);
+        return new \action_link($url, get_string('pluginname', 'tool_objectfs'));
+    }
+
+    /**
+     * Get result
+     * @return result
+     */
+    public function get_result(): result {
+        if (!tag_manager::is_tagging_enabled_and_supported()) {
+            return new result(result::NA, 'Not enabled or supported by fs'); // TODO lang.
+        }
+
+        // Do a tag set test.
+        $config = \tool_objectfs\local\manager::get_objectfs_config();
+        $client = \tool_objectfs\local\manager::get_client($config);
+        $result = $client->test_set_object_tag();
+
+        if ($result->success) {
+            return new result(result::OK, get_string('check:tagging:ok', 'tool_objectfs'), $result->details);
+        } else {
+            return new result(result::ERROR, get_string('check:tagging:error', 'tool_objectfs'), $result->details);
+        }
+    }
+}

classes/local/manager.php

@@ -64,6 +64,7 @@ public static function get_objectfs_config() {
         $config->batchsize = 10000;
         $config->useproxy = 0;
         $config->deleteexternal = 0;
+        $config->enabletagging = false;
 
         $config->filesystem = '';
         $config->enablepresignedurls = 0;
@@ -329,6 +330,10 @@ public static function get_available_fs_list() {
      * @return string
      */
     public static function get_client_classname_from_fs($filesystem) {
+        // Unit tests need to return the test client.
+        if ($filesystem == '\tool_objectfs\tests\test_file_system') {
+            return '\tool_objectfs\tests\test_client';
+        }
         $clientclass = str_replace('_file_system', '', $filesystem);
         return str_replace('tool_objectfs\\', 'tool_objectfs\\local\\store\\', $clientclass.'\\client');
     }
@@ -378,4 +383,16 @@ public static function get_filename_from_header($header) {
         }
         return $filename;
     }
+
+    /**
+     * If is able to override existing objects in the external store.
+     * @return bool
+     */
+    public static function can_override_existing_objects(): bool {
+        // This is either "1" or "0".
+        $config = get_config('tool_objectfs', 'canoverrideobjects');
+
+        // Default to true if not set.
+        return is_string($config) ? (bool) $config : true;
+    }
 }

classes/local/report/objectfs_report.php

@@ -166,6 +166,7 @@ public static function get_report_types() {
             'location',
             'log_size',
             'mime_type',
+            'tag_count',
         ];
     }
 

classes/local/report/tag_count_report_builder.php

@@ -0,0 +1,51 @@
+<?php
+// This file is part of Moodle - http://moodle.org/
+//
+// Moodle is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// Moodle is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with Moodle.  If not, see <http://www.gnu.org/licenses/>.
+
+namespace tool_objectfs\local\report;
+
+/**
+ * Tag count report builder.
+ *
+ * @package   tool_objectfs
+ * @author    Matthew Hilton <[email protected]>
+ * @copyright Catalyst IT
+ * @license   http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later
+ */
+class tag_count_report_builder extends objectfs_report_builder {
+    /**
+     * Builds report
+     * @param int $reportid
+     * @return objectfs_report
+     */
+    public function build_report($reportid) {
+        global $DB;
+        $report = new objectfs_report('tag_count', $reportid);
+
+        // Returns counts + sizes of key:value.
+        $sql = "
+            SELECT CONCAT(COALESCE(object_tags.tagkey, '(untagged)'), ': ', object_tags.tagvalue) as datakey,
+                   COUNT(objects.id) as objectcount,
+                   SUM(objects.filesize) as objectsum
+              FROM {tool_objectfs_objects} objects
+         LEFT JOIN {tool_objectfs_object_tags} object_tags
+                ON objects.contenthash = object_tags.contenthash
+             GROUP BY object_tags.tagkey, object_tags.tagvalue
+        ";
+        $result = $DB->get_records_sql($sql);
+        $report->add_rows($result);
+        return $report;
+    }
+}

classes/local/store/object_client.php

@@ -25,6 +25,8 @@
 
 namespace tool_objectfs\local\store;
 
+use stdClass;
+
 interface object_client {
 
     /**
@@ -137,6 +139,31 @@ public function proxy_range_request(\stored_file $file, $ranges);
      */
     public function test_range_request($filesystem);
 
+    /**
+     * Tests setting an objects tag.
+     * @return stdClass containing 'success' and 'details' properties
+     */
+    public function test_set_object_tag(): stdClass;
+
+    /**
+     * Set the given objects tags in the external store.
+     * @param string $contenthash file content hash
+     * @param array $tags array of key=>value pairs to set as tags.
+     */
+    public function set_object_tags(string $contenthash, array $tags);
+
+    /**
+     * Returns given objects tags queried from the external store. External object must exist.
+     * @param string $contenthash file content has
+     * @return array array of key=>value tag pairs
+     */
+    public function get_object_tags(string $contenthash): array;
+
+    /**
+     * If the client supports object tagging feature.
+     * @return bool true if supports, else false
+     */
+    public function supports_object_tagging(): bool;
 }
 
 

classes/local/store/object_client_base.php

@@ -25,6 +25,8 @@
 
 namespace tool_objectfs\local\store;
 
+use stdClass;
+
 /**
  * [Description object_client_base]
  */
@@ -187,4 +189,38 @@ public function test_connection() {
     public function test_permissions($testdelete) {
         return (object)['success' => false, 'details' => ''];
     }
+
+    /**
+     * Tests setting an objects tag.
+     * @return stdClass containing 'success' and 'details' properties
+     */
+    public function test_set_object_tag(): stdClass {
+        return (object)['success' => false, 'details' => ''];
+    }
+
+    /**
+     * Set the given objects tags in the external store.
+     * @param string $contenthash file content hash
+     * @param array $tags array of key=>value pairs to set as tags.
+     */
+    public function set_object_tags(string $contenthash, array $tags) {
+        return [];
+    }
+
+    /**
+     * Returns given objects tags queried from the external store. External object must exist.
+     * @param string $contenthash file content has
+     * @return array array of key=>value tag pairs
+     */
+    public function get_object_tags(string $contenthash): array {
+        return [];
+    }
+
+    /**
+     * If the client supports object tagging feature.
+     * @return bool true if supports, else false
+     */
+    public function supports_object_tagging(): bool {
+        return false;
+    }
 }

classes/local/store/object_file_system.php

@@ -36,7 +36,10 @@
 use stored_file;
 use file_storage;
 use BlobRestProxy;
+use coding_exception;
+use Throwable;
 use tool_objectfs\local\manager;
+use tool_objectfs\local\tag\tag_manager;
 
 defined('MOODLE_INTERNAL') || die();
 
@@ -1154,4 +1157,53 @@ private function update_object(array $result): array {
 
         return $result;
     }
+
+    /**
+     * Syncs tags post upload for a given hash.
+     * External client must support tagging.
+     *
+     * @param string $contenthash file to sync tags for
+     */
+    public function sync_object_tags(string $contenthash) {
+        if (!$this->get_external_client()->supports_object_tagging()) {
+            throw new coding_exception("Cannot sync tags, external client does not support tagging.");
+        }
+
+        // Get a lock before syncing, to ensure other parts of objectfs are not moving/interacting with this object.
+        $lock = $this->acquire_object_lock($contenthash, 10);
+
+        // No lock - just skip it.
+        if (!$lock) {
+            throw new coding_exception("Could not get object lock"); // TODO different ex type?
+        }
+
+        try {
+            $objectexists = $this->is_file_readable_externally_by_hash($contenthash);
+            // If object does not exist, cannot sync tags to nothing, abort.
+            if (!$objectexists) {
+                tag_manager::mark_object_tag_sync_status($contenthash, tag_manager::SYNC_STATUS_SYNC_NOT_REQUIRED);
+                return;
+            }
+
+            // Cannot override and object exists, query existing and store locally.
+            if ($objectexists && !manager::can_override_existing_objects()) {
+                // Query existing tags and store them.
+                $existingtags = $this->get_external_client()->get_object_tags($contenthash);
+                tag_manager::store_tags_locally($contenthash, $existingtags);
+
+                // Else can override, upload new and store locally.
+            } else {
+                $tags = tag_manager::gather_object_tags_for_upload($contenthash);
+                $this->get_external_client()->set_object_tags($contenthash, $tags);
+                tag_manager::store_tags_locally($contenthash, $tags);
+            }
+
+            // Either way, it has synced.
+            tag_manager::mark_object_tag_sync_status($contenthash, tag_manager::SYNC_STATUS_SYNC_NOT_REQUIRED);
+        } catch (Throwable $e) {
+            $lock->release();
+            throw $e;
+        }
+        $lock->release();
+    }
 }