Add object tagging - Moodle 310 (#619)

* feat: add object tagging * test: fix unit test count checking * tagging: don't wait for object lock * tagging: improve migration controls and progress visibility * feat: move tagging status reports to check api * feat: display header with status report * refactor: integrate tagpushedtime into single update query * refactor: store tags against object id instead of hash * chore: organise tagging lang strings * bugfix: fix mysql query compatibility * tagging: move mimetype to metadata, add location/orphan tag source * tagging: check environment config length * settings: use admin_setting_check if available * report: remove object size from tag count report * tagging: ignore if cannot get lock * ci: small fixups * tagging: switch to admin setting for tagging environment * refactor: get object tag sync status count details separately * refactor: tweak defaults and add tagging adhoc task spawn limit * tests: reset static file storage before tests * bugfix: fix test * ci: fixup
catalyst · Dec 6, 2024 · 847c111 · 847c111
1 parent ab2face
commit 847c111
Show file tree

Hide file tree

Showing 39 changed files with 2,579 additions and 29 deletions.
diff --git a/TAGGING.md b/TAGGING.md
@@ -0,0 +1,65 @@
+# Tagging
+Tagging allows extra metadata about your files to be send to the external object store. These sources are defined in code, and currently cannot be configured on/off from the UI.
+
+Currently, this is only implemented for the S3 file system client. 
+**Tagging vs metadata**
+
+Note object tags are different from object metadata.
+
+Object metadata is immutable, and attached to the object on upload. With metadata, if you wish to update it (for example during a migration, or the sources changed), you have to copy the object with the new metadata, and delete the old object. This is not ideal, since deletion is optional in objectfs.
+
+Object tags are more suitable, since their permissions can be managed separately (e.g. a client can be allowed to modify tags, but not delete objects).
+
+## File system setup
+### S3
+[See the S3 docs for more information about tagging](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html).
+
+You must allow `s3:GetObjectTagging` and `s3:PutObjectTagging` permission to the objectfs client.
+
+## Sources
+The following sources are implemented currently:
+### Environment
+What environment the file was uploaded in. Configure the environment using `taggingenvironment` in the objectfs plugin settings.
+
+This tag is also used by objectfs to determine if tags can be overwritten. See [Multiple environments setup](#multiple-environments-setup) for more information.
+
+### Location
+Either `orphan` if the file no longer exists in the `files` table in Moodle, otherwise `active`.
+
+## Multiple environments setup
+This feature is designed to work in situations where multiple environments (e.g. prod, staging) points to the same bucket, however, some setup is needed:
+
+1. Turn off `overwriteobjecttags` in every environment except the production environment.
+2. Configure `taggingenvironment` to be unique for all environments.
+
+By doing the above two steps, it will allow the production environment to always set its own tags, even if a file was first uploaded to staging and then to production.
+
+Lower environments can still update tags, but only if the `environment` matches theirs. This allows staging to manage object tags on objects only it knows about, but as soon as the file is uploaded from production (and therefore have it's environment tag replaced with `prod`), staging will no longer touch it.
+
+## Migration
+Only new objects uploaded after enabling this feature will have tags added. To backfill tags for previously uploaded objects, you must do the following:
+
+- Manually run `trigger_update_object_tags` scheduled task from the UI, which queues a `update_object_tags` adhoc task that will process all objects marked as needing sync.
+or
+- Call the CLI to execute a `update_object_tags` adhoc task manually.
+
+You may need to update the DB to mark objects tag sync status as needing sync if the object has previously been synced before.
+## Reporting
+There is an additional graph added to the object summary report showing the tag value combinations and counts of each.
+
+Note, this is only for files that have been uploaded from the respective environment, and may not be consistent for environments where `overwriteobjecttags` is disabled (because the site does not know if a file was overwritten in the external store by another client).
+
+## For developers
+
+### Adding a new source
+Note the rules about sources:
+- Identifier must be < 32 chars long.
+- Value must be < 128 chars long.
+
+While external providers allow longer key/values, we intentionally limit it to reserve space for future use. These limits may change in the future as the feature matures.
+
+To add a new source:
+- Implement `tag_source`
+- Add to the `tag_manager` class
+- As part of an upgrade step, mark all objects `tagsyncstatus` to needing sync (using `tag_manager` class, or manually in the DB)
+- As part of an upgrade step, queue a `update_object_tags` adhoc task to process the tag migration.
diff --git a/classes/check/tagging_migration_status.php b/classes/check/tagging_migration_status.php
@@ -0,0 +1,80 @@
+<?php
+// This file is part of Moodle - http://moodle.org/
+//
+// Moodle is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// Moodle is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with Moodle.  If not, see <http://www.gnu.org/licenses/>.
+
+namespace tool_objectfs\check;
+
+use core\check\check;
+use core\check\result;
+use core\task\manager;
+use html_table;
+use html_writer;
+use tool_objectfs\task\update_object_tags;
+
+/**
+ * Tagging migration status check
+ *
+ * @package   tool_objectfs
+ * @author    Matthew Hilton <[email protected]>
+ * @copyright Catalyst IT
+ * @license   http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later
+ */
+class tagging_migration_status extends check {
+    /**
+     * Link to ObjectFS settings page.
+     *
+     * @return \action_link|null
+     */
+    public function get_action_link(): ?\action_link {
+        $url = new \moodle_url('/admin/category.php', ['category' => 'tool_objectfs']);
+        return new \action_link($url, get_string('pluginname', 'tool_objectfs'));
+    }
+
+    /**
+     * Get result
+     * @return result
+     */
+    public function get_result(): result {
+        // We want to check this regardless if enabled or supported and not exit early.
+        // Because it may have been turned off accidentally thus causing the migration to fail.
+        $tasks = manager::get_adhoc_tasks(update_object_tags::class);
+
+        if (empty($tasks)) {
+            return new result(result::NA, get_string('tagging:migration:nothingrunning', 'tool_objectfs'));
+        }
+
+        $table = new html_table();
+        $table->head = [
+            get_string('table:taskid', 'tool_objectfs'),
+            get_string('table:iteration', 'tool_objectfs'),
+            get_string('table:status', 'tool_objectfs'),
+        ];
+
+        foreach ($tasks as $task) {
+            $table->data[$task->get_id()] = [$task->get_id(), $task->get_iteration(), $task->get_status_badge()];
+        }
+        $html = html_writer::table($table);
+
+        $ataskisfailing = !empty(array_filter($tasks, function($task) {
+            return $task->get_fail_delay() > 0;
+        }));
+
+        if ($ataskisfailing) {
+            return new result(result::WARNING, get_string('check:tagging:migrationerror', 'tool_objectfs'), $html);
+        }
+
+        return new result(result::OK, get_string('check:tagging:migrationok', 'tool_objectfs'), $html);
+    }
+}
diff --git a/classes/check/tagging_status.php b/classes/check/tagging_status.php
@@ -0,0 +1,62 @@
+<?php
+// This file is part of Moodle - http://moodle.org/
+//
+// Moodle is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// Moodle is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with Moodle.  If not, see <http://www.gnu.org/licenses/>.
+
+namespace tool_objectfs\check;
+
+use core\check\check;
+use core\check\result;
+use tool_objectfs\local\tag\tag_manager;
+
+/**
+ * Tagging status check
+ *
+ * @package   tool_objectfs
+ * @author    Matthew Hilton <[email protected]>
+ * @copyright Catalyst IT
+ * @license   http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later
+ */
+class tagging_status extends check {
+    /**
+     * Link to ObjectFS settings page.
+     *
+     * @return \action_link|null
+     */
+    public function get_action_link(): ?\action_link {
+        $url = new \moodle_url('/admin/category.php', ['category' => 'tool_objectfs']);
+        return new \action_link($url, get_string('pluginname', 'tool_objectfs'));
+    }
+
+    /**
+     * Get result
+     * @return result
+     */
+    public function get_result(): result {
+        if (!tag_manager::is_tagging_enabled_and_supported()) {
+            return new result(result::NA, get_string('check:tagging:na', 'tool_objectfs'));
+        }
+
+        // Do a tag set test.
+        $config = \tool_objectfs\local\manager::get_objectfs_config();
+        $client = \tool_objectfs\local\manager::get_client($config);
+        $result = $client->test_set_object_tag();
+
+        if ($result->success) {
+            return new result(result::OK, get_string('check:tagging:ok', 'tool_objectfs'), $result->details);
+        } else {
+            return new result(result::ERROR, get_string('check:tagging:error', 'tool_objectfs'), $result->details);
+        }
+    }
+}
diff --git a/classes/check/tagging_sync_status.php b/classes/check/tagging_sync_status.php
@@ -0,0 +1,60 @@
+<?php
+// This file is part of Moodle - http://moodle.org/
+//
+// Moodle is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// Moodle is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with Moodle.  If not, see <http://www.gnu.org/licenses/>.
+
+namespace tool_objectfs\check;
+
+use core\check\check;
+use core\check\result;
+use tool_objectfs\local\tag\tag_manager;
+use tool_objectfs\local\tag_sync_count_result;
+
+/**
+ * Tagging sync status check
+ *
+ * @package   tool_objectfs
+ * @author    Matthew Hilton <[email protected]>
+ * @copyright Catalyst IT
+ * @license   http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later
+ */
+class tagging_sync_status extends check {
+    /**
+     * Link to ObjectFS settings page.
+     *
+     * @return \action_link|null
+     */
+    public function get_action_link(): ?\action_link {
+        $url = new \moodle_url('/admin/category.php', ['category' => 'tool_objectfs']);
+        return new \action_link($url, get_string('pluginname', 'tool_objectfs'));
+    }
+
+    /**
+     * Get result
+     * @return result
+     */
+    public function get_result(): result {
+        if (!tag_manager::is_tagging_enabled_and_supported()) {
+            return new tag_sync_count_result(result::NA, get_string('check:tagging:na', 'tool_objectfs'));
+        }
+
+        // We only do a lightweight check here, the get_details is overwritten in tag_sync_status_result
+        // to provide more information that is more computationally expensive to calculate.
+        if (tag_manager::tag_sync_errors_exist()) {
+            return new tag_sync_count_result(result::WARNING, get_string('check:tagging:syncerror', 'tool_objectfs'));
+        }
+
+        return new tag_sync_count_result(result::OK, get_string('check:tagging:syncok', 'tool_objectfs'));
+    }
+}
diff --git a/classes/local/manager.php b/classes/local/manager.php
@@ -27,6 +27,7 @@
 
 use stdClass;
 use tool_objectfs\local\store\object_file_system;
+use tool_objectfs\local\tag\tag_manager;
 
 /**
  * [Description manager]
@@ -64,6 +65,7 @@ public static function get_objectfs_config() {
         $config->batchsize = 10000;
         $config->useproxy = 0;
         $config->deleteexternal = 0;
+        $config->enabletagging = false;
 
         $config->filesystem = '';
         $config->enablepresignedurls = 0;
@@ -159,7 +161,7 @@ public static function update_object_by_hash($contenthash, $newlocation, $filesi
             $newobject->filesize = isset($oldobject->filesize) ? $oldobject->filesize :
                     $DB->get_field('files', 'filesize', ['contenthash' => $contenthash], IGNORE_MULTIPLE);
 
-            return self::update_object($newobject, $newlocation);
+            return self::upsert_object($newobject, $newlocation);
         }
         $newobject->location = $newlocation;
 
@@ -172,9 +174,7 @@ public static function update_object_by_hash($contenthash, $newlocation, $filesi
             $newobject->filesize = $filesize;
             $newobject->timeduplicated = time();
         }
-        $DB->insert_record('tool_objectfs_objects', $newobject);
-
-        return $newobject;
+        return self::upsert_object($newobject, $newlocation);
     }
 
     /**
@@ -184,16 +184,29 @@ public static function update_object_by_hash($contenthash, $newlocation, $filesi
      * @return stdClass
      * @throws \dml_exception
      */
-    public static function update_object(stdClass $object, $newlocation) {
+    public static function upsert_object(stdClass $object, $newlocation) {
         global $DB;
 
         // If location change is 'duplicated' we update timeduplicated.
         if ($newlocation === OBJECT_LOCATION_DUPLICATED) {
             $object->timeduplicated = time();
         }
 
+        $locationchanged = !isset($object->location) || $object->location != $newlocation;
         $object->location = $newlocation;
-        $DB->update_record('tool_objectfs_objects', $object);
+
+        // If id is set, update, else insert new.
+        if (empty($object->id)) {
+            $object->id = $DB->insert_record('tool_objectfs_objects', $object);
+        } else {
+            $DB->update_record('tool_objectfs_objects', $object);
+        }
+
+        // Post update, notify tag manager since the location tag likely needs changing.
+        if ($locationchanged && tag_manager::is_tagging_enabled_and_supported()) {
+            $fs = get_file_storage()->get_file_system();
+            $fs->push_object_tags($object->contenthash);
+        }
 
         return $object;
     }

diff --git a/classes/local/object_manipulator/manipulator.php b/classes/local/object_manipulator/manipulator.php
@@ -111,7 +111,7 @@ public function execute(array $objectrecords) {
 
             $newlocation = $this->manipulate_object($objectrecord);
             if (!empty($objectrecord->id)) {
-                manager::update_object($objectrecord, $newlocation);
+                manager::upsert_object($objectrecord, $newlocation);
             } else {
                 manager::update_object_by_hash($objectrecord->contenthash, $newlocation);
             }

diff --git a/classes/local/report/object_status_history_table.php b/classes/local/report/object_status_history_table.php
@@ -74,6 +74,11 @@ public function __construct($reporttype, $reportid) {
             $columnheaders['runningsize'] = get_string('object_status:runningsize', 'tool_objectfs');
         }
 
+        // Tag count report does not display the size.
+        if ($this->reporttype == 'tag_count') {
+            unset($columnheaders['size']);
+        }
+
         $this->set_attribute('class', 'table-sm');
         $this->define_columns(array_keys($columnheaders));
         $this->define_headers(array_values($columnheaders));

diff --git a/classes/local/report/objectfs_report.php b/classes/local/report/objectfs_report.php
@@ -78,7 +78,8 @@ public function add_row($datakey, $objectcount, $objectsum) {
      */
     public function add_rows(array $rows) {
         foreach ($rows as $row) {
-            $this->add_row($row->datakey, $row->objectcount, $row->objectsum);
+            // Note objectsum is optional.
+            $this->add_row($row->datakey, $row->objectcount, $row->objectsum ?? 0);
         }
     }
 
@@ -166,6 +167,7 @@ public static function get_report_types() {
             'location',
             'log_size',
             'mime_type',
+            'tag_count',
         ];
     }