From 2595b4f13d724efe16e93a3b6986870a48ce7fe6 Mon Sep 17 00:00:00 2001
From: Matthew Evans <7916000+ml-evs@users.noreply.github.com>
Date: Sun, 20 Oct 2024 14:02:47 +0100
Subject: [PATCH 1/3] Add models for per-entry & per-property metadata (#2167)

* Addded meta field to entry model

* Added validator for meta field.

* Update pyyaml version in requirements.txt

* Update requirements.txt

* Update requirements.txt

* Remove test structures to get an idea of what triggers yaml import error on github.

* Readding meta field to test_good_structures.

* Readding meta field to test_structures.json.

* Added handling None for property_metadata to validator + small correctionds.

* Added test for validator per entry meta field.

* Added test for presence metadata field in test_structures.py.

* Remove metadata fields when the fields that they belong to are not returned.

* add extra test for bad prefix.

* Test if dependancy conflict causes error.

* Revert "Test if dependancy conflict causes error."

This reverts commit 6251e48f0efa741fdc3156b90cf2cdc598541658.

* Revert "Revert "Test if dependancy conflict causes error.""

This reverts commit bf9692a8ffdfa584a915227520990d7120c69b50.

* correct version httpx.

* commenting out validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue github.

* Slowly reassembling validator to see if this resolves issue Github.

* Slowly reassembling validator to see if this resolves issue Github.

* Slowly reassembling validator to see if this resolves issue Github..

* Slowly reassembling validator to see if this resolves issue Github..

* Placed yaml import in try except block.

* Added more cases to test data and added bugfix for removing associated metadata when a field is excluded.

* Removed seemingly unneccesary mypy ignore exception statement.

* remove change in version pyyaml in requirements.txt.

* correct spelling mistake

* moved starts_with_supported_prefix and check_starts_with_supported_prefix functions to BaseresouceMapper.

* Expanded docstring check_starts_with_supported_prefix

* Expanded docstring check_starts_with_supported_prefix

* Added return type for  starts_with_supported_prefix and check_starts_with_supported_prefix.

* Added return type for  starts_with_supported_prefix and check_starts_with_supported_prefix.

* Adjusted validators baseed on suggestion Matthew.

Co-authored-by: ml-evs ml-evs@users.noreply.github.com

* small corrections validators.

* small correction in removing unrequested metadata.

* Added spaces before values 'exmpl_originates_from _project'.

* update openapijson

* update openapijson

* update openapijson

* use lstrip instead of manually removing / one at a time.

* Update optimade/models/entries.py

update description property_metadata field

Co-authored-by: Matthew Evans <7916000+ml-evs@users.noreply.github.com>

* Update description per entry meta datafield.

* update openapijson.

* try to see if moving yaml is still necessary.

* Revert "try to see if moving yaml is still necessary."

This reverts commit 68f14d3fe6524e7e7475fec6e70e9889f722ccc7.

* Added supported_prefixes field to config.py

* Update requirements-client.txt (#1813)

removed duplicate requirement

* Modernize all Python 3.8 annotations (#1815)

* Use Python 3.9 as the 'base' CI version for linting

* Update pre-commit hooks

* Run `pyupgrade --py39-plus` to upgrade legacy annotations

* Add `--exit-non-zero-on-fix` for ruff

Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com>

* Use f-string over format

Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com>

---------

Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com>

* Update pydantic validators and model configs for 2024

* Remove validation of 'supported prefix' in lieu of just checking leading underscores

---------

Co-authored-by: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com>
Co-authored-by: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com>
---
 openapi/index_openapi.json                    | 21 ++++-
 openapi/openapi.json                          | 29 +++++--
 optimade/models/entries.py                    | 80 ++++++++++++++++++-
 optimade/server/data/test_structures.json     | 58 ++++++++++++++
 optimade/server/mappers/entries.py            | 20 +++--
 optimade/server/routers/utils.py              |  7 ++
 .../test_data/test_good_structures.json       |  7 ++
 tests/models/test_entries.py                  | 52 +++++++++++-
 tests/server/query_params/conftest.py         |  4 +-
 tests/server/routers/test_structures.py       |  6 ++
 10 files changed, 257 insertions(+), 27 deletions(-)

diff --git a/openapi/index_openapi.json b/openapi/index_openapi.json
index 9c4df1437..5d5aa841a 100644
--- a/openapi/index_openapi.json
+++ b/openapi/index_openapi.json
@@ -477,6 +477,19 @@
         "title": "BaseRelationshipResource",
         "description": "Minimum requirements to represent a relationship resource"
       },
+      "EntryMetadata": {
+        "properties": {
+          "property_metadata": {
+            "type": "object",
+            "title": "Property Metadata",
+            "description": "An object containing per-entry and per-property metadata. The keys are the names of the fields in attributes for which metadata is available. The values belonging to these keys are dictionaries containing the relevant metadata fields. See also [Metadata properties](https://github.com/Materials-Consortia/OPTIMADE/blob/develop/optimade.rst#metadata-properties)"
+          }
+        },
+        "additionalProperties": true,
+        "type": "object",
+        "title": "EntryMetadata",
+        "description": "Contains the metadata for the attributes of an entry"
+      },
       "EntryRelationships": {
         "properties": {
           "references": {
@@ -536,13 +549,13 @@
           "meta": {
             "anyOf": [
               {
-                "$ref": "#/components/schemas/Meta"
+                "$ref": "#/components/schemas/EntryMetadata"
               },
               {
                 "type": "null"
               }
             ],
-            "description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
+            "description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
           },
           "attributes": {
             "$ref": "#/components/schemas/EntryResourceAttributes",
@@ -1283,13 +1296,13 @@
           "meta": {
             "anyOf": [
               {
-                "$ref": "#/components/schemas/Meta"
+                "$ref": "#/components/schemas/EntryMetadata"
               },
               {
                 "type": "null"
               }
             ],
-            "description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
+            "description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
           },
           "attributes": {
             "$ref": "#/components/schemas/LinksResourceAttributes",
diff --git a/openapi/openapi.json b/openapi/openapi.json
index c06693278..5854e3d2a 100644
--- a/openapi/openapi.json
+++ b/openapi/openapi.json
@@ -1797,6 +1797,19 @@
         ],
         "title": "EntryInfoResponse"
       },
+      "EntryMetadata": {
+        "properties": {
+          "property_metadata": {
+            "type": "object",
+            "title": "Property Metadata",
+            "description": "An object containing per-entry and per-property metadata. The keys are the names of the fields in attributes for which metadata is available. The values belonging to these keys are dictionaries containing the relevant metadata fields. See also [Metadata properties](https://github.com/Materials-Consortia/OPTIMADE/blob/develop/optimade.rst#metadata-properties)"
+          }
+        },
+        "additionalProperties": true,
+        "type": "object",
+        "title": "EntryMetadata",
+        "description": "Contains the metadata for the attributes of an entry"
+      },
       "EntryRelationships": {
         "properties": {
           "references": {
@@ -1856,13 +1869,13 @@
           "meta": {
             "anyOf": [
               {
-                "$ref": "#/components/schemas/Meta"
+                "$ref": "#/components/schemas/EntryMetadata"
               },
               {
                 "type": "null"
               }
             ],
-            "description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
+            "description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
           },
           "attributes": {
             "$ref": "#/components/schemas/EntryResourceAttributes",
@@ -2443,13 +2456,13 @@
           "meta": {
             "anyOf": [
               {
-                "$ref": "#/components/schemas/Meta"
+                "$ref": "#/components/schemas/EntryMetadata"
               },
               {
                 "type": "null"
               }
             ],
-            "description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
+            "description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
           },
           "attributes": {
             "$ref": "#/components/schemas/LinksResourceAttributes",
@@ -2943,13 +2956,13 @@
           "meta": {
             "anyOf": [
               {
-                "$ref": "#/components/schemas/Meta"
+                "$ref": "#/components/schemas/EntryMetadata"
               },
               {
                 "type": "null"
               }
             ],
-            "description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
+            "description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
           },
           "attributes": {
             "$ref": "#/components/schemas/ReferenceResourceAttributes"
@@ -4070,13 +4083,13 @@
           "meta": {
             "anyOf": [
               {
-                "$ref": "#/components/schemas/Meta"
+                "$ref": "#/components/schemas/EntryMetadata"
               },
               {
                 "type": "null"
               }
             ],
-            "description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
+            "description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
           },
           "attributes": {
             "$ref": "#/components/schemas/StructureResourceAttributes"
diff --git a/optimade/models/entries.py b/optimade/models/entries.py
index e9a0075bb..6249414dd 100644
--- a/optimade/models/entries.py
+++ b/optimade/models/entries.py
@@ -1,9 +1,9 @@
 from datetime import datetime
 from typing import Annotated, Any, ClassVar, Literal
 
-from pydantic import BaseModel, field_validator
+from pydantic import BaseModel, ValidationInfo, field_validator
 
-from optimade.models.jsonapi import Attributes, Relationships, Resource
+from optimade.models.jsonapi import Attributes, Meta, Relationships, Resource
 from optimade.models.optimade_json import (
     BaseRelationshipResource,
     DataType,
@@ -117,6 +117,36 @@ def cast_immutable_id_to_str(cls, value: Any) -> str:
         return value
 
 
+class EntryMetadata(Meta):
+    """Contains the metadata for the attributes of an entry"""
+
+    property_metadata: dict = StrictField(
+        None,
+        description="""An object containing per-entry and per-property metadata. The keys are the names of the fields in attributes for which metadata is available. The values belonging to these keys are dictionaries containing the relevant metadata fields. See also [Metadata properties](https://github.com/Materials-Consortia/OPTIMADE/blob/develop/optimade.rst#metadata-properties)""",
+    )
+
+    @field_validator("property_metadata", mode="before")
+    def check_property_metadata_subfields(cls, value: Any) -> dict:
+        """Loop through any per-property metadata field and check that
+        the subfields are prefixed correctly.
+
+        """
+        error_fields: list[str] = []
+
+        if value is not None and isinstance(value, dict):
+            for field in value:
+                if property_metadata := value.get(field):
+                    for subfield in property_metadata:
+                        if not subfield.startswith("_"):
+                            error_fields.append(subfield)
+
+        if error_fields:
+            raise ValueError(
+                f"The keys under the field `property_metadata` need to be prefixed. The field(s) {error_fields} are not prefixed."
+            )
+        return value
+
+
 class EntryResource(Resource):
     """The base model for an entry resource."""
 
@@ -171,6 +201,14 @@ class EntryResource(Resource):
         ),
     ]
 
+    meta: Annotated[
+        EntryMetadata | None,
+        StrictField(
+            None,
+            description="""A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata.""",
+        ),
+    ] = None
+
     relationships: Annotated[
         EntryRelationships | None,
         StrictField(
@@ -179,6 +217,44 @@ class EntryResource(Resource):
         ),
     ] = None
 
+    @field_validator("meta", mode="before")
+    def check_meta(cls, meta: Any, info: ValidationInfo) -> dict | None:
+        """Validator to check whether the per-entry `meta` field is valid,
+        including stripping out any per-property metadata for properties that
+        do not otherwise appear in the model.
+
+        """
+        if not meta:
+            return meta
+
+        if property_metadata := meta.pop("property_metadata", None):
+            # check that all the fields under property metadata are in attributes
+            attributes = info.data.get("attributes", {})
+            property_error_fields: list[str] = []
+            for subfield in property_metadata:
+                if subfield not in attributes:
+                    property_error_fields.append(subfield)
+
+            if property_error_fields:
+                raise ValueError(
+                    f"The keys under the field `property_metadata` need to match with the field names in attributes. The field(s) {property_error_fields} are however not present in attributes."
+                )
+
+        meta["property_metadata"] = property_metadata
+
+        meta_error_fields: list[str] = []
+        for field in meta:
+            if field not in EntryMetadata.model_fields:
+                if not field.startswith("_"):
+                    meta_error_fields.append(field)
+
+        if meta_error_fields:
+            raise ValueError(
+                f"The keys under the field `meta` need to be prefixed if not otherwise defined. The field(s) {meta_error_fields} are not defined for per-entry `meta`."
+            )
+
+        return meta
+
 
 class EntryInfoProperty(BaseModel):
     description: Annotated[
diff --git a/optimade/server/data/test_structures.json b/optimade/server/data/test_structures.json
index 5a91e3b61..c189ad981 100644
--- a/optimade/server/data/test_structures.json
+++ b/optimade/server/data/test_structures.json
@@ -3,6 +3,13 @@
     "_id": {
       "$oid": "5cfb441f053b174410700d02"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project": "Pure Metals"
+        }
+      }
+    },
     "assemblies": null,
     "chemsys": "Ac",
     "cartesian_site_positions": [
@@ -80,6 +87,13 @@
     "_id": {
       "$oid": "5cfb441f053b174410700d03"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project": "Actinides_Alloys"
+        }
+      }
+    },
     "assemblies": null,
     "chemsys": "Ac-Ag-Ir",
     "cartesian_site_positions": [
@@ -197,6 +211,13 @@
     "_id": {
       "$oid": "5cfb441f053b174410700d04"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project": "Actinides_Alloys"
+        }
+      }
+    },
     "assemblies": null,
     "chemsys": "Ac-Ag-Pb",
     "cartesian_site_positions": [
@@ -323,6 +344,13 @@
     "_id": {
       "$oid": "5cfb441f053b174410700d18"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project": "Actinides_Alloys"
+        }
+      }
+    },
     "assemblies": null,
     "chemsys": "Ac-Mg",
     "cartesian_site_positions": [
@@ -413,6 +441,13 @@
     "_id": {
       "$oid": "5cfb441f053b174410700d1f"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project": null
+        }
+      }
+    },
     "assemblies": null,
     "chemsys": "Ac-O",
     "cartesian_site_positions": [
@@ -515,6 +550,11 @@
     "_id": {
       "$oid": "5cfb441f053b174410700d6f"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {}
+      }
+    },
     "assemblies": null,
     "chemsys": "Ac-Cu-F-O",
     "cartesian_site_positions": [
@@ -639,6 +679,13 @@
     "_id": {
       "$oid": "5cfb441f053b174410700dc9"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project": "Pure Metals"
+        }
+      }
+    },
     "assemblies": null,
     "chemsys": "Ag",
     "cartesian_site_positions": [
@@ -706,6 +753,11 @@
     "_id": {
       "$oid": "5cfb441f053b174410700ddd"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": null
+      }
+    },
     "assemblies": null,
     "chemsys": "Ag-Br-Cl-Te",
     "cartesian_site_positions": [
@@ -896,6 +948,9 @@
     "_id": {
       "$oid": "5cfb441f053b174410700e04"
     },
+    "meta": {
+      "property_metadata": {}
+    },
     "assemblies": null,
     "chemsys": "Ag-C-Cl-N-O-S",
     "cartesian_site_positions": [
@@ -1072,6 +1127,9 @@
     "_id": {
       "$oid": "5cfb441f053b174410700e11"
     },
+    "meta": {
+      "property_metadata": null
+    },
     "assemblies": null,
     "chemsys": "Ag-C-Cl-H-N",
     "cartesian_site_positions": [
diff --git a/optimade/server/mappers/entries.py b/optimade/server/mappers/entries.py
index 8f499f9b5..d9f4aaaad 100644
--- a/optimade/server/mappers/entries.py
+++ b/optimade/server/mappers/entries.py
@@ -74,7 +74,13 @@ class BaseResourceMapper:
     PROVIDER_FIELDS: tuple[str, ...] = ()
     ENTRY_RESOURCE_CLASS: type[EntryResource] = EntryResource
     RELATIONSHIP_ENTRY_TYPES: set[str] = {"references", "structures"}
-    TOP_LEVEL_NON_ATTRIBUTES_FIELDS: set[str] = {"id", "type", "relationships", "links"}
+    TOP_LEVEL_NON_ATTRIBUTES_FIELDS: set[str] = {
+        "id",
+        "type",
+        "relationships",
+        "links",
+        "meta",
+    }
 
     @classmethod
     @lru_cache(maxsize=NUM_ENTRY_TYPES)
@@ -118,18 +124,10 @@ def all_aliases(cls) -> Iterable[tuple[str, str]]:
     @classproperty
     @lru_cache(maxsize=1)
     def SUPPORTED_PREFIXES(cls) -> set[str]:
-        """A set of prefixes handled by this entry type.
-
-        !!! note
-            This implementation only includes the provider prefix,
-            but in the future this property may be extended to include other
-            namespaces (for serving fields from, e.g., other providers or
-            domain-specific terms).
-
-        """
+        """A set of prefixes handled by this entry type."""
         from optimade.server.config import CONFIG
 
-        return {CONFIG.provider.prefix}
+        return set(CONFIG.supported_prefixes)
 
     @classproperty
     def ALL_ATTRIBUTES(cls) -> set[str]:
diff --git a/optimade/server/routers/utils.py b/optimade/server/routers/utils.py
index 1d0a32446..2bd570139 100644
--- a/optimade/server/routers/utils.py
+++ b/optimade/server/routers/utils.py
@@ -118,6 +118,13 @@ def handle_response_fields(
         for field in exclude_fields:
             if field in new_entry["attributes"]:
                 del new_entry["attributes"][field]
+            if new_entry.get("meta") and (
+                property_meta_data_fields := new_entry.get("meta").get(  # type: ignore[union-attr]
+                    "property_metadata"
+                )
+            ):
+                if field in property_meta_data_fields:
+                    del new_entry["meta"]["property_metadata"][field]
 
         # Include missing fields that were requested in `response_fields`
         for field in include_fields:
diff --git a/tests/models/test_data/test_good_structures.json b/tests/models/test_data/test_good_structures.json
index b84605832..63c7c6044 100644
--- a/tests/models/test_data/test_good_structures.json
+++ b/tests/models/test_data/test_good_structures.json
@@ -165,6 +165,13 @@
     "last_modified": {
       "$date": "2019-06-08T05:13:37.331Z"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project":"piezoelectic_perovskites"
+        }
+      }
+    },
     "band_gap": 1.23456,
     "chemsys": "C-H-Cl-N-Na-O-Os-P",
     "elements": ["C", "Cl", "H", "N", "Na", "O", "Os", "P"],
diff --git a/tests/models/test_entries.py b/tests/models/test_entries.py
index a3ab7e318..e1a8a492e 100644
--- a/tests/models/test_entries.py
+++ b/tests/models/test_entries.py
@@ -1,7 +1,7 @@
 import pytest
 from pydantic import ValidationError
 
-from optimade.models.entries import EntryRelationships
+from optimade.models.entries import EntryRelationships, EntryResource
 
 
 def test_simple_relationships():
@@ -48,3 +48,53 @@ def test_advanced_relationships():
     }
     with pytest.raises(ValidationError):
         EntryRelationships(**relationship)
+
+
+def test_meta():
+    import copy
+
+    good_entry_resource = {
+        "id": "goodstruct123",
+        "type": "structure",
+        "attributes": {
+            "last_modified": "2023-07-21T05:13:37.331Z",
+            "elements": ["Ac"],
+            "_exmpl_database_specific_property": "value1",
+            "elements_ratios": [1.0],
+        },
+        "meta": {
+            "property_metadata": {
+                "elements_ratios": {
+                    "_exmpl_mearsurement_method": "ICP-OES",
+                },
+                "_exmpl_database_specific_property": {
+                    "_exmpl_metadata_property": "metadata_value"
+                },
+            }
+        },
+    }
+
+    EntryResource(**good_entry_resource)
+
+    bad_entry_resources = [
+        good_entry_resource,
+        copy.deepcopy(good_entry_resource),
+        copy.deepcopy(good_entry_resource),
+        copy.deepcopy(good_entry_resource),
+    ]
+    bad_entry_resources[0]["meta"]["property_metadata"][
+        "_exmpl_database_specific_property"
+    ] = {"metadata_property": "metadata_value"}
+    bad_entry_resources[1]["meta"]["property_metadata"][
+        "database_specific_property"
+    ] = {"_exmpl_metadata_property": "metadata_value"}
+    bad_entry_resources[2]["meta"]["database_specific_property"] = {
+        "_exmpl_metadata_property": "metadata_value"
+    }
+    bad_entry_resources[3]["meta"]["_other_database_specific_property"] = {
+        "_exmpl_metadata_property": "metadata_value"
+    }
+
+    for bad_entry in bad_entry_resources:
+        with pytest.raises(ValueError):
+            EntryResource(**bad_entry)
diff --git a/tests/server/query_params/conftest.py b/tests/server/query_params/conftest.py
index 0d8a2c2a0..b94daf9eb 100644
--- a/tests/server/query_params/conftest.py
+++ b/tests/server/query_params/conftest.py
@@ -77,11 +77,13 @@ def inner(
 
         response = get_good_response(request, server)
         expected_fields.add("attributes")
-
+        expected_fields.discard("meta")
         response_fields = set()
         for entry in response["data"]:
             response_fields.update(set(entry.keys()))
             response_fields.update(set(entry["attributes"].keys()))
+            # As "meta" is an optional field the response may or may not have it, so we remove it here to prevent problems in the assert below.
+            response_fields.discard("meta")
         assert sorted(expected_fields) == sorted(response_fields)
 
     return inner
diff --git a/tests/server/routers/test_structures.py b/tests/server/routers/test_structures.py
index a0e089f2d..7f8bfa726 100644
--- a/tests/server/routers/test_structures.py
+++ b/tests/server/routers/test_structures.py
@@ -69,6 +69,12 @@ def test_structures_endpoint_data(self):
         assert self.json_response["data"]["type"] == "structures"
         assert "attributes" in self.json_response["data"]
         assert "_exmpl_chemsys" in self.json_response["data"]["attributes"]
+        assert (
+            self.json_response["data"]["meta"]["property_metadata"]["elements_ratios"][
+                "_exmpl_originates_from_project"
+            ]
+            == "Pure Metals"
+        )
 
 
 def test_check_response_single_structure(check_response):

From e93c553339f4c7b210c06349542a026dbbb8e271 Mon Sep 17 00:00:00 2001
From: Matthew Evans <git@ml-evs.science>
Date: Sun, 20 Oct 2024 14:29:57 +0100
Subject: [PATCH 2/3] Switch back to model validator for meta check

---
 optimade/models/entries.py | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/optimade/models/entries.py b/optimade/models/entries.py
index 6249414dd..b167aedf7 100644
--- a/optimade/models/entries.py
+++ b/optimade/models/entries.py
@@ -1,7 +1,7 @@
 from datetime import datetime
 from typing import Annotated, Any, ClassVar, Literal
 
-from pydantic import BaseModel, ValidationInfo, field_validator
+from pydantic import BaseModel, field_validator, model_validator
 
 from optimade.models.jsonapi import Attributes, Meta, Relationships, Resource
 from optimade.models.optimade_json import (
@@ -217,19 +217,21 @@ class EntryResource(Resource):
         ),
     ] = None
 
-    @field_validator("meta", mode="before")
-    def check_meta(cls, meta: Any, info: ValidationInfo) -> dict | None:
+    @model_validator(mode="before")
+    def check_meta(cls, data: Any) -> dict | None:
         """Validator to check whether the per-entry `meta` field is valid,
         including stripping out any per-property metadata for properties that
         do not otherwise appear in the model.
 
         """
+
+        meta = data.get("meta")
         if not meta:
-            return meta
+            return data
 
         if property_metadata := meta.pop("property_metadata", None):
             # check that all the fields under property metadata are in attributes
-            attributes = info.data.get("attributes", {})
+            attributes = data.get("attributes", {})
             property_error_fields: list[str] = []
             for subfield in property_metadata:
                 if subfield not in attributes:
@@ -237,7 +239,7 @@ def check_meta(cls, meta: Any, info: ValidationInfo) -> dict | None:
 
             if property_error_fields:
                 raise ValueError(
-                    f"The keys under the field `property_metadata` need to match with the field names in attributes. The field(s) {property_error_fields} are however not present in attributes."
+                    f"The keys under the field `property_metadata` need to match with the field names in attributes. The field(s) {property_error_fields} are however not present in attributes {attributes}"
                 )
 
         meta["property_metadata"] = property_metadata
@@ -253,7 +255,8 @@ def check_meta(cls, meta: Any, info: ValidationInfo) -> dict | None:
                 f"The keys under the field `meta` need to be prefixed if not otherwise defined. The field(s) {meta_error_fields} are not defined for per-entry `meta`."
             )
 
-        return meta
+        data["meta"] = meta
+        return data
 
 
 class EntryInfoProperty(BaseModel):

From aee56b757a8a5c9ffdaba2e613cd98418d1aba1d Mon Sep 17 00:00:00 2001
From: Matthew Evans <git@ml-evs.science>
Date: Sun, 20 Oct 2024 14:30:12 +0100
Subject: [PATCH 3/3] Update test for meta with now-allowed case

---
 tests/models/test_entries.py | 25 +++++++++++--------------
 1 file changed, 11 insertions(+), 14 deletions(-)

diff --git a/tests/models/test_entries.py b/tests/models/test_entries.py
index e1a8a492e..7f383abee 100644
--- a/tests/models/test_entries.py
+++ b/tests/models/test_entries.py
@@ -51,8 +51,6 @@ def test_advanced_relationships():
 
 
 def test_meta():
-    import copy
-
     good_entry_resource = {
         "id": "goodstruct123",
         "type": "structure",
@@ -76,23 +74,22 @@ def test_meta():
 
     EntryResource(**good_entry_resource)
 
-    bad_entry_resources = [
-        good_entry_resource,
-        copy.deepcopy(good_entry_resource),
-        copy.deepcopy(good_entry_resource),
-        copy.deepcopy(good_entry_resource),
-    ]
+    # Test that other prefixed fields are allowed in meta
+    good_entry_resource["meta"]["_other_database_specific_property"] = {
+        "_exmpl_metadata_property": "entry 3"
+    }
+
+    EntryResource(**good_entry_resource)
+
+    bad_entry_resources = [good_entry_resource.copy() for _ in range(4)]
     bad_entry_resources[0]["meta"]["property_metadata"][
         "_exmpl_database_specific_property"
-    ] = {"metadata_property": "metadata_value"}
+    ] = {"metadata_property": "entry 0"}
     bad_entry_resources[1]["meta"]["property_metadata"][
         "database_specific_property"
-    ] = {"_exmpl_metadata_property": "metadata_value"}
+    ] = {"_exmpl_metadata_property": "entry 1"}
     bad_entry_resources[2]["meta"]["database_specific_property"] = {
-        "_exmpl_metadata_property": "metadata_value"
-    }
-    bad_entry_resources[3]["meta"]["_other_database_specific_property"] = {
-        "_exmpl_metadata_property": "metadata_value"
+        "_exmpl_metadata_property": "entry 2"
     }
 
     for bad_entry in bad_entry_resources: