Add field lists to CoST IDS mapping files, update core CoST mapping to reflect latest schema

docs/conf.py

@@ -67,7 +67,7 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static', 'examples']
+html_static_path = ['_static', 'examples', '../mapping/sustainability.yaml']
 
 
 # -- Local configuration -----------------------------------------------------
@@ -159,6 +159,7 @@ def setup(app):
                     reader = csv.reader(i)
                     writer = csv.writer(o, lineterminator='\n')
                     for row in reader:
-                        del row[column_index]
+                        del row[column_index]  # Drop mapping column
+                        row = row[0:3]  # Drop OC4IDS Fields and OC4IDS Codes columns
                         writer.writerow(row)
             path.unlink()

docs/reference/changelog.md

@@ -5,6 +5,7 @@
 ### Documentation
 
 * [#478](https://github.com/open-contracting/infrastructure/pull/478), [#479](https://github.com/open-contracting/infrastructure/pull/479) - Fix validation errors in `example.json`.
+* [#483](https://github.com/open-contracting/infrastructure/pull/483) - Update core CoST IDS mapping to reflect latest schema.
 
 ### Schema
 
@@ -22,6 +23,10 @@
     * 'airManagement'
     * 'waterManagement'
 
+### Other
+
+* [#483](https://github.com/open-contracting/infrastructure/pull/483) - Add field lists to CoST IDS mappings.
+
 ## [0.9.4] - 2023-12-12
 
 ### Documentation

manage.py

@@ -1068,5 +1068,45 @@ def update_sustainability_docs():
         f.writelines(docs)
 
 
+@cli.command()
+def update_sustainability_fields():
+    """
+    Update the list of fields used in each element's example in mapping/sustainability.yaml.
+    """
+
+    def __get_paths(d):
+        """
+        Get a list of paths from a JSON object.
+        """
+        if isinstance(d, dict):
+            for key, value in d.items():
+                yield f'/{key}'
+                yield from (f'/{key}{p}' for p in __get_paths(value))
+
+        elif isinstance(d, list):
+            for i, value in enumerate(d):
+                yield from (f'{p}' for p in __get_paths(value))
+
+    # Load sustainability modules mapping
+    filename = basedir / 'mapping' / 'sustainability.yaml'
+    with open(filename) as f:
+        mapping = yaml.safe_load(f)
+
+    mapping = {element["id"]: element for element in mapping}
+
+    # Get list of fields in example
+    for element in mapping.values():
+        if element["example"] != '':
+            fields = set()
+            element["fields"] = [fields.add(path) or path for path in __get_paths(
+                json.loads(element["example"])) if path not in fields]
+        # Handle elements that reference another element and have a blank example
+        elif element["refs"] != '':
+            element["fields"] = [fields.add(path) or path for path in __get_paths(
+                json.loads(mapping[element["refs"]]["example"])) if path not in fields]
+
+    write_yaml_file(filename, list(mapping.values()))
+
+
 if __name__ == '__main__':
     cli()

mapping/process-level-implementation.csv

@@ -1,7 +1,7 @@
-CoST IDS element,CoST IDS draft definition,Mapping to OC4IDS,Mapping from OCDS
-Variation to contract price,Difference between the price at contract award and the current projected price,"Contracting process: For each variation, publish an entry with a `.date` and `.description` in `.summary.modifications` with `.type` of 'value'. Provide the contract value before the variation in `.oldContractValue` and the contract value after the variation in `.newContractValue`",*Monitor `/contracts/value` for changes. Copy `/contracts/amendments/description` to `.description` and `/date` to `.date`.*
-Escalation of contract price,"Escalation to date of the price of materials, labour, equipment etc. due to fluctuations in inflation, currency etc.","Contracting process: For each escalation, publish an entry with a `.date` and `.description` in `.summary.modifications` with `.type` of 'value'. Provide the contract value before the escalation in `.oldContractValue` and the contract value after the escalation in `.newContractValue`",*Monitor `/contracts/value` for changes. Copy `/contracts/amendments/description` to `.description` and `/date` to `.date`.*
-Variation to contract duration,Difference between original duration at contract award and the current projected duration in weeks.,"Contracting process: For each variation, publish an entry with a `.date` and `.description` in `.summary.modifications` with `.type` of 'duration'. Provide the contract period before the variation in `.oldContractPeriod` and the contract period after the variation in `.newContractPeriod`",*Monitor `/contracts/period` for changes. Copy `/contracts/amendments/description` to `.description` and `/date` to `.date`.*
-Variation to contract scope,Any changes between original scope at contract award and the current scope,"Contracting process: For each variation, publish an entry with a `.date` and `.description` in `.summary.modifications` with `.type` of 'scope'","*Monitor `/contracts/description`, `/contracts/items` and `/contracts/implementation/milestones` for changes. Copy `/contracts/amendments/description` to `.description` and `/date` to `.date`.*"
-Reasons for price changes,"Summary of reasons for primary changes (e.g. variations) that then lead to changes in contract price, major price fluctuations and accumulative increase or decrease in price.","Contracting process: For each variation, provide a `.rationale` in `.summary.modifications`.",*Copy `contracts/amendments/rationale` to `.rationale`.*
-Reasons for scope and duration changes,Summary of reasons for primary changes (e.g. variations) that then lead to changes in the scope and duration.,"Contracting process: For each variation, provide a `.rationale` in `.summary.modifications`.",*Copy `contracts/amendments/rationale` to `.rationale`.*
+CoST IDS element,Description,Mapping to OC4IDS,Mapping from OCDS,OC4IDS Fields,OC4IDS Codes
+Variation to contract price,Difference between the price at contract award and the current projected price,"Contracting process: For each variation, publish an entry with a `.date` and `.description` in `.summary.modifications` with `.type` of 'value'. Provide the contract value before the variation in `.oldContractValue` and the contract value after the variation in `.newContractValue`",*Monitor `/contracts/value` for changes. Copy `/contracts/amendments/description` to `.description` and `/date` to `.date`.*,"contractingProcesses,contractingProcesses/id,contractingProcesses/summary,contractingProcesses/summary/modifications,contractingProcesses/summary/modifications/id,contractingProcesses/summary/modifications/date,contractingProcesses/summary/modifications/description,contractingProcesses/summary/modifications/type,contractingProcesses/summary/modifications/oldContractValue,contractingProcesses/summary/modifications/oldContractValue/amount,contractingProcesses/summary/modifications/oldContractValue/currency,contractingProcesses/summary/modifications/newContractValue,contractingProcesses/summary/modifications/newContractValue/amount,contractingProcesses/summary/modifications/newContractValue/currency",contractingProcesses/summary/modifications/type=’value’
+Escalation of contract price,"Escalation to date of the price of materials, labour, equipment etc. due to fluctuations in inflation, currency etc.","Contracting process: For each escalation, publish an entry with a `.date` and `.description` in `.summary.modifications` with `.type` of 'value'. Provide the contract value before the escalation in `.oldContractValue` and the contract value after the escalation in `.newContractValue`",*Monitor `/contracts/value` for changes. Copy `/contracts/amendments/description` to `.description` and `/date` to `.date`.*,"contractingProcesses,contractingProcesses/id,contractingProcesses/summary,contractingProcesses/summary/modifications,contractingProcesses/summary/modifications/id,contractingProcesses/summary/modifications/date,contractingProcesses/summary/modifications/description,contractingProcesses/summary/modifications/type,contractingProcesses/summary/modifications/oldContractValue,contractingProcesses/summary/modifications/oldContractValue/amount,contractingProcesses/summary/modifications/oldContractValue/currency,contractingProcesses/summary/modifications/newContractValue,contractingProcesses/summary/modifications/newContractValue/amount,contractingProcesses/summary/modifications/newContractValue/currency",contractingProcesses/summary/modifications/type=’value’
+Variation to contract duration,Difference between original duration at contract award and the current projected duration in weeks.,"Contracting process: For each variation, publish an entry with a `.date` and `.description` in `.summary.modifications` with `.type` of 'duration'. Provide the contract period before the variation in `.oldContractPeriod` and the contract period after the variation in `.newContractPeriod`",*Monitor `/contracts/period` for changes. Copy `/contracts/amendments/description` to `.description` and `/date` to `.date`.*,"contractingProcesses,contractingProcesses/id,contractingProcesses/summary,contractingProcesses/summary/modifications,contractingProcesses/summary/modifications/id,contractingProcesses/summary/modifications/date,contractingProcesses/summary/modifications/description,contractingProcesses/summary/modifications/type,contractingProcesses/summary/modifications/oldContractPeriod,contractingProcesses/summary/modifications/oldContractPeriod/startDate,contractingProcesses/summary/modifications/oldContractPeriod/endDate,contractingProcesses/summary/modifications/newContractPeriod,contractingProcesses/summary/modifications/newContractPeriod/startDate,contractingProcesses/summary/modifications/newContractPeriod/endDate",contractingProcesses/summary/modifications/type=’date’
+Variation to contract scope,Any changes between original scope at contract award and the current scope,"Contracting process: For each variation, publish an entry with a `.date` and `.description` in `.summary.modifications` with `.type` of 'scope'","*Monitor `/contracts/description`, `/contracts/items` and `/contracts/implementation/milestones` for changes. Copy `/contracts/amendments/description` to `.description` and `/date` to `.date`.*","contractingProcesses,contractingProcesses/id,contractingProcesses/summary,contractingProcesses/summary/modifications,contractingProcesses/summary/modifications/id,contractingProcesses/summary/modifications/date,contractingProcesses/summary/modifications/description,contractingProcesses/summary/modifications/type",contractingProcesses/summary/modifications/type=’scope’
+Reasons for price changes,"Summary of reasons for primary changes (e.g. variations) that then lead to changes in contract price, major price fluctuations and accumulative increase or decrease in price.","Contracting process: For each variation, provide a `.rationale` in `.summary.modifications`.",*Copy `contracts/amendments/rationale` to `.rationale`.*,"contractingProcesses,contractingProcesses/id,contractingProcesses/summary,contractingProcesses/summary/modifications,contractingProcesses/summary/modifications/id,contractingProcesses/summary/modifications/rationale",contractingProcesses/summary/modifications/type=’value’
+Reasons for scope and duration changes,Summary of reasons for primary changes (e.g. variations) that then lead to changes in the scope and duration.,"Contracting process: For each variation, provide a `.rationale` in `.summary.modifications`.",*Copy `contracts/amendments/rationale` to `.rationale`.*,"contractingProcesses,contractingProcesses/id,contractingProcesses/summary,contractingProcesses/summary/modifications,contractingProcesses/summary/modifications/id,contractingProcesses/summary/modifications/rationale",contractingProcesses/summary/modifications/type=’scope’|’duration’