XX

Signed-off-by: Matthew Ballance <[email protected]>

docs/Notes.md

@@ -160,4 +160,35 @@ package:
 # DV Flow Manager 1.0.0
 - Packages and Fragments
 - Package Import
-- 
+- Python implementation of YAML-defined tasks
+- Parameters and Properties
+- Modifying the value of input parameters
+- Strategy for tool-specific options
+- Tasks for reusable sim-image components (sim-image, library, ip)
+- Work-avoidance schemes for SV
+
+## Use Cases
+- Compile simulation image (debug, optimized)
+- Run user-specified simulations
+
+# DV Flow Manager 2.0.0
+- Package inheritance ("project templates")
+- VSCode log browser - visualize task graph ; navigate / view individual task logs
+  - See execution times
+  - See which tasks performed real work (output of 'changed')
+- Task/sub-graph execution delegation interface (implementation for LSF, SLURM, etc)
+- 
+
+# DV Flow Manager 3.0.0
+
+# DV Flow Manager 4.0.0
+- VSCode flow-file development support (error checking, completion, navigation)
+- VSCode run integration (launch run from VSCode ; monitor execution/completion)
+
+# DV Flow Manager 5.0.0
+- VSCode debug support (step through task graph ; inspect variable values)
+
+
+
+
+

docs/Roadmap.md

@@ -0,0 +1,75 @@
+
+# Flow Specification
+The Flow Specification is comprised of the Flow YAML Schema and the 
+semantic definition of how task graphs defined using the flow specification
+are evaluated.
+
+## 1.0.0
+- Package definition
+- Package import
+- Task definition
+  - "with" variable usage
+  - Operations on input and output data
+  - Operations on task parameters
+- Package fragments
+
+## 2.0.0
+- Parameterized package definition and use
+- Package "uses" (type/inheritance)
+- Task "with"-data definition (tasks can add their own parameters)
+- Task Groups / Sub-DAG
+
+## 3.0.0
+- JQ-based data extraction
+- YAML task templates / expansions
+- Support for annotating job requirements 
+
+# Library
+
+## 1.0.0
+- Std
+  - Null (combine dependencies, set variables). Implements tasks that do not specify 'uses'
+  - Exec
+  - FileSet
+  - PyClass - implement a task as a Python class (load from a module)
+
+- HdlSim
+  - Library  - creates a reusable simulator-specific library
+  - IP       - create a reusable single-module IP library
+  - SimImage - creates a simulation image 
+  - SimRun
+
+
+## 2.0.0
+- Std
+  - DefineData (Declares a data structure to pass)
+  - Export   - Captures the result of some task in a dedicated directory
+
+## 3.0.0
+- Std
+  - 
+
+# DV Flow Manager
+
+## 1.0.0
+- Simple 'max-jobs' process scheduler
+
+## 2.0.0
+- Progress meter and status line to monitor builds (non-verbose)
+- Multi-level mechanism for monitoring jobs
+  - High-level with a progress bar
+  - 
+- Log-creation feature that assembles a total log from per-task logs
+
+## 3.0.0
+- Provide link to source for error messages
+- Improve debug logging
+
+## 4.0.0
+- Improve status display by allowing actions to signal count-like information (+pass, +fail)
+- OpenTelemetry support
+
+
+## 5.0.0
+- Need some form of site settings
+

docs/quickstart.rst

@@ -60,16 +60,19 @@ simulate this module.
         tasks:
           - name: rtl
             type: std.FileSet
-            params:
+            with:
+              type: "systemVerilogSource"
               include: "*.sv"
 
           - name: sim-image
             type: hdl.sim.SimImage
-            depends: [rtl]
+            with:
+              - top: [top]
+            needs: [rtl]
 
           - name: sim-run
             type: hdl.sim.SimRun
-            depends: [sim-image]
+            needs: [sim-image]
 
 
 If we run the `dvfm run` command, DV Flow Manager will:
@@ -80,6 +83,52 @@ If we run the `dvfm run` command, DV Flow Manager will:
 
 .. code-block:: bash
 
-    % dvfm run
+    % dvfm run sim-run
 
-The command should complete successfully, and we should see the following
+This will compile the source, build a simulation image for module `top`,
+and run the resulting image. Not too bad for 20-odd lines of build specification.
+
+A Bit More Detail
+=================
+Let's break this down just a bit:
+
+.. code-block:: yaml
+
+    package:
+        name: my_design
+
+        imports:
+          - name: hdl.sim.vlt
+            as: hdl.sim
+
+DV Flow Manager views the world as a series of *packages* that reference each
+other and contain *tasks* to operate on sources within the *packages*.
+
+Here, we have declared a new package (my_design) and specified that it 
+references a built-in package named `hdl.sim.vlt`. This is a package that
+implements tasks for performing HDL simulation with the Verilator simulator.
+
+Note that we specify an alias (hdl.sim) for the package when importing it.
+This will allow us to easily swap in a different simulator without changing
+anything else within our package definition.
+
+.. code-block:: yaml
+    :highlight: 8,12
+
+    package:
+        name: my_design
+
+        imports:
+          - name: hdl.sim.vlt
+            as: hdl.sim
+
+        tasks:
+          - name: rtl
+            type: std.FileSet
+            with:
+              type: "systemVerilogSource"
+              include: "*.sv"
+
+Our first task is to specify the sources we want to process. This is done
+by specifying a `FileSet` task. The parameters of this task specify where
+the task should look for sources and which sources it should include

src/dv_flow_mgr/fragment_def.py

@@ -23,11 +23,9 @@
 import json
 from pydantic import BaseModel
 from typing import Any, Dict, List
-from .flow import Flow
 from .package import Package
-from .package_def import PackageImportSpec
-from .task import TaskParamCtor
-from .task_def import TaskDef, TaskSpec
+from .package_import_spec import PackageImportSpec
+from .task_def import TaskDef
 
 class FragmentDef(BaseModel):
     tasks : List[TaskDef] = dc.Field(default_factory=list)

src/dv_flow_mgr/package_def.py

@@ -24,36 +24,12 @@
 from pydantic import BaseModel
 from typing import Any, Dict, List
 from .flow import Flow
+from .fragment_def import FragmentDef
 from .package import Package
+from .package_import_spec import PackageImportSpec, PackageSpec
 from .task import TaskParamCtor
 from .task_def import TaskDef, TaskSpec
 
-@dc.dataclass
-class PackageSpec(object):
-    name : str
-    params : Dict[str,Any] = dc.Field(default_factory=dict)
-    _fullname : str = None
-
-    def get_fullname(self) -> str:
-        if self._fullname is None:
-            if len(self.params) != 0:
-                self._fullname = "%s%s}" % (
-                    self.name,
-                    json.dumps(self.params, separators=(',', ':')))
-            else:
-                self._fullname = self.name
-        return self._fullname    
-
-    def __hash__(self):
-        return hash(self.get_fullname())
-
-    def __eq__(self, value):
-        return isinstance(value, PackageSpec) and value.get_fullname() == self.get_fullname()
-
-@dc.dataclass
-class PackageImportSpec(PackageSpec):
-    path : str = dc.Field(default=None, alias="from")
-    alias : str = dc.Field(default=None, alias="as")
 
 class PackageDef(BaseModel):
     name : str
@@ -63,6 +39,8 @@ class PackageDef(BaseModel):
     imports : List[PackageImportSpec] = dc.Field(default_factory=list)
     fragments: List[str] = dc.Field(default_factory=list)
 
+    fragment_l : List['FragmentDef'] = dc.Field(default_factory=list, exclude=True)
+
 #    import_m : Dict['PackageSpec','Package'] = dc.Field(default_factory=dict)
 
     basedir : str = None
@@ -90,8 +68,31 @@ def mkPackage(self, session, params : Dict[str,Any] = None) -> 'Package':
                     basedir=self.basedir,
                     depend_refs=task.depends)
             else:
+                # We use the Null task from the std package
                 raise Exception("")
             ret.tasks[task.name] = ctor_t
 
+        for frag in self.fragment_l:
+            for task in frag.tasks:
+                if task.type is not None:
+                    # Find package (not package_def) that implements this task
+                    # Insert an indirect reference to that tasks's constructor
+
+                    # Only call getTaskCtor if the task is in a different package
+                    task_t = task.type if isinstance(task.type, TaskSpec) else TaskSpec(task.type)
+                    ctor_t = session.getTaskCtor(task_t, self)
+
+                    ctor_t = TaskParamCtor(
+                        base=ctor_t, 
+                        params=task.params, 
+                        basedir=frag.basedir,
+                        depend_refs=task.depends)
+                else:
+                    # We use the Null task from the std package
+                    raise Exception("")
+                if task.name in ret.tasks:
+                    raise Exception("Task %s already defined" % task.name)
+                ret.tasks[task.name] = ctor_t
+
         return ret
 

src/dv_flow_mgr/package_import_spec.py

@@ -0,0 +1,31 @@
+
+import pydantic.dataclasses as dc
+import json
+from typing import Dict, Any
+
+@dc.dataclass
+class PackageSpec(object):
+    name : str
+    params : Dict[str,Any] = dc.Field(default_factory=dict)
+    _fullname : str = None
+
+    def get_fullname(self) -> str:
+        if self._fullname is None:
+            if len(self.params) != 0:
+                self._fullname = "%s%s}" % (
+                    self.name,
+                    json.dumps(self.params, separators=(',', ':')))
+            else:
+                self._fullname = self.name
+        return self._fullname    
+
+    def __hash__(self):
+        return hash(self.get_fullname())
+
+    def __eq__(self, value):
+        return isinstance(value, PackageSpec) and value.get_fullname() == self.get_fullname()
+
+@dc.dataclass
+class PackageImportSpec(PackageSpec):
+    path : str = dc.Field(default=None, alias="from")
+    alias : str = dc.Field(default=None, alias="as")