Merge branch 'master' into benc-test-bad-reg-interchange-exit

.github/workflows/ci.yaml

@@ -11,7 +11,7 @@ jobs:
   main-test-suite:
     strategy:
       matrix:
-        python-version: ["3.9", "3.10", "3.11", "3.12"]
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
     runs-on: ubuntu-20.04
     timeout-minutes: 60
 

README.rst

@@ -1,6 +1,6 @@
 Parsl - Parallel Scripting Library
 ==================================
-|licence| |docs| |NSF-1550588| |NSF-1550476| |NSF-1550562| |NSF-1550528| |CZI-EOSS|
+|licence| |docs| |NSF-1550588| |NSF-1550476| |NSF-1550562| |NSF-1550528| |NumFOCUS| |CZI-EOSS| 
 
 Parsl extends parallelism in Python beyond a single computer.
 
@@ -64,6 +64,9 @@ then explore the `parallel computing patterns <https://parsl.readthedocs.io/en/s
 .. |CZI-EOSS| image:: https://chanzuckerberg.github.io/open-science/badges/CZI-EOSS.svg
    :target: https://czi.co/EOSS
    :alt: CZI's Essential Open Source Software for Science
+.. |NumFOCUS| image:: https://img.shields.io/badge/powered%20by-NumFOCUS-orange.svg?style=flat&colorA=E1523D&colorB=007D8A
+    :target: https://numfocus.org
+    :alt: Powered by NumFOCUS
 
 
 Quickstart

docs/devguide/roadmap.rst

@@ -15,7 +15,7 @@ Code Maintenance
 * **Type Annotations and Static Type Checking**: Add static type annotations throughout the codebase and add typeguard checks.
 * **Release Process**: `Improve the overall release process <https://github.com/Parsl/parsl/issues?q=is%3Aopen+is%3Aissue+label%3Arelease_process>`_ to synchronize docs and code releases, automatically produce changelog documentation.
 * **Components Maturity Model**: Defines the `component maturity model <https://github.com/Parsl/parsl/issues/2554>`_ and tags components with their appropriate maturity level.
-* **Define and Document Interfaces**: Identify and document interfaces via which `external components <https://parsl.readthedocs.io/en/stable/userguide/plugins.html>`_ can augment the Parsl ecosystem.
+* **Define and Document Interfaces**: Identify and document interfaces via which `external components <https://parsl.readthedocs.io/en/stable/userguide/advanced/plugins.html>`_ can augment the Parsl ecosystem.
 * **Distributed Testing Process**: All tests should be run against all possible schedulers, using different executors, on a variety of remote systems. Explore the use of containerized schedulers and remote testing on real systems.
 
 New Features and Integrations

docs/index.rst

@@ -4,7 +4,7 @@ Parsl - Parallel Scripting Library
 Parsl extends parallelism in Python beyond a single computer.
 
 You can use Parsl
-`just like Python's parallel executors <userguide/workflow.html#parallel-workflows-with-loops>`_
+`just like Python's parallel executors <userguide/workflows/workflow.html#parallel-workflows-with-loops>`_
 but across *multiple cores and nodes*.
 However, the real power of Parsl is in expressing multi-step workflows of functions.
 Parsl lets you chain functions together and will launch each function as inputs and computing resources are available.
@@ -37,8 +37,8 @@ Parsl lets you chain functions together and will launch each function as inputs
 
 
 Start with the `configuration quickstart <quickstart.html#getting-started>`_ to learn how to tell Parsl how to use your computing resource,
-see if `a template configuration for your supercomputer <userguide/configuring.html>`_ is already available,
-then explore the `parallel computing patterns <userguide/workflow.html>`_ to determine how to use parallelism best in your application.
+see if `a template configuration for your supercomputer <userguide/configuration/examples.html>`_ is already available,
+then explore the `parallel computing patterns <userguide/workflows/workflows.html>`_ to determine how to use parallelism best in your application.
 
 Parsl is an open-source code, and available on GitHub: https://github.com/parsl/parsl/
 
@@ -57,7 +57,7 @@ Parsl works everywhere
 
 *Parsl can run parallel functions on a laptop and the world's fastest supercomputers.*
 Scaling from laptop to supercomputer is often as simple as changing the resource configuration.
-Parsl is tested `on many of the top supercomputers <userguide/configuring.html>`_.
+Parsl is tested `on many of the top supercomputers <userguide/configuration/examples.html>`_.
 
 Parsl is flexible
 -----------------
@@ -107,7 +107,7 @@ Table of Contents
    quickstart
    1-parsl-introduction.ipynb
    userguide/index
-   userguide/glossary  
+   userguide/glossary
    faq
    reference
    devguide/index

docs/quickstart.rst

@@ -70,7 +70,7 @@ We describe these components briefly here, and link to more details in the `User
 
 .. note::
 
-    Parsl's documentation includes `templates for many supercomputers <userguide/configuring.html>`_.
+    Parsl's documentation includes `templates for many supercomputers <userguide/configuring/examples.html>`_.
     Even though you may not need to write a configuration from a blank slate,
     understanding the basic terminology below will be very useful.
 
@@ -112,7 +112,7 @@ with hello world Python and Bash apps.
     with open('hello-stdout', 'r') as f:
         print(f.read())
 
-Learn more about the types of Apps and their options `here <userguide/apps.html>`__.
+Learn more about the types of Apps and their options `here <userguide/apps/index.html>`__.
 
 Executors
 ^^^^^^^^^
@@ -127,7 +127,7 @@ You can dynamically set the number of workers based on available memory and
 pin each worker to specific GPUs or CPU cores
 among other powerful features.
 
-Learn more about Executors `here <userguide/execution.html#executors>`__.
+Learn more about Executors `here <userguide/configuration/execution.html#executors>`__.
 
 Execution Providers
 ^^^^^^^^^^^^^^^^^^^
@@ -141,7 +141,7 @@ Another key role of Providers is defining how to start an Executor on a remote c
 Often, this simply involves specifying the correct Python environment and
 (described below) how to launch the Executor on each acquired computers.
 
-Learn more about Providers `here <userguide/execution.html#execution-providers>`__.
+Learn more about Providers `here <userguide/configuration/execution.html#execution-providers>`__.
 
 Launchers
 ^^^^^^^^^
@@ -151,7 +151,7 @@ A common example is an :class:`~parsl.launchers.launchers.MPILauncher`, which us
 for starting a single program on multiple computing nodes.
 Like Providers, Parsl comes packaged with Launchers for most supercomputers and clouds.
 
-Learn more about Launchers `here <userguide/execution.html#launchers>`__.
+Learn more about Launchers `here <userguide/configuration/execution.html#launchers>`__.
 
 
 Benefits of a Data-Flow Kernel
@@ -164,7 +164,7 @@ and performs the many other functions needed to execute complex workflows.
 The flexibility and performance of the DFK enables applications with
 intricate dependencies between tasks to execute on thousands of parallel workers.
 
-Start with the Tutorial or the `parallel patterns <userguide/workflow.html>`_
+Start with the Tutorial or the `parallel patterns <userguide/workflows/workflow.html>`_
 to see the complex types of workflows you can make with Parsl.
 
 Starting Parsl
@@ -176,12 +176,17 @@ This script runs on a system that must stay on-line until all of your tasks comp
 much computing power, such as the login node for a supercomputer.
 
 The :class:`~parsl.config.Config` object holds definitions of Executors and the Providers and Launchers they rely on.
-An example which launches 512 workers on 128 nodes of the Polaris supercomputer looks like
+An example which launches 4 workers on 1 node of the Polaris supercomputer looks like
 
 .. code-block:: python
 
+    from parsl import Config
+    from parsl.executors import HighThroughputExecutor
+    from parsl.providers import PBSProProvider
+    from parsl.launchers import MpiExecLauncher
+
     config = Config(
-        retires=1,  # Restart task if they fail once
+        retries=1,  # Restart task if they fail once
         executors=[
             HighThroughputExecutor(
                 available_accelerators=4,  # Maps one worker per GPU
@@ -191,21 +196,21 @@ An example which launches 512 workers on 128 nodes of the Polaris supercomputer
                     account="example",
                     worker_init="module load conda; conda activate parsl",
                     walltime="1:00:00",
-                    queue="prod",
+                    queue="debug",
                     scheduler_options="#PBS -l filesystems=home:eagle",  # Change if data on other filesystem
                     launcher=MpiExecLauncher(
                         bind_cmd="--cpu-bind", overrides="--depth=64 --ppn 1"
                     ),  # Ensures 1 manger per node and allows it to divide work to all 64 cores
                     select_options="ngpus=4",
-                    nodes_per_block=128,
+                    nodes_per_block=1,
                     cpus_per_node=64,
                 ),
             ),
         ]
     )
 
 
-The documentation has examples for other supercomputers `here <userguide/configuring.html>`__.
+The documentation has examples for other supercomputers `here <userguide/configuration/examples.html>`_.
 
 The next step is to load the configuration
 

docs/reference.rst

@@ -32,6 +32,8 @@ Configuration
     parsl.addresses.address_by_interface
     parsl.addresses.address_by_query
     parsl.addresses.address_by_route
+    parsl.addresses.get_all_addresses
+    parsl.addresses.get_any_address
     parsl.utils.get_all_checkpoints
     parsl.utils.get_last_checkpoint
 
@@ -169,8 +171,6 @@ Exceptions
     parsl.providers.errors.ScaleOutFailed
     parsl.providers.errors.SchedulerMissingArgs
     parsl.providers.errors.ScriptPathError
-    parsl.channels.errors.ChannelError
-    parsl.channels.errors.FileCopyException
     parsl.executors.high_throughput.errors.WorkerLost
     parsl.executors.high_throughput.interchange.ManagerLost
     parsl.serialize.errors.DeserializationError
@@ -195,3 +195,13 @@ Internal
     parsl.jobs.job_status_poller.JobStatusPoller
     parsl.jobs.strategy.Strategy
     parsl.utils.Timer
+
+Task Vine configuration
+=======================
+
+.. autosummary::
+    :toctree: stubs
+    :nosignatures:
+
+    parsl.executors.taskvine.TaskVineManagerConfig
+    parsl.executors.taskvine.TaskVineFactoryConfig

docs/userguide/advanced/examples/library/app.py

@@ -0,0 +1,9 @@
+"""Functions used as part of the workflow"""
+from typing import List, Tuple
+
+from .logic import convert_to_binary
+
+
+def convert_many_to_binary(xs: List[int]) -> List[Tuple[bool, ...]]:
+    """Convert a list of nonnegative integers to binary"""
+    return [convert_to_binary(x) for x in xs]

docs/userguide/advanced/examples/library/config.py

@@ -0,0 +1,23 @@
+from parsl.config import Config
+from parsl.executors import HighThroughputExecutor
+from parsl.providers import LocalProvider
+
+
+def make_local_config(cores_per_worker: int = 1) -> Config:
+    """Generate a configuration which runs all tasks on the local system
+
+    Args:
+        cores_per_worker: Number of cores to dedicate for each task
+    Returns:
+        Configuration object with the requested settings
+    """
+    return Config(
+        executors=[
+            HighThroughputExecutor(
+                label="htex_local",
+                cores_per_worker=cores_per_worker,
+                cpu_affinity='block',
+                provider=LocalProvider(),
+            )
+        ],
+    )

docs/userguide/advanced/examples/library/logic.py

@@ -0,0 +1,15 @@
+from typing import Tuple
+
+
+def convert_to_binary(x: int) -> Tuple[bool, ...]:
+    """Convert a nonnegative integer into a binary
+
+    Args:
+        x: Number to be converted
+    Returns:
+        The binary number represented as list of booleans
+    """
+    if x < 0:
+        raise ValueError('`x` must be nonnegative')
+    bin_as_string = bin(x)
+    return tuple(i == '1' for i in bin_as_string[2:])

docs/userguide/advanced/examples/pyproject.toml

@@ -0,0 +1,7 @@
+[project]
+name = "library"
+version = '0.0.0'
+description = 'Example library for Parsl documentation'
+
+[tool.setuptools.packages.find]
+include = ['library*']

docs/userguide/advanced/examples/run.py

@@ -0,0 +1,35 @@
+from argparse import ArgumentParser
+
+import parsl
+
+from library.config import make_local_config
+from library.app import convert_many_to_binary
+from parsl.app.python import PythonApp
+
+# Protect the script from running twice.
+#  See "Safe importing of main module" in Python multiprocessing docs
+#  https://docs.python.org/3/library/multiprocessing.html#multiprocessing-programming
+if __name__ == "__main__":
+    # Get user instructions
+    parser = ArgumentParser()
+    parser.add_argument('--numbers-per-batch', default=8, type=int)
+    parser.add_argument('numbers', nargs='+', type=int)
+    args = parser.parse_args()
+
+    # Prepare the workflow functions
+    convert_app = PythonApp(convert_many_to_binary, cache=False)
+
+    # Load the configuration
+    #  As a context manager so resources are shutdown on exit
+    with parsl.load(make_local_config()):
+
+        # Spawn tasks
+        futures = [
+            convert_app(args.numbers[start:start + args.numbers_per_batch])
+            for start in range(0, len(args.numbers), args.numbers_per_batch)
+        ]
+
+        # Retrieve task results
+        for future in futures:
+            for x, b in zip(future.task_record['args'][0], future.result()):
+                print(f'{x} -> {"".join("1" if i else "0" for i in b)}')

docs/userguide/advanced/index.rst

@@ -0,0 +1,13 @@
+Advanced Topics
+===============
+
+More to learn about Parsl after starting a project.
+
+.. toctree::
+   :maxdepth: 2
+
+   modularizing
+   usage_tracking
+   monitoring
+   parsl_perf
+   plugins