Improve documentation about configuration and workflows #2919

WardLT · 2023-10-16T18:35:45Z

Description

Clarifies two problems users found in the documentation

error in parallel documentation #2915: Decorators are not needed for ParslPoolExecutor.map
Reloading default HighThroughputExecutor after parsl.clear() results in KeyError: 'block_id' exception #2871 : Configs may not be re-used
Fixes Docs: Further clarification on how serialization impacts where imports need to be #2918 : Explain when functions need imports
Fixes Allowing for inputs and outputs to take immutable types #2925 : Explain mutable types are undesirable

Type of change

Choose which options apply, and delete the ones which do not apply.

Documentation update

docs/userguide/configuring.rst

khk-globus

LGTM.

Andrew-S-Rosen · 2023-10-16T19:17:34Z

Thanks for this. One additional thing this helps with is that there was previously no mention of parsl.clear() in the docs.

WardLT · 2023-10-17T14:22:27Z

@Andrew-S-Rosen , do the new explanations about imports make sense to you?

Andrew-S-Rosen

@WardLT: This looks good to me! The point about the type-hints is especially subtle, so I'm glad that was highlighted.

There is some overlap thematically with the section above. It's not a problem, but I just wanted to bring that up as well.

Andrew-S-Rosen · 2023-10-17T15:29:33Z

docs/userguide/apps.rst

+Imports and Global Variables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Parsl apps have access to less information from the script that defined then


"that defined then" --> "that defined them"

Andrew-S-Rosen · 2023-10-17T15:29:48Z

docs/userguide/apps.rst

+
+Parsl apps have access to less information from the script that defined then
+than functions run via Python's native multiprocessing libraries.
+The reasons is that functions are executed on workers that


"The reasons is" --> "The reason is"

WardLT · 2023-10-17T15:44:11Z

Thanks! I'll fix my grammar and de-duplicate some. Better to keep the docs shorter

WardLT · 2023-10-17T16:19:22Z

There are actually a fair amount of duplication in this section. I'm going to clean it out while it's on my attention

- Deduplicated descriptions of rules for functions - Separated discussions of rules for function bodies from inputs/outputs - Made more section heads, bulleted lists for quicker scanning

Andrew-S-Rosen

Thanks for doing this, @WardLT! Here are some minor suggestions. Feel free to accept, reject, or do whatever with it.

Andrew-S-Rosen · 2023-10-17T21:58:43Z

docs/userguide/apps.rst

-or external Bash shell code that can be asynchronously executed.
+An **App** defines a computation that will be executed asynchronously by Parsl.
+Apps are Python functions marked with a decorator which
+designates that the function will run asynchronously and cause it return


"it return" --> "it to return"

Andrew-S-Rosen · 2023-10-17T21:59:06Z

docs/userguide/apps.rst

-Python apps encapsulate pure Python code, while Bash apps wrap calls to external applications and scripts,
-and Join apps allow composition of other apps to form sub-workflows.
+- ``@python_app``: Most Python functions
+- ``@bash_app``: A python function which returns a command line program to execute


"A python" --> "A Python"

Andrew-S-Rosen · 2023-10-17T22:00:01Z

docs/userguide/apps.rst

-and Join apps allow composition of other apps to form sub-workflows.
+- ``@python_app``: Most Python functions
+- ``@bash_app``: A python function which returns a command line program to execute
+- ``@join_app``: A function which returns launches other apps


"which returns launches" --> "which returns one or more Apps"

Andrew-S-Rosen · 2023-10-17T22:00:52Z

docs/userguide/apps.rst


-.. code-block:: python
+Python Apps run Python functions. The code inside a function marked by ``@python_app`` will run on a remote system.


"will run on a remote system" --- of course, in this example things are local. Might be better to say something like "is what will be executed either locally or on the remote system"

Great point.

Andrew-S-Rosen · 2023-10-17T22:07:13Z

docs/userguide/apps.rst


-Serializing Functions from Libraries
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The above rules do not apply for functions defined in Python modules.


Recommended change:

The above rules assume that the user is running the example code from a standalone script or Jupyter Notebook.

Functions that are defined in an installed Python module do not need to abide by these guidelines, as they are sent to workers differently than functions defined locally within a script.

That does read much better, thanks.

Andrew-S-Rosen · 2023-10-17T22:09:17Z

docs/userguide/apps.rst


-Parsl can create Apps directly from functions defined in Python modules.
-Supply the function as an argument to ``python_app`` rather than creating a new function which is decorated.
+Supply a module function as an argument to ``python_app`` rather than creating a new function which is decorated.


I mean, there is technically nothing stopping someone from defining a new function. It's just silly.

Suggestion:

To transform an imported function into an App, you do not need to create a new function to decorate. Rather, you can do the following:

Good call. I changed the wording to reflect that it isn't necessary to define an own function

Andrew-S-Rosen · 2023-10-17T22:10:23Z

docs/userguide/apps.rst

@@ -125,10 +118,67 @@ to include attributes from the original function (e.g., its name).
    my_max_app = python_app(my_max)


This is equivalent to doing:

.. code-block:: python

@python_app my_max_app(*args, **kwargs): return my_max(*args, **kwargs)

Very true. Nice way of tying it to something people would already think of

Andrew-S-Rosen · 2023-10-17T22:12:10Z

docs/userguide/apps.rst

+Special Keyword Arguments
+++++++++++++++++++++++++
+
+Parsl apps can use the following special reserved keyword arguments:


This can use an example to highlight that this is referring to the decorator kwargs.

Absolutely, and I also a added a section which differentiates "function" and "decorator" kwargs.

@Andrew-S-Rosen

- Grammar improvements - More examples - Differentiate kwargs for function and app Thanks, @Andrew-S-Rosen

benclifford · 2023-10-24T19:54:46Z

docs/userguide/configuring.rst

+    from parsl.config import Config
+    import parsl
+
+    def make_config() -> Config:


in tests and a few other places this is called fresh_config()

WardLT added 2 commits October 16, 2023 14:07

Mention that PPEx functions do not need decorators

b8f5062

Explain that configs may not be re-used

2dee78a

WardLT mentioned this pull request Oct 16, 2023

Reasonable Error Messages on Configuration Re-use #2920

Open

khk-globus reviewed Oct 16, 2023

View reviewed changes

docs/userguide/configuring.rst Outdated Show resolved Hide resolved

khk-globus approved these changes Oct 16, 2023

View reviewed changes

WardLT added 2 commits October 17, 2023 10:17

Add explanation about imports and global variables

bbcd64d

Fix grammar

2cfe760

Add a blank line

1b0b274

Andrew-S-Rosen reviewed Oct 17, 2023

View reviewed changes

WardLT added 2 commits October 17, 2023 14:18

Overall the App docs

3b0e1b7

- Deduplicated descriptions of rules for functions - Separated discussions of rules for function bodies from inputs/outputs - Made more section heads, bulleted lists for quicker scanning

Remove env var comment, style fixes

ebabc03

WardLT requested review from Andrew-S-Rosen and khk-globus October 17, 2023 21:55

Andrew-S-Rosen reviewed Oct 17, 2023

View reviewed changes

WardLT and others added 2 commits October 18, 2023 12:38

Clarifications

1256878

- Grammar improvements - More examples - Differentiate kwargs for function and app Thanks, @Andrew-S-Rosen

Merge branch 'master' into exec_docs

dfb0954

WardLT mentioned this pull request Oct 24, 2023

Allowing for inputs and outputs to take immutable types #2925

Closed

WardLT and others added 4 commits October 24, 2023 10:46

Improve documentation about mutable inputs

1bd83ca

Switch list default args to tuples

5c74969

Merge branch 'master' into exec_docs

f9ac3da

Merge branch 'master' into exec_docs

30a1672

benclifford reviewed Oct 24, 2023

View reviewed changes

benclifford merged commit a5aede7 into Parsl:master Oct 24, 2023
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve documentation about configuration and workflows #2919

Improve documentation about configuration and workflows #2919

WardLT commented Oct 16, 2023 •

edited

Loading

khk-globus left a comment

Andrew-S-Rosen commented Oct 16, 2023

WardLT commented Oct 17, 2023

Andrew-S-Rosen left a comment

Andrew-S-Rosen Oct 17, 2023

Andrew-S-Rosen Oct 17, 2023

WardLT commented Oct 17, 2023

WardLT commented Oct 17, 2023

Andrew-S-Rosen left a comment

Andrew-S-Rosen Oct 17, 2023

Andrew-S-Rosen Oct 17, 2023

Andrew-S-Rosen Oct 17, 2023

Andrew-S-Rosen Oct 17, 2023

WardLT Oct 18, 2023

Andrew-S-Rosen Oct 17, 2023

WardLT Oct 18, 2023

Andrew-S-Rosen Oct 17, 2023

WardLT Oct 18, 2023

Andrew-S-Rosen Oct 17, 2023

WardLT Oct 18, 2023

Andrew-S-Rosen Oct 17, 2023 •

edited

Loading

WardLT Oct 18, 2023

benclifford Oct 24, 2023


		.. code-block:: python
		Python Apps run Python functions. The code inside a function marked by ``@python_app`` will run on a remote system.

		@@ -125,10 +118,67 @@ to include attributes from the original function (e.g., its name).
		my_max_app = python_app(my_max)

Improve documentation about configuration and workflows #2919

Improve documentation about configuration and workflows #2919

Conversation

WardLT commented Oct 16, 2023 • edited Loading

Description

Type of change

khk-globus left a comment

Choose a reason for hiding this comment

Andrew-S-Rosen commented Oct 16, 2023

WardLT commented Oct 17, 2023

Andrew-S-Rosen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WardLT commented Oct 17, 2023

WardLT commented Oct 17, 2023

Andrew-S-Rosen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Andrew-S-Rosen Oct 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WardLT commented Oct 16, 2023 •

edited

Loading

Andrew-S-Rosen Oct 17, 2023 •

edited

Loading