Skip to content

Commit

Permalink
multi: Fallback to os.cpu_count() on non UNIX OSes
Browse files Browse the repository at this point in the history
`os.sched_getaffinity()` is the only method in the StdLib that correctly
handles `taskset` restrictions etc. Nevertheless, it is only defined on
Linux. Therefore, on different OSes we fall back to `cpu_count()` and if
that is not available, we set `MAX_AVAILABLE_PROCESSES = 1`.

Further info:

    https://stackoverflow.com/questions/1006289/how-to-find-out-the-number-of-cpus-using-python

Fixes #35
  • Loading branch information
pmav99 committed Jun 29, 2022
1 parent b8eae95 commit d10f741
Show file tree
Hide file tree
Showing 3 changed files with 28 additions and 23 deletions.
34 changes: 17 additions & 17 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions requirements/requirements-dev.txt
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ html5lib==1.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (p
identify==2.5.1; python_version >= "3.7"
idna==3.3; python_version >= "3.7" and python_version < "4"
imagesize==1.3.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
importlib-metadata==4.11.4; python_version < "3.10" and python_version >= "3.7" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6")
importlib-metadata==4.12.0; python_version < "3.10" and python_version >= "3.7" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6")
iniconfig==1.1.1; python_version >= "3.7"
ipykernel==6.15.0; python_version >= "3.7"
ipython==8.4.0; python_version >= "3.8"
Expand All @@ -60,7 +60,7 @@ munch==2.5.0; python_version >= "3.7"
mypy-extensions==0.4.3; python_version >= "3.6"
mypy==0.950; python_version >= "3.6"
nest-asyncio==1.5.5; python_version >= "3.7"
nodeenv==1.6.0; python_version >= "3.7"
nodeenv==1.7.0; python_version >= "3.7" and python_full_version < "3.0.0" or python_full_version >= "3.7.0" and python_version >= "3.7"
numpy==1.23.0
packaging==21.3; python_version >= "3.8" and python_version < "4.0" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6")
pandas==1.4.3; python_version >= "3.8"
Expand All @@ -71,7 +71,7 @@ pickleshare==0.7.5; python_version >= "3.8"
platformdirs==2.5.2; python_full_version >= "3.7.2" and python_version >= "3.7" and python_version < "4.0"
pluggy==1.0.0; python_version >= "3.7"
pre-commit==2.19.0; python_version >= "3.7"
prompt-toolkit==3.0.29; python_full_version >= "3.6.2" and python_version >= "3.8"
prompt-toolkit==3.0.30; python_full_version >= "3.6.2" and python_version >= "3.8"
prospector==1.7.7; python_full_version >= "3.6.2" and python_version < "4.0"
psutil==5.9.1; python_version >= "3.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.7"
ptyprocess==0.7.0; sys_platform != "win32" and python_version >= "3.8"
Expand Down Expand Up @@ -123,12 +123,12 @@ tornado==6.1; python_version >= "3.7"
tqdm==4.64.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
traitlets==5.3.0; python_version >= "3.8"
typepigeon==1.0.14; python_version >= "3.6" and python_version < "4.0"
types-requests==2.27.31
types-requests==2.28.0
types-urllib3==1.26.15
typing-extensions==4.2.0; python_version >= "3.7" and python_full_version >= "3.7.2" and python_version < "3.10"
urllib3==1.26.9; python_version >= "3.7" and python_full_version < "3.0.0" and python_version < "4" or python_full_version >= "3.5.0" and python_version < "4" and python_version >= "3.7"
vcrpy==4.1.1; python_version >= "3.5"
virtualenv==20.14.1; python_version >= "3.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.7"
virtualenv==20.15.0; python_version >= "3.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.7"
wcwidth==0.2.5; python_full_version >= "3.6.2" and python_version >= "3.8"
webencodings==0.5.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
wrapt==1.14.1; python_full_version >= "3.7.2" and python_version >= "3.7" and python_version < "4.0"
Expand Down
7 changes: 6 additions & 1 deletion searvey/multi.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,12 @@


# https://docs.python.org/3/library/os.html#os.cpu_count
MAX_AVAILABLE_PROCESSES = len(os.sched_getaffinity(0))
try:
MAX_AVAILABLE_PROCESSES = len(os.sched_getaffinity(0))
except AttributeError:
MAX_AVAILABLE_PROCESSES = os.cpu_count()
if MAX_AVAILABLE_PROCESSES is None:
MAX_AVAILABLE_PROCESSES = 1


class FutureResult(pydantic.BaseModel):
Expand Down

0 comments on commit d10f741

Please sign in to comment.