Merge pull request #14 from NTIA/refactor_dependencies

Dependencies and Renaming

.pre-commit-config.yaml

@@ -1,30 +1,31 @@
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v2.2.3
+    rev: v4.1.0
     hooks:
       - id: trailing-whitespace
       - id: end-of-file-fixer
       - id: check-docstring-first
       - id: check-yaml
       - id: debug-statements
   - repo: https://github.com/asottile/seed-isort-config
-    rev: v1.9.1
+    rev: v2.2.0
     hooks:
       - id: seed-isort-config
-        language_version: python3.6
+        language_version: python3.7
   - repo: https://github.com/pre-commit/mirrors-isort
-    rev: v4.3.20
+    rev: v5.10.1
     hooks:
       - id: isort
-        language_version: python3.6
+        language_version: python3.7
   - repo: https://github.com/ambv/black
-    rev: stable
+    rev: 22.1.0
     hooks:
       - id: black
-        language_version: python3.6
-  - repo: https://github.com/markdownlint/markdownlint
-    rev: v0.9.0
-    hooks:
-      - id: markdownlint
-        args: [-s, .ml_style.rb, README.md]
-        exclude: GitHubRepoPublicReleaseApproval.md|LICENSE.md
+        language_version: python3.7
+  # TODO markdownlint broken
+  # - repo: https://github.com/markdownlint/markdownlint
+  #   rev: v0.9.0
+  #   hooks:
+  #     - id: markdownlint
+  #       args: [-s, .ml_style.rb, README.md]
+  #       exclude: GitHubRepoPublicReleaseApproval.md|LICENSE.md

README.md

@@ -10,8 +10,8 @@ https://github.com/NTIA/scos-sensor/blob/master/README.md#actions-and-hardware-s
 ) sections which explain the scos-sensor plugin architecture.
 
 This repository includes many 700MHz band actions in [scos_usrp/configs/actions](
-scos_usrp/configs/actions). Action classes, RadioInterface, GPSInterface, and signals
-are used from [scos_actions](https://github.com/NTIA/scos-actions).
+scos_usrp/configs/actions). Action classes, SignalAnalyzerInterface, GPSInterface, and
+signals are used from [scos_actions](https://github.com/NTIA/scos-actions).
 
 For information on adding actions, see the [scos_actions documentation](
 https://github.com/NTIA/scos-actions/blob/master/README.md#adding-actions).
@@ -30,8 +30,8 @@ https://github.com/NTIA/scos-actions/blob/master/README.md#adding-actions).
   initialize the USRP supported actions and sample calibration files.
 - scos_usrp/discover: This includes the code to read yaml files and make actions
   available to scos-sensor.
-- scos_usrp/hardware: This includes the USRP implementation of the radio interface and
-  GPS interface. It also includes supporting calibration and test code.
+- scos_usrp/hardware: This includes the USRP implementation of the signal analyzer
+  interface and GPS interface. It also includes supporting calibration and test code.
 
 ## Running in scos-sensor
 
@@ -56,20 +56,53 @@ Below are steps to run scos-sensor with the scos-usrp plugin:
 ### Requirements and Configuration
 
 Requires pip>=18.1 (upgrade using `python3 -m pip install --upgrade pip`) and
-python>=3.6.
+python>=3.7.
 
 It is highly recommended that you first initialize a virtual development environment
 using a tool such a `conda` or `venv`. The following commands create a virtual
 environment using `venv` and install the required dependencies for development and
 testing.
 
-```python
+```bash
 python3 -m venv ./venv
 source venv/bin/activate
 python3 -m pip install --upgrade pip # upgrade to pip>=18.1
 python3 -m pip install -r requirements-dev.txt
 ```
 
+#### Using pip-tools
+
+It is recommended to keep direct dependencies in a separate file. The direct
+dependencies are in the requirements.in and requirements-dev.in files. Then pip-tools
+can be used to generate files with all the dependencies and transitive dependencies
+(sub-dependencies). The files containing all the dependencies are in requirements.txt
+and requirements-dev.txt. Run the following in the virtual environment to install
+pip-tools.
+
+```bash
+python -m pip install pip-tools
+```
+
+To update requirements.txt after modifying requirements.in:
+
+```bash
+pip-compile requirements.in
+```
+
+To update requirements-dev.txt after modifying requirements.in or requirements-dev.in:
+
+```bash
+pip-compile requirements-dev.in
+```
+
+Use pip-sync to match virtual environment to requirements-dev.txt:
+
+```bash
+pip-sync requirements.txt requirements-dev.txt
+```
+
+For more information about pip-tools, see <https://pip-tools.readthedocs.io/en/latest/#>
+
 ### Running Tests
 
 Since the UHD drivers are required, a docker container is used for testing. [Install

docker/Dockerfile-test

@@ -6,10 +6,10 @@ RUN apt-get update -q && \
                    libboost-date-time1.65.1 libboost-filesystem1.65.1 \
                    libboost-python1.65.1 libboost-regex1.65.1 \
                    libboost-serialization1.65.1 libboost-system1.65.1 \
-                   libboost-thread1.65.1 libusb-1.0-0 libpython3.6 \
+                   libboost-thread1.65.1 libusb-1.0-0 libpython3.7 \
                    libboost-program-options1.65.1 \
-                   git python3-setuptools python3-pip python3-numpy \
-		   swig gcc python3-dev && \
+                   git python3.7 \
+		   swig gcc python3.7-dev && \
     apt-get clean && rm -rf /var/lib/apt/lists/*
 
 ENV PYTHONUNBUFFERED 1
@@ -18,7 +18,7 @@ WORKDIR /src
 COPY ./ /src
 
 ARG DOCKER_GIT_CREDENTIALS
-RUN python3 -m pip install --upgrade pip
-RUN pip3 install --no-cache-dir -r requirements-dev.txt
+RUN python3.7 -m pip install --upgrade pip
+RUN python3.7 -m pip install --no-cache-dir -r requirements-dev.txt
 
 ENTRYPOINT ["pytest"]

docker/Dockerfile-uhd

@@ -38,8 +38,8 @@ RUN apt-get update -q && \
                    libboost-python1.65.1 libboost-regex1.65.1 \
                    libboost-serialization1.65.1 libboost-system1.65.1 \
                    libboost-thread1.65.1 libboost-program-options1.65.1 \
-                   libusb-1.0-0 libpython3.6 python3-setuptools python3-pip \
-                   python3-numpy && \
+                   libusb-1.0-0 libpython3.7 \
+                    && \
     apt-get clean && rm -rf /var/lib/apt/lists/*
 
-CMD /usr/bin/python3
+CMD /usr/bin/python3.7

requirements-dev.in

@@ -0,0 +1,5 @@
+-rrequirements.txt
+
+black>=22.0, <=23.0
+pre-commit>=2.0, <=3.0
+pytest>=7.0, <8.0

requirements-dev.txt

@@ -1,29 +1,156 @@
--rrequirements.txt
-
-appdirs==1.4.4
-attrs==20.2.0
-black==19.10b0
-cfgv==3.2.0
-click==7.1.2
-distlib==0.3.1
-filelock==3.0.12
-identify==1.5.5
-importlib-metadata==2.0.0
-importlib-resources==3.0.0
-more-itertools==8.5.0
-nodeenv==1.5.0
-packaging==20.4
-pathspec==0.8.0
-pluggy==0.13.1
-pre-commit==2.6.0
-py==1.9.0
-pyparsing==2.4.7
-pytest==5.4.1
-PyYAML==5.4
-regex==2020.9.27
-six==1.15.0
-toml==0.10.1
-typed-ast==1.4.1
-virtualenv==20.0.33
-wcwidth==0.2.5
-zipp==3.3.0
+#
+# This file is autogenerated by pip-compile with python 3.7
+# To update, run:
+#
+#    pip-compile requirements-dev.in
+#
+asgiref==3.5.0
+    # via
+    #   -r requirements.txt
+    #   django
+attrs==21.4.0
+    # via pytest
+black==22.3.0
+    # via -r requirements-dev.in
+certifi==2021.10.8
+    # via
+    #   -r requirements.txt
+    #   requests
+cfgv==3.3.1
+    # via pre-commit
+charset-normalizer==2.0.12
+    # via
+    #   -r requirements.txt
+    #   requests
+click==8.1.2
+    # via black
+distlib==0.3.4
+    # via virtualenv
+django==3.2.12
+    # via
+    #   -r requirements.txt
+    #   scos-actions
+environs==9.5.0
+    # via -r requirements.txt
+filelock==3.6.0
+    # via virtualenv
+identify==2.4.12
+    # via pre-commit
+idna==3.3
+    # via
+    #   -r requirements.txt
+    #   requests
+importlib-metadata==4.11.3
+    # via
+    #   click
+    #   pluggy
+    #   pre-commit
+    #   pytest
+    #   virtualenv
+iniconfig==1.1.1
+    # via pytest
+its-preselector @ git+https://github.com/NTIA/[email protected]
+    # via
+    #   -r requirements.txt
+    #   scos-actions
+marshmallow==3.15.0
+    # via
+    #   -r requirements.txt
+    #   environs
+mypy-extensions==0.4.3
+    # via black
+nodeenv==1.6.0
+    # via pre-commit
+numpy==1.21.5
+    # via
+    #   -r requirements.txt
+    #   scipy
+    #   scos-actions
+packaging==21.3
+    # via
+    #   -r requirements.txt
+    #   marshmallow
+    #   pytest
+pathspec==0.9.0
+    # via black
+platformdirs==2.5.1
+    # via
+    #   black
+    #   virtualenv
+pluggy==1.0.0
+    # via pytest
+pre-commit==2.18.1
+    # via -r requirements-dev.in
+py==1.11.0
+    # via pytest
+pyparsing==3.0.7
+    # via
+    #   -r requirements.txt
+    #   packaging
+pytest==7.1.1
+    # via -r requirements-dev.in
+python-dateutil==2.8.2
+    # via
+    #   -r requirements.txt
+    #   scos-actions
+python-dotenv==0.20.0
+    # via
+    #   -r requirements.txt
+    #   environs
+pytz==2022.1
+    # via
+    #   -r requirements.txt
+    #   django
+pyyaml==6.0
+    # via pre-commit
+requests==2.27.1
+    # via -r requirements.txt
+ruamel-yaml==0.17.21
+    # via
+    #   -r requirements.txt
+    #   scos-actions
+ruamel-yaml-clib==0.2.6
+    # via
+    #   -r requirements.txt
+    #   ruamel-yaml
+scipy==1.7.3
+    # via
+    #   -r requirements.txt
+    #   scos-actions
+scos_actions @ git+https://github.com/NTIA/[email protected]
+    # via -r requirements.txt
+sigmf @ git+https://github.com/NTIA/SigMF.git@multi-recording-archive
+    # via
+    #   -r requirements.txt
+    #   scos-actions
+six==1.16.0
+    # via
+    #   -r requirements.txt
+    #   python-dateutil
+    #   virtualenv
+sqlparse==0.4.2
+    # via
+    #   -r requirements.txt
+    #   django
+toml==0.10.2
+    # via pre-commit
+tomli==2.0.1
+    # via
+    #   black
+    #   pytest
+typed-ast==1.5.2
+    # via black
+typing-extensions==4.1.1
+    # via
+    #   -r requirements.txt
+    #   asgiref
+    #   black
+    #   importlib-metadata
+urllib3==1.26.9
+    # via
+    #   -r requirements.txt
+    #   requests
+virtualenv==20.14.0
+    # via pre-commit
+zipp==3.8.0
+    # via importlib-metadata

requirements.in

@@ -0,0 +1,4 @@
+django>=3.1.13, <4.0
+environs>=9.0, <10.0
+numpy>=1.0, <2.0
+scos_actions @ git+https://github.com/NTIA/[email protected]