Merge branch 'master' into jenkins/zshkoor/setup-py-updated-7862129

.github/workflows/ci.yml

@@ -12,16 +12,17 @@ jobs:
   run_tests:
     name: tests
     runs-on: ${{ matrix.os }}
+    continue-on-error: true
     strategy:
       matrix:
-        os: [ubuntu-20.04]
-        python-version: ['3.8']
-        toxenv: [docs, quality, django32, django42]
+        os: [ubuntu-latest]
+        python-version: ['3.12']
+        toxenv: [docs, quality, django42]
 
     steps:
-    - uses: actions/checkout@v1
+    - uses: actions/checkout@v4
     - name: setup python
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
 
@@ -37,8 +38,9 @@ jobs:
       run: tox
 
     - name: Run coverage
-      if: matrix.python-version == '3.8' && matrix.toxenv == 'django42'
-      uses: codecov/codecov-action@v3
+      if: matrix.python-version == '3.12' && matrix.toxenv == 'django42'
+      uses: codecov/codecov-action@v4
       with:
+        token: ${{ secrets.CODECOV_TOKEN }}
         flags: unittests
         fail_ci_if_error: true

.github/workflows/pypi-publish.yml

@@ -8,15 +8,15 @@ on:
 jobs:
 
   push:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
 
     steps:
       - name: Checkout
-        uses: actions/checkout@v1
+        uses: actions/checkout@v4
       - name: setup python
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v5
         with:
-          python-version: 3.8
+          python-version: 3.12
 
       - name: Install pip
         run: pip install -r requirements/pip.txt
@@ -25,7 +25,7 @@ jobs:
         run: python setup.py sdist bdist_wheel
 
       - name: Publish to PyPi
-        uses: pypa/gh-action-pypi-publish@master
+        uses: pypa/gh-action-pypi-publish@release/v1
         with:
           user: __token__
           password: ${{ secrets.PYPI_UPLOAD_TOKEN }}

.gitignore

@@ -22,6 +22,7 @@ lib64
 
 # Installer logs
 pip-log.txt
+venv
 
 # Unit test / coverage reports
 .cache/
@@ -69,9 +70,6 @@ docs/edx_django_utils.*.rst
 requirements/private.in
 requirements/private.txt
 
-# tox environment temporary artifacts
-tests/__init__.py
-
 # Development task artifacts
 default.db
 

CHANGELOG.rst

@@ -11,6 +11,117 @@ Change Log
 
 .. There should always be an "Unreleased" section for changes pending release.
 
+7.1.0 - 2024-12-05
+------------------
+Added
+~~~~~
+* Added signals monitoring_support_process_request, monitoring_support_process_response, and monitoring_support_process_exception to the MonitoringSupportMiddleware to enable plugging in monitoring capabilities.
+
+7.0.1 - 2024-11-21
+------------------
+Fixed
+~~~~~
+* Fix bug where code owner custom attributes were being defined, even when the CODE_OWNER_MAPPINGS setting was not defined.
+
+7.0.0 - 2024-10-16
+------------------
+Removed
+~~~~~~~
+* Remove unused ``background_task`` monitoring function.
+* Remove ``get_current_transaction`` (used internally only) from the public API.
+
+[6.1.0] - 2024-10-15
+---------------------
+Changed
+~~~~~~~
+* Added Datadog implementation of  ``set_monitoring_transaction_name`` and refactored the functionality.
+
+[6.0.0] - 2024-10-09
+---------------------
+Added
+~~~~~
+* Added support for python3.12
+* Dropped support for python<3.12 versions
+
+[5.16.0] - 2024-09-27
+---------------------
+Added
+~~~~~
+* Added a new method to backends for ``tag_root_span_with_error`` and added Datadog implementation of the functionality.
+* Uses the new method to tag the root span when processing exceptions.
+
+Changed
+~~~~~~~
+* Renamed ``CachedCustomMonitoringMiddleware`` to ``MonitoringSupportMiddleware`` and deprecated the old name. It will be removed in a future release.
+
+[5.15.0] - 2024-07-29
+---------------------
+Added
+~~~~~
+* Added Datadog implementation of ``function_trace`` and allowed implementation to be configurable.
+
+[5.14.2] - 2024-05-31
+---------------------
+Fixed
+~~~~~
+* FrontendMonitoringMiddleware now updates the Content-Length header if its already set.
+* FrontendMonitoringMiddleware will now be enabled once waffle switch named ``edx_django_utils.monitoring.enable_frontend_monitoring_middleware`` is enabled.
+
+[5.14.1] - 2024-05-22
+---------------------
+Fixed
+~~~~~
+* Added default value while getting content-type header to avoid KeyError.
+
+[5.14.0] - 2024-05-22
+---------------------
+Added
+~~~~~
+* Added middleware named ``FrontendMonitoringMiddleware`` for inserting frontend monitoring HTML script tags to response, configured by new Django setting ``OPENEDX_TELEMETRY_FRONTEND_SCRIPTS``.
+
+[5.13.0] - 2024-04-30
+---------------------
+Added
+~~~~~
+* Initial support for sending monitoring data to OpenTelemetry collector or Datadog agent, configured by new Django setting ``OPENEDX_TELEMETRY``. See monitoring README for details.
+
+[5.12.0] - 2024-03-29
+---------------------
+Added
+~~~~~
+* Added support for ``Python 3.11``
+
+[5.11.0] - 2024-03-06
+---------------------
+Added
+~~~~~
+* Added support for ``Python 3.12``
+
+Removed
+~~~~~~~
+* Dropped support for ``Django 3.2``
+
+[5.10.1] - 2024-01-17
+---------------------
+
+Added
+~~~~~
+* Added manufacture_data management command
+
+[5.9.0] - 2023-11-27
+--------------------
+
+Removed
+~~~~~~~
+* Removed ``edx_django_utils.cache.disable_forced_cache_miss_for_none`` which was added in ``5.7.0``.
+
+[5.8.0] - 2023-11-03
+--------------------
+
+Changed
+~~~~~~~
+* Adjusted ``get_plugin_apps`` to log at info level rather than debug and with more detail, though with a comment that this may not actually end up logging.
+
 [5.7.0] - 2023-08-04
 --------------------
 

Makefile

@@ -74,8 +74,7 @@ test: clean ## run tests in the current virtualenv
 diff_cover: test ## find diff lines that need test coverage
 	diff-cover coverage.xml
 
-test-all: ## run tests on every supported Python/Django combination
-	tox -e quality
+test-all: clean ## run quality checks as well as tests on every supported Python/Django combination
 	tox
 
 validate: quality test ## run tests and quality checks

README.rst

@@ -30,6 +30,8 @@ This repository includes shared utilities for:
 
 * `Security Utilities`_: Includes a middleware to add CSP response headers.
 
+* `Data Generation`_: Management command for generating Django data based on model factories.
+
 .. _Cache Utilities: edx_django_utils/cache/README.rst
 
 .. _Django User and Group Utilities: edx_django_utils/user/README.rst
@@ -44,61 +46,17 @@ This repository includes shared utilities for:
 
 .. _Security Utilities: edx_django_utils/security/README.rst
 
+.. _Data Generation: edx_django_utils/data_generation/README.rst
+
 Documentation
 -------------
 
 The full documentation is in the docs directory, and is published to https://edx-django-utils.readthedocs.org.
 
-Getting Started
----------------
-
-One Time Setup
-~~~~~~~~~~~~~~
-.. code-block::
-
-  # clone the repo
-  git clone [email protected]:edx/edx-django-utils.git
-  cd edx-django-utils
-
-  # setup a virtualenv using virtualenvwrapper with the same name as the repo and activate it.
-  # $(basename $(pwd)) will give you the name of the current working directory, in this case ``edx-django-utils``
-  mkvirtualenv -p python3 $(basename $(pwd))
-
-
-Every time you develop something in this repo
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-.. code-block::
-
-  # Activate the virtualenv.
-  workon edx-django-utils
-
-  # Grab the latest code.
-  git checkout master
-  git pull
-
-  # Install the dev requirements
-  make requirements
-
-  # Run the tests
-  make test
-
-  # Make a new branch for your changes
-  git checkout -b <your_github_username>/<short_description>
-
-  # Using your favorite editor, edit the code to make your change.
-  vim …
-
-  # Run your new tests
-  pytest ./path/to/new/tests
-
-  # Run all the test
-  make test
-
-  # Commit all your changes
-  git commit …
-  git push
+Getting Started with Development
+--------------------------------
 
-  # Open a PR and ask for review.
+Please see the Open edX documentation for `guidance on Python development <https://docs.openedx.org/en/latest/developers/how-tos/get-ready-for-python-dev.html>`_ in this repo.
 
 Design Pattern followed by packages
 -----------------------------------
@@ -160,7 +118,7 @@ file in this repo.
 Reporting Security Issues
 -------------------------
 
-Please do not report security issues in public. Please email security@edx.org.
+Please do not report security issues in public. Please email security@openedx.org.
 
 License
 -------

catalog-info.yaml

@@ -13,6 +13,6 @@ metadata:
   annotations:
     openedx.org/arch-interest-groups: ""
 spec:
-  owner: group:arch-bom
+  owner: group:2u-arch-bom
   type: 'library'
   lifecycle: 'production'

docs/decisions/0006-content-security-policy-middleware.rst

@@ -17,14 +17,18 @@ Context
 - Gather reports of violations, allowing deployers to discover vulnerabilities even while CSP prevents them from being exploited
 - Enforce the use of a business process for adding new external scripts to the site
 
-At the most basic level, CSP allows the server to tell the browser to refuse to load any Javascript that isn't on an allowlist. More advanced deployments can restrict other kinds of resources (including service workers, child frames, images, and XHR connections). A static set of headers can only support an allowlist that is based on domain names and paths, but features such as ``strict-dynamic`` and ``nonce`` allow the server to vouch for scripts individually. This is more secure but requires more integration with application code.
+The most basic use of CSP is to allow the server to tell the browser to refuse to load any Javascript that isn't on an allowlist. More complete deployments can restrict other kinds of resources (including service workers, child frames, images, and XHR connections). A static set of headers can only support an allowlist that is based on domain names and paths, but features such as ``strict-dynamic`` and ``nonce`` allow the server to vouch for ``<script>`` elements individually. This is more secure but requires more integration with application code.
 
 If fully deployed, there is a reduction of risk to learners, partners, and the site's reputation. However, for full effect, these security headers must be returned on all responses, including non-HTML resources and error pages. CSP headers must also be tailored not only to each IDA and MFE, but to each *deployment*, as different deployers will load images and scripts from different domains.
 
 CSP policies are often very long (1 kB in one example) and are built from a number of directives that each contain an allowlist or configuration. They cannot be loosened by the addition of later headers, but only tightened. This means that the server must return the entire policy in one shot, or alternatively emit some directives as separate headers. The use of ``strict-dynamic`` can shrink the header and make the policy more flexible, but requires integration between body and headers: The headers must either contain hashes of the scripts, or nonces that are also referenced in the HTML.
 
 Microfrontends are generally served from static file hosting such as Amazon S3 and cannot take advantage of ``strict-dynamic``; they are limited to source allowlists set by the fileserver or a CDN. In contrast, Django-based IDAs could use ``strict-dynamic`` (via nonces and hashes) because of the ability to customize header and body contents for each response.
 
+Different views also have different needs. For example, some of our views need to allow framing by arbitrary other websites, but by default we want all other pages to have a restrictive ``frame-ancestors`` directive. Because CSP is purely additive, this means we can't take the approach of returning one default header with a restrictive ``frame-ancestors`` and then a second header to override it for this handful of views; unlike ``X-Frame-Options``, CSP does not have a concept of overrides. Instead, these views need to return a different version of the default header. String manipulation in a post-response middleware is unpalatable, so an ideally flexible solution might involve views registering their individual needs with a framework that then compiles the results into a final header (with all overrides applied at the individual directive level). Mozilla's `django-csp package <https://github.com/mozilla/django-csp>`__ takes this approach.
+
+Because it's easy to accidentally break the entire website by use of an overly restrictive CSP, there is a way to try out a policy without enforcing it, via the ``Content-Security-Policy-Report-Only`` header. This runs the same policy but only reports violations to a specified server rather than enforcing the policy. For example, if you have a policy that restricts to loading scripts from domains A, B, C, and D and you want to see if you can remove D from that list, you would deploy the pair ``Content-Security-Policy: script-src A B C D`` and ``Content-Security-Policy-Report-Only: script-src A B C`` and check for reports about domain D. This reporting mechanism is critical for low-impact rollouts, so any CSP solution needs to accomodate it.
+
 As of April 2023, most MFEs and other IDAs include inline Javascript, which severely limits the protection that CSP can provide. However, an initial deployment of CSP that locks down script source domains to an allowlist and reports on the locations of remaining inline scripts would be a good start towards cleaning up these inline scripts. In other words, the goal here is to get *started* on CSP.
 
 Decision
@@ -37,7 +41,7 @@ We will enable this middleware for all Django-based IDAs.
 Consequences
 ************
 
-Django-based IDAs will for now be restricted to using static headers, and will not support ``strict-dynamic``. This is equivalent in configurability to having a CDN attach headers, and is good enough for a first pass, but will almost certainly need to be replaced with a more integrated system later as IDAs are adjusted to become amenable to ``strict-dynamic``.
+Django-based IDAs will for now be restricted to using static headers, and will not support ``strict-dynamic`` or view-specific variation. This is equivalent in configurability to having a CDN attach headers, and is good enough for a first pass, but will almost certainly need to be replaced with a more integrated system later as IDAs are adjusted to become amenable to ``strict-dynamic`` and we've already collected all the gains we can achieve with service-wide policies.
 
 Rejected Alternatives
 *********************

edx_django_utils/__init__.py

@@ -2,7 +2,7 @@
 EdX utilities for Django Application development..
 """
 
-__version__ = "5.7.0"
+__version__ = "7.1.0"
 
 default_app_config = (
     "edx_django_utils.apps.EdxDjangoUtilsConfig"

edx_django_utils/cache/README.rst

@@ -3,7 +3,7 @@ Cache Utils
 
 Cache utilities that implement `OEP-0022: Caching in Django`_.
 
-.. _`OEP-0022: Caching in Django`: https://github.com/openedx/open-edx-proposals/blob/master/oeps/oep-0022-bp-django-caches.rst
+.. _`OEP-0022: Caching in Django`: https://github.com/openedx/open-edx-proposals/blob/master/oeps/best-practices/oep-0022-bp-django-caches.rst
 
 get_cache_key
 -------------