Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DH-66] move conda and R install locations, fix popcon deployment path, install temp version of popcon #5170

Merged
merged 15 commits into from
Nov 1, 2023
Merged
1 change: 1 addition & 0 deletions deployments/a11y/image/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ dependencies:
# https://github.com/berkeley-dsep-infra/datahub/issues/3693
- notebook==7.0.2
- jupyterlab==4.0.4
- git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# ###
# The items below are from infra-requirements, however lab conflicts with the
# alpha notebook.
Expand Down
2 changes: 1 addition & 1 deletion deployments/a11y/image/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
7 changes: 3 additions & 4 deletions deployments/astro/image/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@ dependencies:
- protobuf
- mpich
- mpi4py

- scikit-learn==1.0.*
- scikit-image==0.19.*
- scipy==1.11.3
- pip

- pip:
Expand All @@ -33,9 +35,6 @@ dependencies:
- opencv-python>=4.5.5.62
- nose2>=0.10.0
- sympy
- scipy
- scikit-learn==1.0.*
- scikit-image==0.19.*
- beautifulsoup4
- tqdm
- h5py
Expand Down
4 changes: 2 additions & 2 deletions deployments/biology/image/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ ENV DEBIAN_FRONTEND=noninteractive
ENV NB_USER jovyan
ENV NB_UID 1000

ENV CONDA_DIR /opt/conda
ENV R_LIBS_USER /opt/r
ENV CONDA_DIR /srv/conda
ENV R_LIBS_USER /srv/r

# Explicitly add littler to PATH
# See https://github.com/conda-forge/r-littler-feedstock/issues/6
Expand Down
3 changes: 3 additions & 0 deletions deployments/biology/image/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ dependencies:
# compbio BioE c146, Fall 22 and into the future, issue 3785
- scikit-learn=1.1.2

# bug w/notebook and traitlets: https://github.com/jupyter/notebook/issues/7048
- traitlets=5.9.*

# For https://github.com/berkeley-dsep-infra/datahub/issues/1846
# Conda does not have these
- pip:
Expand Down
2 changes: 1 addition & 1 deletion deployments/biology/image/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
2 changes: 1 addition & 1 deletion deployments/cee/image/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
2 changes: 1 addition & 1 deletion deployments/data100/image/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,4 +43,4 @@ dependencies:
- nbconvert==7.6.0
- pytest-notebook==0.8.1
- gh-scoped-creds==4.1
- popularity-contest==0.4.1
- git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
2 changes: 1 addition & 1 deletion deployments/data100/image/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
2 changes: 1 addition & 1 deletion deployments/data101/image/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -101,4 +101,4 @@ dependencies:
- pytest-notebook==0.8.1
- pymongo==4.4.1
- dbt-postgres==1.6.0
- popularity-contest==0.4.1
- git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
2 changes: 1 addition & 1 deletion deployments/data101/image/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
2 changes: 1 addition & 1 deletion deployments/data102/image/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,4 @@ dependencies:
- gh-scoped-creds==4.1
- nb2pdf==0.6.2
- nbpdfexport==0.2.1
- popularity-contest==0.4.1
- git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
2 changes: 1 addition & 1 deletion deployments/data102/image/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
4 changes: 2 additions & 2 deletions deployments/data8/image/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
FROM buildpack-deps:focal-scm

ENV CONDA_DIR /opt/conda
ENV CONDA_DIR /srv/conda

# Set up common env variables
ENV TZ=America/Los_Angeles
Expand All @@ -17,7 +17,7 @@ RUN adduser --disabled-password --gecos "Default Jupyter user" ${NB_USER}

# Create user owned R libs dir
# This lets users temporarily install packages
ENV R_LIBS_USER /opt/r
ENV R_LIBS_USER /srv/r
RUN install -d -o ${NB_USER} -g ${NB_USER} ${R_LIBS_USER}

RUN apt-get -qq update --yes && \
Expand Down
3 changes: 3 additions & 0 deletions deployments/data8/image/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ dependencies:
# Spring 2023 https://github.com/berkeley-dsep-infra/datahub/issues/4291
- ipykernel==6.19.4

# bug w/notebook and traitlets: https://github.com/jupyter/notebook/issues/7048
- traitlets=5.9.*

# Items not in conda forge
- pip:
# Export notebooks as PDFs with Chrome
Expand Down
6 changes: 3 additions & 3 deletions deployments/datahub/images/default/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ ENV DEBIAN_FRONTEND=noninteractive
ENV NB_USER jovyan
ENV NB_UID 1000

ENV CONDA_DIR /opt/conda
ENV R_LIBS_USER /opt/r
ENV CONDA_DIR /srv/conda
ENV R_LIBS_USER /srv/r

RUN apt-get -qq update --yes && \
apt-get -qq install --yes locales && \
Expand All @@ -36,7 +36,7 @@ RUN install -d -o ${NB_USER} -g ${NB_USER} ${R_LIBS_USER}

# Install R.
# These packages must be installed into the base stage since they are in system
# paths rather than /opt.
# paths rather than /srv.
# Pre-built R packages from rspm are built against system libs in jammy.
ENV R_VERSION=4.3.1-1.2204.0
ENV LITTLER_VERSION=0.3.18-2.2204.0
Expand Down
2 changes: 1 addition & 1 deletion deployments/datahub/images/default/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
20 changes: 10 additions & 10 deletions deployments/dev-r/images/default/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ ENV DEBIAN_FRONTEND=noninteractive
ENV NB_USER jovyan
ENV NB_UID 1000
# These are used by the python, R, and final stages
ENV CONDA_DIR /opt/conda
ENV R_LIBS_USER /opt/r
ENV CONDA_DIR /srv/conda
ENV R_LIBS_USER /srv/r

RUN apt-get -qq update --yes && \
apt-get -qq install --yes locales && \
Expand All @@ -32,7 +32,7 @@ RUN apt-get -qq update --yes && \

# Install R.
# These packages must be installed into the base stage since they are in system
# paths rather than /opt.
# paths rather than /srv.
# Pre-built R packages from rspm are built against system libs in jammy.
ENV R_VERSION=4.3.1-1.2204.0
ENV LITTLER_VERSION=0.3.18-2.2204.0
Expand Down Expand Up @@ -73,8 +73,8 @@ RUN sed -i -e '/^R_LIBS_USER=/s/^/#/' /etc/R/Renviron && \
echo "TZ=${TZ}" >> /etc/R/Renviron

# =============================================================================
# This stage exists to build /opt/r.
FROM base as opt-r
# This stage exists to build /srv/r.
FROM base as srv-r

# Create user owned R libs dir
# This lets users temporarily install packages
Expand All @@ -96,8 +96,8 @@ COPY install.R /tmp/install.R
RUN /tmp/install.R && rm -rf /tmp/downloaded_packages

# =============================================================================
# This stage exists to build /opt/conda.
FROM base as opt-conda
# This stage exists to build /srv/conda.
FROM base as srv-conda

COPY install-mambaforge.bash /tmp/install-mambaforge.bash
RUN /tmp/install-mambaforge.bash
Expand All @@ -114,10 +114,10 @@ RUN mamba env update -p ${CONDA_DIR} -f /tmp/environment.yml && \
mamba clean -afy

# =============================================================================
# This stage consumes base and import /opt/r and /opt/conda.
# This stage consumes base and import /srv/r and /srv/conda.
FROM base as final
COPY --from=opt-r /opt/r /opt/r
COPY --from=opt-conda /opt/conda /opt/conda
COPY --from=srv-r /srv/r /srv/r
COPY --from=srv-conda /srv/conda /srv/conda

# Install IR kernelspec. Requires python and R.
ENV PATH ${CONDA_DIR}/bin:${PATH}:${R_LIBS_USER}/bin
Expand Down
2 changes: 2 additions & 0 deletions deployments/dev-r/images/default/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ dependencies:
- python=3.10.*
- pip=23.1.*
- nodejs=18.*
# bug w/notebook and traitlets: https://github.com/jupyter/notebook/issues/7048
- traitlets=5.9.*

- jupyter-server-proxy==4.0.0
- jupyter-rsession-proxy==2.2.0
Expand Down
2 changes: 1 addition & 1 deletion deployments/dev-r/images/default/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
3 changes: 3 additions & 0 deletions deployments/dev-r/images/secondary/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ dependencies:

# for jupyter-tree-download
- zip==3.0

# bug w/notebook and traitlets: https://github.com/jupyter/notebook/issues/7048
- traitlets=5.9.*
- pip:
- -r infra-requirements.txt
- jupyter-shiny-proxy==1.1
2 changes: 1 addition & 1 deletion deployments/dev-r/images/secondary/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
2 changes: 1 addition & 1 deletion deployments/eecs/image/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
2 changes: 1 addition & 1 deletion deployments/ischool/image/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
2 changes: 1 addition & 1 deletion deployments/publichealth/image/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
2 changes: 1 addition & 1 deletion deployments/shiny/image/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
2 changes: 1 addition & 1 deletion deployments/stat159/image/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
2 changes: 1 addition & 1 deletion deployments/stat20/image/infra-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down
2 changes: 1 addition & 1 deletion hub/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ apiVersion: v1
appVersion: '1.0'
description: Deployment Chart for JupyterHub
name: hub
version: 0.0.1-0.dev.git.6847.h37aa9975
version: 0.0.1-0.dev.git.6956.h5e23ef5e
2 changes: 1 addition & 1 deletion hub/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ jupyterhub:
# https://github.com/ipython/ipython/issues/9163
db_file: ":memory:"
popularity-contest:
mountPath: /opt/conda/etc/ipython/startup/000-popularity-contest.py
mountPath: /etc/ipython/startup/000-popularity-contest.py
stringData: |
import popularity_contest.reporter
popularity_contest.reporter.setup_reporter()
Expand Down
2 changes: 1 addition & 1 deletion node-placeholder/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.0.1-0.dev.git.6821.hfab4a361
version: 0.0.1-0.dev.git.6946.hdccbf2f4

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
Expand Down
2 changes: 1 addition & 1 deletion node-placeholder/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ image:
repository: us-central1-docker.pkg.dev/ucb-datahub-2018/core/node-placeholder-scaler
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: "0.0.1-0.dev.git.6607.h11654cae"
tag: "0.0.1-0.dev.git.6946.hdccbf2f4"

imagePullSecrets: []
nameOverride: ""
Expand Down
2 changes: 1 addition & 1 deletion scripts/infra-packages/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
popularity-contest==git+https://github.com/yuvipanda/python-popularity-contest@9e30ec414e45615e75a0e86424141b7093c8f686
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
Expand Down