[Superset] Superset Docker Swarm files

Docker-Swarm-deployment/analytics/superset/Dockerfile

@@ -0,0 +1,10 @@
+FROM ghcr.io/datakaveri/superset:4.0.2-5
+USER root
+RUN apt-get update && apt-get install -y \
+    gcc \
+    python3-dev \
+    libpq-dev \
+    && rm -rf /var/lib/apt/lists/* 
+RUN pip3 install psycopg2
+USER 1000
+

Docker-Swarm-deployment/analytics/superset/README.md

@@ -0,0 +1,28 @@
+## Getting started with Superset(visualization tool) using docker swarm Deploy
+
+#### To deploy superset dashboard:
+```sh
+docker stack deploy -c superset-stack.yaml superset
+```
+#### To deploy superset middleware:
+```sh
+docker stack deploy -c superset-middleware-stack.yaml superset-middleware
+```
+#### To Check the status :
+```sh
+docker service ls
+
+ID             NAME                                  MODE             REPLICAS              IMAGE                                                     PORTS
+
+7ztp4yx1d1gc   superset_redis                        replicated       1/1                   redis:7                                                   
+k2lkdttsrgrw   superset_superset                     replicated       1/1                   ghcr.io/datakaveri/superset:4.0.2-1                       *:8088->8088/tcp
+ijzzqgxx8rd1   superset_superset-worker              replicated       1/1                   ghcr.io/datakaveri/superset:4.0.2-1                       
+x1ojkx3smg0y   superset_superset-worker-beat         replicated       1/1                   ghcr.io/datakaveri/superset:4.0.2-1                       
+rv2yw340gsd0   superset_superset_init                replicated       0/1                   ghcr.io/datakaveri/superset:4.0.2-1                       
+```
+
+**superset_superset_init** service will be down once it performs bootstrap operations.
+
+##### NOTE: 
+1. To install custom modules add them in docker/requirements-local.txt file and redeploy the stack
+2. Replace all placeholder in .env files under secrets/ and gra-superset-guesttoken-middleware/

Docker-Swarm-deployment/analytics/superset/docker/README.md

@@ -0,0 +1,75 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting Started with Superset using Docker
+
+Docker is an easy way to get started with Superset.
+
+## Prerequisites
+
+1. [Docker](https://www.docker.com/get-started)
+2. [Docker Compose](https://docs.docker.com/compose/install/)
+
+## Configuration
+
+The `/app/pythonpath` folder is mounted from [`./docker/pythonpath_dev`](./pythonpath_dev)
+which contains a base configuration [`./docker/pythonpath_dev/superset_config.py`](./pythonpath_dev/superset_config.py)
+intended for use with local development.
+
+### Local overrides
+
+In order to override configuration settings locally, simply make a copy of [`./docker/pythonpath_dev/superset_config_local.example`](./pythonpath_dev/superset_config_local.example)
+into `./docker/pythonpath_dev/superset_config_docker.py` (git ignored) and fill in your overrides.
+
+### Local packages
+
+If you want to add Python packages in order to test things like databases locally, you can simply add a local requirements.txt (`./docker/requirements-local.txt`)
+and rebuild your Docker stack.
+
+Steps:
+
+1. Create `./docker/requirements-local.txt`
+2. Add your new packages
+3. Rebuild docker compose
+    1. `docker compose down -v`
+    2. `docker compose up`
+
+## Initializing Database
+
+The database will initialize itself upon startup via the init container ([`superset-init`](./docker-init.sh)). This may take a minute.
+
+## Normal Operation
+
+To run the container, simply run: `docker compose up`
+
+After waiting several minutes for Superset initialization to finish, you can open a browser and view [`http://localhost:8088`](http://localhost:8088)
+to start your journey.
+
+## Developing
+
+While running, the container server will reload on modification of the Superset Python and JavaScript source code.
+Don't forget to reload the page to take the new frontend into account though.
+
+## Production
+
+It is possible to run Superset in non-development mode by using [`docker-compose-non-dev.yml`](../docker-compose-non-dev.yml). This file excludes the volumes needed for development and uses [`./docker/.env-non-dev`](./.env-non-dev) which sets the variable `SUPERSET_ENV` to `production`.
+
+## Resource Constraints
+
+If you are attempting to build on macOS and it exits with 137 you need to increase your Docker resources. See instructions [here](https://docs.docker.com/docker-for-mac/#advanced) (search for memory)

Docker-Swarm-deployment/analytics/superset/docker/docker-bootstrap.sh

@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -eo pipefail
+
+REQUIREMENTS_LOCAL="/app/docker/requirements-local.txt"
+# If Cypress run – overwrite the password for admin and export env variables
+if [ "$CYPRESS_CONFIG" == "true" ]; then
+    export SUPERSET_CONFIG=tests.integration_tests.superset_test_config
+    export SUPERSET_TESTENV=true
+    export SUPERSET__SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://superset:superset@db:5432/superset
+fi
+#
+# Make sure we have dev requirements installed
+#
+if [ -f "${REQUIREMENTS_LOCAL}" ]; then
+  echo "Installing local overrides at ${REQUIREMENTS_LOCAL}"
+  pip install --no-cache-dir -r "${REQUIREMENTS_LOCAL}"
+else
+  echo "Skipping local overrides"
+fi
+
+case "${1}" in
+  worker)
+    echo "Starting Celery worker..."
+    # setting up only 2 workers by default to contain memory usage in dev environments
+    celery --app=superset.tasks.celery_app:app worker -O fair -l INFO --concurrency=${CELERYD_CONCURRENCY:-2}
+    ;;
+  beat)
+    echo "Starting Celery beat..."
+    rm -f /tmp/celerybeat.pid
+    celery --app=superset.tasks.celery_app:app beat --pidfile /tmp/celerybeat.pid -l INFO -s "${SUPERSET_HOME}"/celerybeat-schedule
+    ;;
+  app)
+    echo "Starting web app (using development server)..."
+    flask run -p 8088 --with-threads --reload --debugger --host=0.0.0.0
+    ;;
+  app-gunicorn)
+    echo "Starting web app..."
+    /usr/bin/run-server.sh
+    ;;
+  *)
+    echo "Unknown Operation!!!"
+    ;;
+esac

Docker-Swarm-deployment/analytics/superset/docker/docker-ci.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+/app/docker/docker-init.sh
+
+# TODO: copy config overrides from ENV vars
+
+# TODO: run celery in detached state
+export SERVER_THREADS_AMOUNT=8
+# start up the web server
+
+/usr/bin/run-server.sh

Docker-Swarm-deployment/analytics/superset/docker/docker-frontend.sh

@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -e
+
+# Packages needed for puppeteer:
+if [ "$PUPPETEER_SKIP_CHROMIUM_DOWNLOAD" = "false" ]; then
+    apt update
+    apt install -y chromium
+fi
+
+if [ "$BUILD_SUPERSET_FRONTEND_IN_DOCKER" = "true" ]; then
+    echo "Building Superset frontend in dev mode inside docker container"
+    cd /app/superset-frontend
+
+    echo "Running `npm install`"
+    npm install
+
+    echo "Running frontend"
+    npm run dev
+
+else
+    echo "Skipping frontend build steps - YOU NEED TO RUN IT MANUALLY ON THE HOST!"
+    echo "https://superset.apache.org/docs/contributing/development/#webpack-dev-server"
+fi

Docker-Swarm-deployment/analytics/superset/docker/docker-init.sh

@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -e
+
+#
+# Always install local overrides first
+#
+/app/docker/docker-bootstrap.sh
+
+STEP_CNT=4
+
+echo_step() {
+cat <<EOF
+
+######################################################################
+
+
+Init Step ${1}/${STEP_CNT} [${2}] -- ${3}
+
+
+######################################################################
+
+EOF
+}
+ADMIN_PASSWORD="${ADMIN_PASSWORD:-admin}"
+# If Cypress run – overwrite the password for admin and export env variables
+if [ "$CYPRESS_CONFIG" == "true" ]; then
+    ADMIN_PASSWORD="general"
+    export SUPERSET_CONFIG=tests.integration_tests.superset_test_config
+    export SUPERSET_TESTENV=true
+    export SUPERSET__SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://superset:superset@db:5432/superset
+fi
+# Initialize the database
+echo_step "1" "Starting" "Applying DB migrations"
+superset db upgrade
+echo_step "1" "Complete" "Applying DB migrations"
+
+# Create an admin user
+echo_step "2" "Starting" "Setting up admin user ( admin / $ADMIN_PASSWORD )"
+superset fab create-admin \
+              --username admin \
+              --firstname Superset \
+              --lastname Admin \
+              --email [email protected] \
+              --password "$ADMIN_PASSWORD"
+echo_step "2" "Complete" "Setting up admin user"
+# Create default roles and permissions
+echo_step "3" "Starting" "Setting up roles and perms"
+superset init
+echo_step "3" "Complete" "Setting up roles and perms"
+
+if [ "$SUPERSET_LOAD_EXAMPLES" = "yes" ]; then
+    # Load some data to play with
+    echo_step "4" "Starting" "Loading examples"
+    # If Cypress run which consumes superset_test_config – load required data for tests
+    if [ "$CYPRESS_CONFIG" == "true" ]; then
+        superset load_test_users
+        superset load_examples --load-test-data
+    else
+        superset load_examples --force
+    fi
+    echo_step "4" "Complete" "Loading examples"
+fi

Docker-Swarm-deployment/analytics/superset/docker/frontend-mem-nag.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+# We need at least 3GB of free mem...
+MIN_MEM_FREE_GB=3
+MIN_MEM_FREE_KB=$(($MIN_MEM_FREE_GB*1000000))
+
+echo_mem_warn() {
+  MEM_FREE_KB=$(awk '/MemFree/ { printf "%s \n", $2 }' /proc/meminfo)
+  MEM_FREE_GB=$(awk '/MemFree/ { printf "%s \n", $2/1024/1024 }' /proc/meminfo)
+
+  if [[ "${MEM_FREE_KB}" -lt "${MIN_MEM_FREE_KB}" ]]; then
+    cat <<EOF
+    ===============================================
+    ========  Memory Insufficient Warning =========
+    ===============================================
+
+    It looks like you only have ${MEM_FREE_GB}GB of
+    memory free. Please increase your Docker
+    resources to at least ${MIN_MEM_FREE_GB}GB
+
+    ===============================================
+    ========  Memory Insufficient Warning =========
+    ===============================================
+EOF
+  else
+    echo "Memory check Ok [${MEM_FREE_GB}GB free]"
+  fi
+}
+
+# Always nag if they're low on mem...
+echo_mem_warn