-MLHub is based on [Jupyterhub](https://github.com/jupyterhub/jupyterhub). MLHub allows to create and manage multiple [workspaces](https://github.com/ml-tooling/ml-workspace), for example to distribute them to a group of people or within a team.
+MLHub is based on [Jupyterhub](https://github.com/jupyterhub/jupyterhub) with complete focus on Docker and Kubernetes. MLHub allows to create and manage multiple [workspaces](https://github.com/ml-tooling/ml-workspace), for example to distribute them to a group of people or within a team.
## Highlights
@@ -33,6 +33,7 @@ MLHub is based on [Jupyterhub](https://github.com/jupyterhub/jupyterhub). MLHub
- 🖊️ Set configuration parameters such as CPU-limits for started workspaces.
- 🖥 Access additional tools within the started workspaces by having secured routes.
- 🎛 Tunnel SSH connections to workspace containers.
+- 🐳 Focused on Docker and Kubernetes with enhanced functionality.
## Getting Started
@@ -61,7 +62,10 @@ For Kubernetes deployment, we forked and modified [zero-to-jupyterhub-k8s](https
### Configuration
-In the default config, a user named `admin` can register and access the hub. If you use a different authenticator, you might want to set a different user as initial admin user as well.
+#### Default Login
+
+When using the default config - so leaving the Jupyterhub config `c.Authenticator.admin_users` as it is -, a user named `admin` can access the hub with admin rights. If you use the default `NativeAuthenticator` as authenticator, youc must register the user `admin` with a password of your choice first before login in.
+If you use a different authenticator, you might want to set a different user as initial admin user as well, for example in case of using oauth you want to set `c.Authenticator.admin_users` to a username returned by the oauth login.
#### Environment Variables
@@ -81,6 +85,18 @@ Here are the additional environment variables for the hub:
mlhub
+
+
EXECUTION_MODE
+
Defines in which execution mode the hub is running in. Value is one of [docker | k8s]
+
local
+
+
+
CLEANUP_INTERVAL_SECONDS
+
+ Interval in which expired and not-used resources are deleted. Set to -1 to disable the automatic cleanup. For more information, see Section Cleanup Service.
+
+
3600
+
SSL_ENABLED
Enable SSL. If you don't provide an ssl certificate as described in Section "Enable SSL/HTTPS", certificates will be generated automatically. As this auto-generated certificate is not signed, you have to trust it in the browser. Without ssl enabled, ssh access won't work as the container uses a single port and has to tell https and ssh traffic apart.
@@ -114,6 +130,8 @@ Here are the additional environment variables for the hub:
#### Jupyterhub Config
+##### Docker-local
+
Jupyterhub itself is configured via a `config.py` file. In case of MLHub, a default config file is stored under `/resources/jupyterhub_config.py`. If you want to override settings or set extra ones, you can put another config file under `/resources/jupyterhub_user_config.py`. Following settings should probably not be overriden:
- `c.Spawner.environment` - we set default variables there. Instead of overriding it, you can add extra variables to the existing dict, e.g. via `c.Spawner.environment["myvar"] = "myvalue"`.
- `c.DockerSpawner.prefix` and `c.DockerSpawner.name_template` - if you change those, check whether your SSH environment variables permit those names a target. Also, think about setting `c.Authenticator.username_pattern` to prevent a user having a username that is also a valid container name.
@@ -199,6 +217,15 @@ The "Days to live" flag is purely informational currently and can be seen in the
+### Cleanup Service
+
+JupyterHub was originally not created with Docker or Kubernetes in mind, which can result in unfavorable scenarios such as that containers are stopped but not deleted on the host. Furthermore, our custom spawners might create some artifacts that should be cleaned up as well. MLHub contains a cleanup service that is started as a [JupyterHub service](https://jupyterhub.readthedocs.io/en/stable/reference/services.html) inside the hub container. It can be accessed as a REST-API by an admin, but it is also triggered automatically every X timesteps when not disabled (see config for `CLEANUP_INTERVAL_SECONDS`). The service enhances the JupyterHub functionality with regards to the Docker and Kubernetes world. "Containers" is hereby used interchangeably for Docker containers and Kubernetes pods.
+The service has two endpoints which can be reached under the Hub service url `/services/cleanup-service/*` with admin permissions.
+
+- `GET /services/cleanup-service/users`: This endpoint is currently doing anything only in Docker-local mode. There, it will check for resources of deleted users, so users who are not in the JupyterHub database anymore, and delete them. This includes containers, networks, and volumes. This is done by looking for labeled Docker resources that point to containers started by hub and belonging to the specific users.
+
+- `GET /services/cleanup-service/expired`: When starting a named workspace, an expiration date can be assigned to it. This endpoint will delete all containers that are expired. The respective named server is deleted from the JupyterHub database and also the Docker/Kubernetes resource is deleted.
+
## Contribution
- Pull requests are encouraged and always welcome. Read [`CONTRIBUTING.md`](https://github.com/ml-tooling/ml-hub/tree/master/CONTRIBUTING.md) and check out [help-wanted](https://github.com/ml-tooling/ml-hub/issues?utf8=%E2%9C%93&q=is%3Aopen+is%3Aissue+label%3A"help+wanted"+sort%3Areactions-%2B1-desc+) issues.
diff --git a/resources/jupyterhub-mod/cleanup-service.py b/resources/jupyterhub-mod/cleanup-service.py
new file mode 100644
index 0000000..a0435e8
--- /dev/null
+++ b/resources/jupyterhub-mod/cleanup-service.py
@@ -0,0 +1,266 @@
+"""
+Web service that is supposed to be started via JupyterHub.
+By this, the service has access to some information passed
+by JupyterHub. For more information check out https://jupyterhub.readthedocs.io/en/stable/reference/services.html
+
+Note: Logs probably don't appear in stdout, as the service is started as a subprocess by JupyterHub
+"""
+
+import os
+import urllib3
+import json
+import time
+import math
+from threading import Thread
+import logging
+
+from tornado import web, ioloop
+from jupyterhub.services.auth import HubAuthenticated
+
+import docker.errors
+from kubernetes import client, config, stream
+
+from mlhubspawner import utils
+
+# Environment variables passed by JupyterHub to the service
+prefix = os.environ.get('JUPYTERHUB_SERVICE_PREFIX', '/')
+service_url = os.getenv('JUPYTERHUB_SERVICE_URL')
+jupyterhub_api_url = os.getenv('JUPYTERHUB_API_URL')
+jupyterhub_api_token = os.getenv('JUPYTERHUB_API_TOKEN')
+
+auth_header = {"Authorization": "token " + jupyterhub_api_token}
+
+execution_mode = os.environ[utils.ENV_NAME_EXECUTION_MODE]
+
+http = urllib3.PoolManager()
+
+if execution_mode == utils.EXECUTION_MODE_LOCAL:
+ docker_client_kwargs = json.loads(os.getenv("DOCKER_CLIENT_KWARGS"))
+ docker_tls_kwargs = json.loads(os.getenv("DOCKER_TLS_CONFIG"))
+ docker_client = utils.init_docker_client(docker_client_kwargs, docker_tls_kwargs)
+elif execution_mode == utils.EXECUTION_MODE_KUBERNETES:
+ # incluster config is the config given by a service account and it's role permissions
+ config.load_incluster_config()
+ kubernetes_client = client.CoreV1Api()
+
+hub_name = utils.ENV_HUB_NAME
+origin_label = "{}={}".format(utils.LABEL_MLHUB_ORIGIN, hub_name)
+origin_label_filter = {"label": origin_label}
+
+class UnifiedContainer():
+
+ def __init__(self, resource):
+ self.remove = lambda: logging.info("Remove property is not defined")
+ self.resource = resource
+
+ def with_id(self, id):
+ self.id = id
+ return self
+
+ def with_name(self, name):
+ self.name = name
+ return self
+
+ def with_labels(self, labels):
+ self.labels = labels
+ return self
+
+ def with_remove(self, func):
+ self.remove = lambda: func(self.resource)
+ return self
+
+def extract_container(resource):
+ if execution_mode == utils.EXECUTION_MODE_LOCAL:
+ unified_container = UnifiedContainer(resource) \
+ .with_id(resource.id) \
+ .with_name(resource.name) \
+ .with_labels(resource.labels) \
+ .with_remove(lambda container: container.remove(v=True, force=True))
+ elif execution_mode == utils.EXECUTION_MODE_KUBERNETES:
+ unified_container = UnifiedContainer(resource) \
+ .with_id(resource.metadata.uid) \
+ .with_name(resource.metadata.name) \
+ .with_labels(resource.metadata.labels)
+
+ if unified_container == None:
+ raise UserWarning("The execution mode environment variable is not set correctly")
+
+ return unified_container
+
+def get_hub_docker_resources(docker_client_obj):
+ return docker_client_obj.list(filters=origin_label_filter)
+
+def get_hub_kubernetes_resources(namespaced_list_command, **kwargs):
+ return namespaced_list_command(hub_name, **kwargs).items
+
+def get_hub_containers():
+ if execution_mode == utils.EXECUTION_MODE_LOCAL:
+ hub_containers = get_hub_docker_resources(docker_client.containers)
+ elif execution_mode == utils.EXECUTION_MODE_KUBERNETES:
+ hub_containers = get_hub_kubernetes_resources(kubernetes_client.list_namespaced_pod, field_selector="status.phase=Running", label_selector=origin_label)
+
+ return hub_containers
+
+def remove_deleted_user_resources(existing_user_names: []):
+ """Remove resources for which no user exists anymore by checking whether the label of user name occurs in the existing
+ users list.
+
+ Args:
+ existing_user_names: list of user names that exist in the JupyterHub database
+
+ Raises:
+ UserWarning: in Kubernetes mode, the function does not work
+ """
+
+ if execution_mode == utils.EXECUTION_MODE_KUBERNETES:
+ raise UserWarning("This method cannot be used in following hub execution mode " + execution_mode)
+
+ def try_to_remove(remove_callback, resource) -> bool:
+ """Call the remove callback until the call succeeds or until the number of tries is exceeded.
+
+ Returns:
+ bool: True if it could be removed, False if it was not removable within the number of tries
+ """
+
+ for i in range(3):
+ try:
+ remove_callback()
+ return True
+ except docker.errors.APIError:
+ time.sleep(3)
+
+ logging.info("Could not remove " + resource.name)
+ return False
+
+
+ def find_and_remove(docker_client_obj, get_labels, action_callback) -> None:
+ """List all resources belonging to `docker_client_obj` which were created by MLHub.
+ Then check the list of resources for resources that belong to a user who does not exist anymore
+ and call the remove function on them.
+
+ Args:
+ docker_client_obj: A Python docker client object, such as docker_client.containers, docker_client.networks,... It must implement a .list() function (check https://docker-py.readthedocs.io/en/stable/containers.html)
+ get_labels (func): function to call on the docker resource to get the labels
+ remove (func): function to call on the docker resource to remove it
+ """
+
+ resources = get_hub_docker_resources(docker_client_obj)
+ for resource in resources:
+ user_label = get_labels(resource)[utils.LABEL_MLHUB_USER]
+ if user_label not in existing_user_names:
+ action_callback(resource)
+ # successful = try_to_remove(remove, resource)
+
+ def container_action(container):
+ try_to_remove(
+ lambda: container.remove(v=True, force=True),
+ container
+ )
+
+ find_and_remove(
+ docker_client.containers,
+ lambda res: res.labels,
+ container_action
+ )
+
+ def network_action(network):
+ try:
+ network.disconnect(hub_name)
+ except docker.errors.APIError:
+ pass
+
+ try_to_remove(network.remove, network)
+
+ find_and_remove(
+ docker_client.networks,
+ lambda res: res.attrs["Labels"],
+ network_action
+ )
+
+ find_and_remove(
+ docker_client.volumes,
+ lambda res: res.attrs["Labels"],
+ lambda res: try_to_remove(res.remove, res)
+ )
+
+def get_hub_usernames() -> []:
+ r = http.request('GET', jupyterhub_api_url + "/users",
+ headers = {**auth_header}
+ )
+
+ data = json.loads(r.data.decode("utf-8"))
+ existing_user_names = []
+ for user in data:
+ existing_user_names.append(user["name"])
+
+ return existing_user_names
+
+def remove_expired_workspaces():
+ hub_containers = get_hub_containers()
+ for container in hub_containers:
+ unified_container = extract_container(container)
+ lifetime_timestamp = utils.get_lifetime_timestamp(unified_container.labels)
+ if lifetime_timestamp != 0:
+ difference = math.ceil(lifetime_timestamp - time.time())
+ # container lifetime is exceeded (remaining lifetime is negative)
+ if difference < 0:
+ user_name = unified_container.labels[utils.LABEL_MLHUB_USER]
+ server_name = unified_container.labels[utils.LABEL_MLHUB_SERVER_NAME]
+ url = jupyterhub_api_url + "/users/{user_name}/servers/{server_name}".format(user_name=user_name, server_name=server_name)
+ r = http.request('DELETE', url,
+ body = json.dumps({"remove": True}).encode('utf-8'),
+ headers = {**auth_header}
+ )
+
+ if r.status == 202 or r.status == 204:
+ logging.info("Delete expired container " + unified_container.name)
+ unified_container.remove()
+
+class CleanupUserResources(HubAuthenticated, web.RequestHandler):
+
+ @web.authenticated
+ def get(self):
+ current_user = self.get_current_user()
+ if current_user["admin"] is False:
+ self.set_status(401)
+ self.finish()
+ return
+
+ try:
+ remove_deleted_user_resources(get_hub_usernames())
+ except UserWarning as e:
+ self.finish(str(e))
+
+class CleanupExpiredContainers(HubAuthenticated, web.RequestHandler):
+
+ @web.authenticated
+ def get(self):
+ current_user = self.get_current_user()
+ if current_user["admin"] is False:
+ self.set_status(401)
+ self.finish()
+ return
+
+ remove_expired_workspaces()
+
+app = web.Application([
+ (r"{}users".format(prefix), CleanupUserResources),
+ (r"{}expired".format(prefix), CleanupExpiredContainers)
+])
+
+service_port = int(service_url.split(":")[-1])
+app.listen(service_port)
+
+def internal_service_caller():
+ clean_interval_seconds = int(os.getenv(utils.ENV_NAME_CLEANUP_INTERVAL_SECONDS))
+ while True and clean_interval_seconds != -1:
+ time.sleep(clean_interval_seconds)
+ try:
+ remove_deleted_user_resources(get_hub_usernames())
+ except UserWarning:
+ pass
+ remove_expired_workspaces()
+
+Thread(target=internal_service_caller).start()
+
+ioloop.IOLoop.current().start()
diff --git a/resources/jupyterhub-mod/info-dialog-snippet.html b/resources/jupyterhub-mod/info-dialog-snippet.html
new file mode 100644
index 0000000..2770d71
--- /dev/null
+++ b/resources/jupyterhub-mod/info-dialog-snippet.html
@@ -0,0 +1,36 @@
+
+
+
+
+
+
+
+
+{% call modal('Workspace Info', btn_label='', btn_class='hidden') %}
+ Workspace Info for :
+
+
+
+{% endcall %}
diff --git a/resources/jupyterhub-mod/jsonpresenter/jquery.jsonPresenter.css b/resources/jupyterhub-mod/jsonpresenter/jquery.jsonPresenter.css
new file mode 100644
index 0000000..2098418
--- /dev/null
+++ b/resources/jupyterhub-mod/jsonpresenter/jquery.jsonPresenter.css
@@ -0,0 +1,59 @@
+/**
+ * jQuery Json Presenter Plugin v1.0.0
+ *
+ * Copyright 2014 Steven Pease
+ * Released under the MIT license:
+ * http://www.opensource.org/licenses/mit-license.php
+ */
+.parsed-json {
+ font-size: 11px !important;
+ font-family: Menlo, monospace;
+ margin-top: 0px;
+ margin-bottom: 0px;
+}
+
+.parsed-json-expandable-ellipsis:hover,
+.parsed-json-property-expandable:hover,
+.parsed-json-property-toggleable:hover,
+.parsed-json-has-alternate-value:hover {
+ cursor: hand;
+ cursor: pointer;
+}
+
+.parsed-json-property-name {
+ color: rgb(136, 19, 145);
+}
+
+.parsed-json-object-comma {
+ color: #333;
+}
+
+.parsed-json-value-boolean {
+ color: #0000ff;
+}
+
+.parsed-json-value-function,
+.parsed-json-array-bracket,
+.parsed-json-object-bracket {
+ color: #333;
+}
+
+.parsed-json-value-null,
+.parsed-json-value-undefined {
+ color: rgb(128, 128, 128);
+}
+
+.parsed-json-value-number {
+ color: #5FB526;
+}
+
+.parsed-json-value-regexp,
+.parsed-json-value-string {
+ color: rgb(196, 26, 22);
+ white-space: pre;
+ unicode-bidi: -webkit-isolate;
+}
+
+.parsed-json .hidden {
+ display: none;
+}
diff --git a/resources/jupyterhub-mod/jsonpresenter/jquery.jsonPresenter.js b/resources/jupyterhub-mod/jsonpresenter/jquery.jsonPresenter.js
new file mode 100644
index 0000000..a3837bc
--- /dev/null
+++ b/resources/jupyterhub-mod/jsonpresenter/jquery.jsonPresenter.js
@@ -0,0 +1,347 @@
+/**
+ * jQuery Json Presenter Plugin v1.0.0
+ *
+ * Copyright 2014 Steven Pease
+ * Released under the MIT license:
+ * http://www.opensource.org/licenses/mit-license.php
+ */
+( function( $ ) {
+ /**
+ * @param numberOfIndents
+ */
+ function getIndentString( numberOfIndents ) {
+ if ( typeof numberOfIndents === "undefined" ) {
+ numberOfIndents = 1;
+ }
+
+ var result = '';
+ for ( var i = 0; i < numberOfIndents; i++ ) {
+
+ // Use two spaces to represent an indentation
+ result += ' ';
+ }
+ return result;
+ }
+
+ function isJsonArray( jsonValue ) {
+ return jsonValue && typeof jsonValue === 'object' && typeof jsonValue.length === 'number' && !jsonValue.propertyIsEnumerable( 'length' );
+ }
+
+ /**
+ * @param {unknown_type} jsonValue The JSON value to test
+ * @return {Boolean} Whether the provided JSON value is a Date object
+ */
+ var isJsonDate = ( function() {
+ var dateObject = new Date();
+
+ return function( jsonValue ) {
+ return jsonValue && jsonValue.constructor === dateObject.constructor;
+ };
+ } )();
+
+ /**
+ * @param {unknown_type} jsonValue The JSON value to test
+ * @return {Boolean} Whether the provided JSON value is a NULL value
+ */
+ var isJsonNull = function( jsonValue ) {
+ return jsonValue === null;
+ };
+
+ /**
+ * @param {unknown_type} jsonValue The JSON value to test
+ * @return {Boolean} Whether the provided JSON value is a RegExp object
+ */
+ var isJsonRegExp = ( function() {
+ var regExpObject = new RegExp();
+
+ return function( jsonValue ) {
+ return jsonValue && jsonValue.constructor === regExpObject.constructor;
+ };
+ } )();
+
+ function processJsonPrimitive( className, value, alternateDisplayValue ) {
+ var cleanValue = function( value ) {
+ var result = value;
+
+ // Remove any "<" or ">" characters that could be interpretted as HTML
+ if ( typeof result === 'string' ) {
+ result = result.replace( //g, '>' );
+ }
+
+ return result;
+ };
+
+ if ( alternateDisplayValue ) {
+ value = '' + cleanValue( value ) + '' + cleanValue( alternateDisplayValue ) + '';
+ } else {
+ value = cleanValue( value );
+ }
+
+ return '' + value + '';
+ }
+
+ function processJsonValue( settings, jsonValue, indentLevel, propertyName ) {
+ if ( typeof indentLevel === 'undefined' ) {
+ indentLevel = 0;
+ }
+
+ var isExpandable = false,
+ isToggleable = false,
+ result = '';
+
+ if ( isJsonArray( jsonValue ) ) {
+ if ( jsonValue.length ) {
+ result += '[...';
+
+ for ( var i = 0; i < jsonValue.length; i++ ) {
+ result += processJsonValue( settings, jsonValue[ i ], indentLevel + 1 );
+
+ if ( i < jsonValue.length - 1 ) {
+ result += ',';
+ }
+ }
+
+ result += "\n" + getIndentString( indentLevel ) + ']';
+ isExpandable = true;
+ } else {
+ result += '[]';
+ }
+ } else {
+ var valueType = typeof jsonValue;
+
+ switch ( valueType ) {
+ case 'object':
+ if ( isJsonNull( jsonValue ) ) {
+ result += processJsonPrimitive( 'null', null );
+ } else if ( isJsonDate( jsonValue ) ) {
+ result += processJsonPrimitive( 'date', 'new Date(' + jsonValue.getTime() + ')', jsonValue.toString() );
+ isToggleable = true;
+ } else if ( isJsonRegExp( jsonValue ) ) {
+ result += processJsonPrimitive( 'regexp', jsonValue );
+ } else {
+
+ // Determine the number of properties this object has
+ var propertyCount = ( function() {
+ var result = 0;
+ for ( var i in jsonValue ) { // jshint ignore:line
+ result++;
+ }
+ return result;
+ } )();
+
+ if ( propertyCount > 0 ) {
+ result += '{...';
+ ( function() {
+ var propertyCounter = 0;
+ for ( var propertyName in jsonValue ) {
+ result += processJsonValue( settings, jsonValue[ propertyName ], indentLevel + 1, propertyName );
+
+ if ( ++propertyCounter < propertyCount ) {
+ result += ',';
+ }
+ }
+ } )();
+ result += "\n" + getIndentString( indentLevel ) + '}';
+ isExpandable = true;
+ } else {
+ result += '{}';
+ }
+ }
+ break;
+ case 'number':
+ result += processJsonPrimitive( 'number', jsonValue );
+ break;
+ case 'boolean':
+ result += processJsonPrimitive( 'boolean', jsonValue );
+ break;
+ case 'function':
+ var expandedFunction = ( jsonValue.toString() ).replace( /\n/g, "\n" + getIndentString( indentLevel ) ),
+ nonExpandedFunction = ( jsonValue.toString() ).replace( /\s+/g, ' ' );
+
+ if ( expandedFunction !== nonExpandedFunction ) {
+ result += processJsonPrimitive( 'function', nonExpandedFunction, expandedFunction );
+ isToggleable = true;
+ } else {
+ result += processJsonPrimitive( 'function', nonExpandedFunction );
+ }
+ break;
+ case 'undefined':
+ result += processJsonPrimitive( 'undefined', jsonValue );
+ break;
+ default:
+ var displayValue = '"' + jsonValue.replace( /\n/g, "\\n" ) + '"',
+ alternateDisplayValue = '"' + jsonValue + '"';
+
+ if ( displayValue !== alternateDisplayValue ) {
+ result += processJsonPrimitive( 'string', displayValue, alternateDisplayValue );
+ isToggleable = true;
+ } else {
+ result += processJsonPrimitive( 'string', displayValue );
+ }
+
+ break;
+ }
+ }
+
+ var resultPrefix = ( indentLevel !== 0 ? "\n" : '' ) + getIndentString( indentLevel );
+ if ( typeof propertyName !== 'undefined' ) {
+ var propertyNameLabel = settings.wrapPropertiesInQuotes ? '"' + propertyName + '"' : propertyName;
+ resultPrefix += '' + propertyNameLabel + ': ';
+ }
+
+ result = resultPrefix + result;
+
+ if ( isExpandable || isToggleable ) {
+ return '' + result + '';
+ } else {
+ return result;
+ }
+ }
+
+ function expandNode( expandableNodeElement ) {
+ if ( expandableNodeElement.children( '.parsed-json-expandable' ).is( ':not(:visible)' ) ) {
+ toggleExpandNode( expandableNodeElement );
+ }
+ }
+
+ function collapseNode( expandableNodeElement ) {
+ if ( expandableNodeElement.children( '.parsed-json-expandable' ).is( ':visible' ) ) {
+ toggleExpandNode( expandableNodeElement );
+ }
+ }
+
+ function toggleExpandNode( expandableNodeElement ) {
+ expandableNodeElement.children( '.parsed-json-expandable,.parsed-json-expandable-ellipsis' ).toggleClass( 'hidden' );
+ }
+
+ function togglePresentationNode( toggleableNodeElement ) {
+ toggleableNodeElement.children( '.parsed-json-has-alternate-value' ).find( 'span' ).toggleClass( 'hidden' );
+ }
+
+ function getExpandableChildNodes( expandableNodeElement ) {
+ return expandableNodeElement.find( '> .parsed-json-expandable > .parsed-json-node-expandable' );
+ }
+
+ function expandAll( expandableElement ) {
+ expand( expandableElement );
+ }
+
+ function collapseAll( expandableElement ) {
+ collapse( expandableElement );
+ }
+
+ function expand( expandableNodeElement, depth ) {
+ expandNode( expandableNodeElement );
+ if ( !( typeof depth === 'number' && depth <= 0 ) ) {
+ getExpandableChildNodes( expandableNodeElement ).each( function() {
+ expandNode( $( this ) );
+ expand( $( this ), typeof depth !== 'undefined' ? depth - 1 : depth );
+ } );
+ }
+ }
+
+ function collapse( expandableNodeElement, depth ) {
+ if ( !( typeof depth === 'number' && depth <= 0 ) ) {
+ getExpandableChildNodes( expandableNodeElement ).each( function() {
+ collapse( $( this ), typeof depth !== 'undefined' ? depth - 1 : depth );
+ collapseNode( $( this ) );
+ } );
+ }
+ collapseNode( expandableNodeElement );
+ }
+
+ function getRootNode( containerElement ) {
+ return containerElement.find( '> .parsed-json > .parsed-json-node-expandable' );
+ }
+
+ function onToggleableValueClick( event ) {
+ togglePresentationNode( $( $( event.currentTarget ).parents( '.parsed-json-node-toggleable' ).get( 0 ) ) );
+ event.stopPropagation();
+ }
+
+ function onExpandableValueClick( event ) {
+ toggleExpandNode( $( $( event.currentTarget ).parents( '.parsed-json-node-expandable' ).get( 0 ) ) );
+ event.stopPropagation();
+ }
+
+ function onExpandablePropertyClick( event ) {
+ toggleExpandNode( $( event.currentTarget ).parent() );
+ event.stopPropagation();
+ }
+
+ function onToggleablePropertyClick( event ) {
+ togglePresentationNode( $( event.currentTarget ).parent() );
+ event.stopPropagation();
+ }
+
+ function destroy( containerElement ) {
+ containerElement
+ .off( 'click', '.parsed-json-has-alternate-value span', onToggleableValueClick )
+ .off( 'click', '.parsed-json-property-expandable', onExpandablePropertyClick )
+ .off( 'click', '.parsed-json-expandable-ellipsis', onExpandableValueClick )
+ .off( 'click', '.parsed-json-property-toggleable', onToggleablePropertyClick );
+
+ containerElement.html( '' );
+ }
+
+ function create( containerElement, settings ) {
+
+ // Make sure that the JSON Presenter is not stacking event listeners on top of existing ones
+ if ( isAlreadyPresentingJson( containerElement ) ) {
+ destroy( containerElement );
+ }
+
+ containerElement
+ .on( 'click', '.parsed-json-has-alternate-value span', onToggleableValueClick )
+ .on( 'click', '.parsed-json-property-expandable', onExpandablePropertyClick )
+ .on( 'click', '.parsed-json-expandable-ellipsis', onExpandableValueClick )
+ .on( 'click', '.parsed-json-property-toggleable', onToggleablePropertyClick );
+
+ containerElement.html( '
}).catch(() => {});
});
+
+{% include 'ssh-dialog-snippet.html' %}
+
+{% include 'info-dialog-snippet.html' %}
+
{% endblock %}
diff --git a/resources/jupyterhub_config.py b/resources/jupyterhub_config.py
index 778e61f..12d23d5 100644
--- a/resources/jupyterhub_config.py
+++ b/resources/jupyterhub_config.py
@@ -3,7 +3,13 @@
"""
import os
+import signal
import socket
+import sys
+
+import docker.errors
+
+import json
from mlhubspawner import utils
from subprocess import call
@@ -18,8 +24,19 @@
original_normalize_username = Authenticator.normalize_username
def custom_normalize_username(self, username):
username = original_normalize_username(self, username)
- for forbidden_username_char in [" ", ",", ";", "."]:
- username = username.replace(forbidden_username_char, "")
+ more_than_one_forbidden_char = False
+ for forbidden_username_char in [" ", ",", ";", ".", "-"]:
+ # Replace special characters with a non-special character. Cannot just be empty, like "", because then it could happen that two distinct user names are transformed into the same username.
+ # Example: "foo, bar" and "fo, obar" would both become "foobar".
+ replace_char = "0"
+ # If there is more than one special character, just replace one of them. Otherwise, "foo, bar" would become "foo00bar" instead of "foo0bar"
+ if more_than_one_forbidden_char == True:
+ replace_char = ""
+ temp_username = username
+ username = username.replace(forbidden_username_char, replace_char)
+ if username != temp_username:
+ more_than_one_forbidden_char = True
+
return username
Authenticator.normalize_username = custom_normalize_username
@@ -40,7 +57,9 @@ def combine_config_dicts(*configs) -> dict:
### END HELPER FUNCTIONS###
-ENV_HUB_NAME = os.environ['HUB_NAME']
+ENV_NAME_HUB_NAME = 'HUB_NAME'
+ENV_HUB_NAME = os.environ[ENV_NAME_HUB_NAME]
+ENV_EXECUTION_MODE = os.environ[utils.ENV_NAME_EXECUTION_MODE]
# User containers will access hub by container name on the Docker network
c.JupyterHub.hub_ip = '0.0.0.0' #'research-hub'
@@ -65,7 +84,7 @@ def combine_config_dicts(*configs) -> dict:
c.Spawner.environment = default_env
# Workaround to prevent api problems
-c.Spawner.will_resume = True
+#c.Spawner.will_resume = True
# --- Spawner-specific ----
c.JupyterHub.spawner_class = 'mlhubspawner.MLHubDockerSpawner' # override in your config if you want to have a different spawner. If it is the or inherits from DockerSpawner, the c.DockerSpawner config can have an effect.
@@ -106,9 +125,17 @@ def combine_config_dicts(*configs) -> dict:
# See https://traitlets.readthedocs.io/en/stable/config.html#configuration-files-inheritance
load_subconfig("{}/jupyterhub_user_config.py".format(os.getenv("_RESOURCES_PATH")))
+
+
+service_environment = {
+ ENV_NAME_HUB_NAME: ENV_HUB_NAME,
+ utils.ENV_NAME_EXECUTION_MODE: ENV_EXECUTION_MODE,
+ utils.ENV_NAME_CLEANUP_INTERVAL_SECONDS: os.getenv(utils.ENV_NAME_CLEANUP_INTERVAL_SECONDS),
+}
+
# In Kubernetes mode, load the Kubernetes Jupyterhub config that can be configured via a config.yaml.
# Those values will override the values set above, as it is loaded afterwards.
-if os.environ['EXECUTION_MODE'] == "k8s":
+if ENV_EXECUTION_MODE == utils.EXECUTION_MODE_KUBERNETES:
load_subconfig("{}/kubernetes/jupyterhub_chart_config.py".format(os.getenv("_RESOURCES_PATH")))
c.JupyterHub.spawner_class = 'mlhubspawner.MLHubKubernetesSpawner'
@@ -121,13 +148,36 @@ def combine_config_dicts(*configs) -> dict:
# if not isinstance(c.KubeSpawner.environment, dict):
# c.KubeSpawner.environment = {}
c.KubeSpawner.environment.update(default_env)
-else:
+
+ # For cleanup-service
+ ## Env variables that are used by the Python Kubernetes library to load the incluster config
+ SERVICE_HOST_ENV_NAME = "KUBERNETES_SERVICE_HOST"
+ SERVICE_PORT_ENV_NAME = "KUBERNETES_SERVICE_PORT"
+ service_environment.update({
+ SERVICE_HOST_ENV_NAME: os.getenv(SERVICE_HOST_ENV_NAME),
+ SERVICE_PORT_ENV_NAME: os.getenv(SERVICE_PORT_ENV_NAME)
+ })
+ service_host = "hub"
+
+
+elif ENV_EXECUTION_MODE == utils.EXECUTION_MODE_LOCAL:
client_kwargs = {**get_or_init(c.DockerSpawner.client_kwargs, dict), **get_or_init(c.MLHubDockerSpawner.client_kwargs, dict)}
tls_config = {**get_or_init(c.DockerSpawner.tls_config, dict), **get_or_init(c.MLHubDockerSpawner.tls_config, dict)}
docker_client = utils.init_docker_client(client_kwargs, tls_config)
- docker_client.containers.list(filters={"id": socket.gethostname()})[0].rename(ENV_HUB_NAME)
- c.MLHubDockerSpawner.hub_name = ENV_HUB_NAME
+ try:
+ container = docker_client.containers.list(filters={"id": socket.gethostname()})[0]
+
+ if container.name.lower() != ENV_HUB_NAME.lower():
+ container.rename(ENV_HUB_NAME.lower())
+ except docker.errors.APIError as e:
+ print("Could not correctly start MLHub container. " + str(e))
+ os.kill(os.getpid(), signal.SIGTERM)
+
+ # For cleanup-service
+ service_environment.update({"DOCKER_CLIENT_KWARGS": json.dumps(client_kwargs), "DOCKER_TLS_CONFIG": json.dumps(tls_config)})
+ service_host = "127.0.0.1"
+ #c.MLHubDockerSpawner.hub_name = ENV_HUB_NAME
# Add nativeauthenticator-specific templates
if c.JupyterHub.authenticator_class == NATIVE_AUTHENTICATOR_CLASS:
@@ -137,3 +187,13 @@ def combine_config_dicts(*configs) -> dict:
# if not isinstance(c.JupyterHub.template_paths, list):
# c.JupyterHub.template_paths = []
c.JupyterHub.template_paths.append("{}/templates/".format(os.path.dirname(nativeauthenticator.__file__)))
+
+c.JupyterHub.services = [
+ {
+ 'name': 'cleanup-service',
+ 'admin': True,
+ 'url': 'http://{}:9000'.format(service_host),
+ 'environment': service_environment,
+ 'command': [sys.executable, '/resources/cleanup-service.py']
+ }
+]
diff --git a/resources/mlhubspawner/mlhubspawner/mlhubkubernetesspawner.py b/resources/mlhubspawner/mlhubspawner/mlhubkubernetesspawner.py
index 8448eac..809e7f5 100644
--- a/resources/mlhubspawner/mlhubspawner/mlhubkubernetesspawner.py
+++ b/resources/mlhubspawner/mlhubspawner/mlhubkubernetesspawner.py
@@ -17,6 +17,8 @@
import re
from mlhubspawner import spawner_options, utils
+
+LABEL_POD_NAME = "pod_name"
class MLHubKubernetesSpawner(KubeSpawner):
"""Provides the possibility to spawn docker containers with specific options, such as resource limits (CPU and Memory), Environment Variables, ..."""
@@ -30,8 +32,9 @@ class MLHubKubernetesSpawner(KubeSpawner):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
- self.hub_name = os.getenv("HUB_NAME", "mlhub")
- self.default_label = {"origin": self.hub_name}
+ self.hub_name = utils.ENV_HUB_NAME
+ self.default_label = {utils.LABEL_MLHUB_ORIGIN: self.hub_name, utils.LABEL_MLHUB_USER: self.user.name, utils.LABEL_MLHUB_SERVER_NAME: self.name, LABEL_POD_NAME: self.pod_name}
+ self.extra_labels.update(self.default_label)
@default('options_form')
def _options_form(self):
@@ -67,6 +70,9 @@ def get_env(self):
@gen.coroutine
def start(self):
"""Set custom configuration during start before calling the super.start method of Dockerspawner"""
+
+ self.saved_user_options = self.user_options
+
if self.user_options.get('image'):
self.image = self.user_options.get('image')
@@ -87,8 +93,7 @@ def start(self):
# self.volumes = {'jhub-user-{username}{servername}': "/workspace"}
# set default label 'origin' to know for sure which containers where started via the hub
- self.extra_labels['origin'] = self.hub_name
- self.extra_labels['pod_name'] = self.pod_name
+ #self.extra_labels['pod_name'] = self.pod_name
if self.user_options.get('days_to_live'):
days_to_live_in_seconds = int(self.user_options.get('days_to_live')) * 24 * 60 * 60 # days * hours_per_day * minutes_per_hour * seconds_per_minute
expiration_timestamp = time.time() + days_to_live_in_seconds
@@ -109,8 +114,8 @@ def start(self):
type='ClusterIP',
ports=[V1ServicePort(port=self.port, target_port=self.port)],
selector={
- 'origin': self.extra_labels['origin'],
- 'pod_name': self.extra_labels['pod_name']
+ utils.LABEL_MLHUB_ORIGIN: self.extra_labels[utils.LABEL_MLHUB_ORIGIN],
+ LABEL_POD_NAME: self.extra_labels[LABEL_POD_NAME]
}
),
metadata = V1ObjectMeta(
@@ -153,8 +158,8 @@ def get_container_metadata(self) -> str:
return utils.get_container_metadata(self)
- def get_lifetime_timestamp(self, labels: dict) -> float:
- return float(labels.get(utils.LABEL_EXPIRATION_TIMESTAMP, '0'))
+ def get_workspace_config(self) -> str:
+ return utils.get_workspace_config(self)
def get_labels(self) -> dict:
try:
@@ -171,3 +176,15 @@ def delete_if_exists(self, kind, safe_name, future):
if e.status != 404:
raise
self.log.warn("Could not delete %s/%s: does not exist", kind, safe_name)
+
+ # get_state and load_state are functions used by Jupyterhub to save and load variables that shall be persisted even if the hub is removed and re-created
+ # Override
+ def get_state(self):
+ state = super(MLHubKubernetesSpawner, self).get_state()
+ state = utils.get_state(self, state)
+ return state
+
+ # Override
+ def load_state(self, state):
+ super(MLHubKubernetesSpawner, self).load_state(state)
+ utils.load_state(self, state)
diff --git a/resources/mlhubspawner/mlhubspawner/mlhubspawner.py b/resources/mlhubspawner/mlhubspawner/mlhubspawner.py
index 3f7ffd6..7a2f0c7 100644
--- a/resources/mlhubspawner/mlhubspawner/mlhubspawner.py
+++ b/resources/mlhubspawner/mlhubspawner/mlhubspawner.py
@@ -47,7 +47,7 @@ def has_complete_network_information(network):
class MLHubDockerSpawner(DockerSpawner):
"""Provides the possibility to spawn docker containers with specific options, such as resource limits (CPU and Memory), Environment Variables, ..."""
- hub_name = Unicode(config=True, help="Name of the hub container.")
+ #hub_name = Unicode(config=True, help="Name of the hub container.")
workspace_images = List(
trait = Unicode(),
@@ -58,11 +58,11 @@ class MLHubDockerSpawner(DockerSpawner):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
-
+ self.hub_name = utils.ENV_HUB_NAME
+ self.default_labels = {utils.LABEL_MLHUB_ORIGIN: self.hub_name, utils.LABEL_MLHUB_USER: self.user.name, utils.LABEL_MLHUB_SERVER_NAME: self.name}
# Get the MLHub container name to be used as the DNS name for the spawned workspaces, so they can connect to the Hub even if the container is
# removed and restarted
client = self.highlevel_docker_client
- self.default_label = {"origin": self.hub_name}
# Connect MLHub to the existing workspace networks (in case of removing / recreation). By this, the hub can connect to the existing
# workspaces and does not have to restart them.
@@ -149,13 +149,15 @@ def start(self) -> (str, int):
(str, int): container's ip address or '127.0.0.1', container's port
"""
+ self.saved_user_options = self.user_options
+
if self.user_options.get('image'):
self.image = self.user_options.get('image')
extra_host_config = {}
if self.user_options.get('cpu_limit'):
# nano_cpus cannot be bigger than the number of CPUs of the machine (this method would currently not work in a cluster, as machines could be different than the machine where the runtime-manager and this code run.
- max_available_cpus = self.resource_information.cpu_count
+ max_available_cpus = self.resource_information["cpu_count"]
limited_cpus = min(
int(self.user_options.get('cpu_limit')), max_available_cpus)
@@ -169,11 +171,12 @@ def start(self) -> (str, int):
if self.user_options.get('is_mount_volume') == 'on':
# {username} and {servername} will be automatically replaced by DockerSpawner with the right values as in template_namespace
#volumeName = self.name_template.format(prefix=self.prefix)
+ self.highlevel_docker_client.volumes.create(name=self.object_name, labels=self.default_labels)
self.volumes = {self.object_name: "/workspace"}
extra_create_kwargs = {}
# set default label 'origin' to know for sure which containers where started via the hub
- extra_create_kwargs['labels'] = self.default_label
+ extra_create_kwargs['labels'] = self.default_labels
if self.user_options.get('days_to_live'):
days_to_live_in_seconds = int(self.user_options.get('days_to_live')) * 24 * 60 * 60 # days * hours_per_day * minutes_per_hour * seconds_per_minute
expiration_timestamp = time.time() + days_to_live_in_seconds
@@ -276,7 +279,7 @@ def create_network(self, name):
ipam_pool = docker.types.IPAMPool(subnet=next_cidr.exploded,
gateway=(next_cidr.network_address + 1).exploded)
ipam_config = docker.types.IPAMConfig(pool_configs=[ipam_pool])
- return client.networks.create(name, ipam=ipam_config, labels=self.default_label)
+ return client.networks.create(name, ipam=ipam_config, labels=self.default_labels)
def connect_hub_to_network(self, network):
try:
@@ -290,18 +293,16 @@ def connect_hub_to_network(self, network):
"Could not connect mlhub to the network and, thus, cannot create the container.")
return
- def get_container_metadata(self) -> str:
- if self.container_id is None or self.container_id == '':
- return ""
-
- return utils.get_container_metadata(self)
-
- def get_lifetime_timestamp(self, labels: dict) -> float:
- return float(labels.get(utils.LABEL_EXPIRATION_TIMESTAMP, '0'))
+ def get_workspace_config(self) -> str:
+ return utils.get_workspace_config(self)
def is_update_available(self):
try:
- return self.image != self.highlevel_docker_client.containers.get(self.container_id).image.tags[0]
+ # compare the last parts of the images, so that also "mltooling/ml-workspace:0.8.7 = ml-workspace:0.8.7" would match
+ config_image = self.image.split("/")[-1]
+ workspace_image = self.highlevel_docker_client.containers.get(self.container_id).image.tags[0].split("/")[-1]
+
+ return config_image != workspace_image
except (docker.errors.NotFound, docker.errors.NullResource):
return False
@@ -319,6 +320,18 @@ def template_namespace(self):
return template
+ # get_state and load_state are functions used by Jupyterhub to save and load variables that shall be persisted even if the hub is removed and re-created
+ # Override
+ def get_state(self):
+ state = super(MLHubDockerSpawner, self).get_state()
+ state = utils.get_state(self, state)
+ return state
+
+ # Override
+ def load_state(self, state):
+ super(MLHubDockerSpawner, self).load_state(state)
+ utils.load_state(self, state)
+
def get_gpu_info(self) -> list:
count_gpu = 0
try:
diff --git a/resources/mlhubspawner/mlhubspawner/spawner_options.py b/resources/mlhubspawner/mlhubspawner/spawner_options.py
index 014942c..c26dcf2 100644
--- a/resources/mlhubspawner/mlhubspawner/spawner_options.py
+++ b/resources/mlhubspawner/mlhubspawner/spawner_options.py
@@ -53,7 +53,7 @@ def get_options_form(spawner, additional_cpu_info="", additional_memory_info="",
-
+
{additional_cpu_info}
@@ -89,11 +89,11 @@ def get_options_form(spawner, additional_cpu_info="", additional_memory_info="",
)
def get_options_form_docker(spawner):
- description_gpus = 'Empty for no GPU, "all" for all GPUs, or a comma-separated list of indices of the GPUs (e.g 0,2).'
+ description_gpus = 'Leave empty for no GPU, "all" for all GPUs, or a comma-separated list of indices of the GPUs (e.g 0,2).'
additional_info = {
- "additional_cpu_info": "Number between 1 and {cpu_count}".format(cpu_count=spawner.resource_information['cpu_count']),
- "additional_memory_info": "Number between 1 and {memory_count_in_gb}".format(memory_count_in_gb=spawner.resource_information['memory_count_in_gb']),
- "additional_gpu_info": "
Available GPUs: {gpu_count}
{description_gpus}
".format(gpu_count=spawner.resource_information['gpu_count'], description_gpus=description_gpus)
+ "additional_cpu_info": "Host has {cpu_count} CPUs".format(cpu_count=spawner.resource_information['cpu_count']),
+ "additional_memory_info": "Host has {memory_count_in_gb}GB memory".format(memory_count_in_gb=spawner.resource_information['memory_count_in_gb']),
+ "additional_gpu_info": "