Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

31.0.1 staging #513

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

31.0.1 staging #513

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3de8072
update version
clintropolis Dec 3, 2024
779ff76
prepare staging website src for 31.0.1
clintropolis Dec 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/latest/development/extensions-contrib/k8s-jobs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,7 +386,7 @@ template:
env:
- name: CUSTOM_ENV_VARIABLE
value: "hello"
image: apache/druid:31.0.0
image: apache/druid:31.0.1
name: main
ports:
- containerPort: 8091
Expand Down
12 changes: 6 additions & 6 deletions docs/latest/operations/pull-deps.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Don't use the default remote repository, https://repo1.maven.org/maven2/. Only u

`-d` or `--defaultVersion`

Version to use for extension coordinate that doesn't have a version information. For example, if extension coordinate is `org.apache.druid.extensions:mysql-metadata-storage`, and default version is `31.0.0`, then this coordinate will be treated as `org.apache.druid.extensions:mysql-metadata-storage:31.0.0`
Version to use for extension coordinate that doesn't have a version information. For example, if extension coordinate is `org.apache.druid.extensions:mysql-metadata-storage`, and default version is `31.0.1`, then this coordinate will be treated as `org.apache.druid.extensions:mysql-metadata-storage:31.0.1`

`--use-proxy`

Expand Down Expand Up @@ -91,10 +91,10 @@ To run `pull-deps`, you should

Example:

Suppose you want to download ```mysql-metadata-storage``` and ```hadoop-client```(both 2.3.0 and 2.4.0) with a specific version, you can run `pull-deps` command with `-c org.apache.druid.extensions:mysql-metadata-storage:31.0.0`, `-h org.apache.hadoop:hadoop-client:2.3.0` and `-h org.apache.hadoop:hadoop-client:2.4.0`, an example command would be:
Suppose you want to download ```mysql-metadata-storage``` and ```hadoop-client```(both 2.3.0 and 2.4.0) with a specific version, you can run `pull-deps` command with `-c org.apache.druid.extensions:mysql-metadata-storage:31.0.1`, `-h org.apache.hadoop:hadoop-client:2.3.0` and `-h org.apache.hadoop:hadoop-client:2.4.0`, an example command would be:

```
java -classpath "/my/druid/lib/*" org.apache.druid.cli.Main tools pull-deps --clean -c org.apache.druid.extensions:mysql-metadata-storage:31.0.0 -h org.apache.hadoop:hadoop-client:2.3.0 -h org.apache.hadoop:hadoop-client:2.4.0
java -classpath "/my/druid/lib/*" org.apache.druid.cli.Main tools pull-deps --clean -c org.apache.druid.extensions:mysql-metadata-storage:31.0.1 -h org.apache.hadoop:hadoop-client:2.3.0 -h org.apache.hadoop:hadoop-client:2.4.0
```

Because `--clean` is supplied, this command will first remove the directories specified at `druid.extensions.directory` and `druid.extensions.hadoopDependenciesDir`, then recreate them and start downloading the extensions there. After finishing downloading, if you go to the extension directories you specified, you will see
Expand All @@ -103,7 +103,7 @@ Because `--clean` is supplied, this command will first remove the directories sp
tree extensions
extensions
└── mysql-metadata-storage
└── mysql-metadata-storage-31.0.0.jar
└── mysql-metadata-storage-31.0.1.jar
```

```
Expand All @@ -128,10 +128,10 @@ hadoop-dependencies/
..... lots of jars
```

Note that if you specify `--defaultVersion`, you don't have to put version information in the coordinate. For example, if you want `mysql-metadata-storage` to use version `31.0.0`, you can change the command above to
Note that if you specify `--defaultVersion`, you don't have to put version information in the coordinate. For example, if you want `mysql-metadata-storage` to use version `31.0.1`, you can change the command above to

```
java -classpath "/my/druid/lib/*" org.apache.druid.cli.Main tools pull-deps --defaultVersion 31.0.0 --clean -c org.apache.druid.extensions:mysql-metadata-storage -h org.apache.hadoop:hadoop-client:2.3.0 -h org.apache.hadoop:hadoop-client:2.4.0
java -classpath "/my/druid/lib/*" org.apache.druid.cli.Main tools pull-deps --defaultVersion 31.0.1 --clean -c org.apache.druid.extensions:mysql-metadata-storage -h org.apache.hadoop:hadoop-client:2.3.0 -h org.apache.hadoop:hadoop-client:2.4.0
```

:::info
Expand Down
688 changes: 11 additions & 677 deletions docs/latest/release-info/release-notes.md
View file
Open in desktop

Large diffs are not rendered by default.

10 changes: 5 additions & 5 deletions docs/latest/tutorials/cluster.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,14 +150,14 @@ First, download and unpack the release archive. It's best to do this on a single
since you will be editing the configurations and then copying the modified distribution out to all
of your servers.

[Download](https://www.apache.org/dyn/closer.cgi?path=/druid/31.0.0/apache-druid-31.0.0-bin.tar.gz)
the 31.0.0 release.
[Download](https://www.apache.org/dyn/closer.cgi?path=/druid/31.0.1/apache-druid-31.0.1-bin.tar.gz)
the 31.0.1 release.

Extract Druid by running the following commands in your terminal:

```bash
tar -xzf apache-druid-31.0.0-bin.tar.gz
cd apache-druid-31.0.0
tar -xzf apache-druid-31.0.1-bin.tar.gz
cd apache-druid-31.0.1
```

In the package, you should find:
Expand Down Expand Up @@ -423,7 +423,7 @@ Copy the Druid distribution and your edited configurations to your Master server
If you have been editing the configurations on your local machine, you can use *rsync* to copy them:

```bash
rsync -az apache-druid-31.0.0/ MASTER_SERVER:apache-druid-31.0.0/
rsync -az apache-druid-31.0.1/ MASTER_SERVER:apache-druid-31.0.1/
```

### No Zookeeper on Master
Expand Down
8 changes: 4 additions & 4 deletions docs/latest/tutorials/docker.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ You can modify the value of `DRUID_SINGLE_NODE_CONF` in the Docker [`environment

Create a directory to hold the Druid Docker files.

The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/31.0.0/distribution/docker/docker-compose.yml) which pulls an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments. [Download](https://raw.githubusercontent.com/apache/druid/31.0.0/distribution/docker/docker-compose.yml) this file to the directory created above.
The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/31.0.1/distribution/docker/docker-compose.yml) which pulls an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments. [Download](https://raw.githubusercontent.com/apache/druid/31.0.1/distribution/docker/docker-compose.yml) this file to the directory created above.

### Compose file

Expand All @@ -58,7 +58,7 @@ It will also create a named volume `druid_shared` as deep storage to keep and sh

### Environment file

The Druid `docker-compose.yml` example uses an [environment file](https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option) to specify the complete Druid configuration, including the environment variables described in [Configuration](#configuration). This file is named `environment` by default, and must be in the same directory as the `docker-compose.yml` file. [Download](https://raw.githubusercontent.com/apache/druid/31.0.0/distribution/docker/environment) the example `environment` file to the directory created above. The options in this file work well for trying Druid and for using the tutorial.
The Druid `docker-compose.yml` example uses an [environment file](https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option) to specify the complete Druid configuration, including the environment variables described in [Configuration](#configuration). This file is named `environment` by default, and must be in the same directory as the `docker-compose.yml` file. [Download](https://raw.githubusercontent.com/apache/druid/31.0.1/distribution/docker/environment) the example `environment` file to the directory created above. The options in this file work well for trying Druid and for using the tutorial.

The single-file approach is inadequate for a production system. Instead we suggest using either `DRUID_COMMON_CONFIG` and `DRUID_CONFIG_${service}` or specially tailored, service-specific environment files.

Expand All @@ -81,7 +81,7 @@ Production configuration:

Logging configuration:

* `DRUID_LOG4J` -- set the entire [`log4j.xml` configuration file](https://logging.apache.org/log4j/2.x/manual/configuration.html#XML) verbatim. ([Example](https://github.com/apache/druid/blob/31.0.0/distribution/docker/environment#L52))
* `DRUID_LOG4J` -- set the entire [`log4j.xml` configuration file](https://logging.apache.org/log4j/2.x/manual/configuration.html#XML) verbatim. ([Example](https://github.com/apache/druid/blob/31.0.1/distribution/docker/environment#L52))
* `DRUID_LOG_LEVEL` -- override the default [Log4j log level](https://en.wikipedia.org/wiki/Log4j#Log4j_log_levels)
* `DRUID_SERVICE_LOG4J` -- set the entire [`log4j.xml` configuration file](https://logging.apache.org/log4j/2.x/manual/configuration.html#XML) verbatim specific to a service.
* `DRUID_SERVICE_LOG_LEVEL` -- override the default [Log4j log level](https://en.wikipedia.org/wiki/Log4j#Log4j_log_levels) in the service specific log4j.
Expand Down Expand Up @@ -131,7 +131,7 @@ You can explore the Druid containers using Docker to start a shell:
docker exec -ti <id> sh
```

Where `<id>` is the container id found with `docker ps`. Druid is installed in `/opt/druid`. The [script](https://github.com/apache/druid/blob/31.0.0/distribution/docker/druid.sh) which consumes the environment variables mentioned above, and which launches Druid, is located at `/druid.sh`.
Where `<id>` is the container id found with `docker ps`. Druid is installed in `/opt/druid`. The [script](https://github.com/apache/druid/blob/31.0.1/distribution/docker/druid.sh) which consumes the environment variables mentioned above, and which launches Druid, is located at `/druid.sh`.

Run `docker compose down` to shut down the cluster. Your data is persisted as a set of [Docker volumes](https://docs.docker.com/storage/volumes/) and will be available when you restart your Druid cluster.

26 changes: 13 additions & 13 deletions docs/latest/tutorials/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,21 +45,21 @@ The software requirements for the installation machine are:
* Perl 5

Java must be available. Either it is on your path, or set one of the `JAVA_HOME` or `DRUID_JAVA_HOME` environment variables.
You can run `apache-druid-31.0.0/bin/verify-java` to verify Java requirements for your environment.
You can run `apache-druid-31.0.1/bin/verify-java` to verify Java requirements for your environment.

Before installing a production Druid instance, be sure to review the [security
overview](../operations/security-overview.md). In general, avoid running Druid as root user. Consider creating a
dedicated user account for running Druid.

## Install Druid

Download the [31.0.0 release](https://druid.apache.org/downloads/) from Apache Druid.
Download the [31.0.1 release](https://druid.apache.org/downloads/) from Apache Druid.

In your terminal, extract the file and change directories to the distribution directory:

```bash
tar -xzf apache-druid-31.0.0-bin.tar.gz
cd apache-druid-31.0.0
tar -xzf apache-druid-31.0.1-bin.tar.gz
cd apache-druid-31.0.1
```

The distribution directory contains `LICENSE` and `NOTICE` files and subdirectories for executable files, configuration files, sample data and more.
Expand All @@ -71,7 +71,7 @@ This configuration includes default settings that are appropriate for this tutor

You can view the default settings in the configuration files located in `conf/druid/auto`.

From the apache-druid-31.0.0 package root, run the following command:
From the apache-druid-31.0.1 package root, run the following command:

```bash
./bin/start-druid
Expand All @@ -85,20 +85,20 @@ $ ./bin/start-druid
[Tue Nov 29 16:31:06 2022] Starting Apache Druid.
[Tue Nov 29 16:31:06 2022] Open http://localhost:8888/ in your browser to access the web console.
[Tue Nov 29 16:31:06 2022] Or, if you have enabled TLS, use https on port 9088.
[Tue Nov 29 16:31:06 2022] Starting services with log directory [/apache-druid-31.0.0/log].
[Tue Nov 29 16:31:06 2022] Starting services with log directory [/apache-druid-31.0.1/log].
[Tue Nov 29 16:31:06 2022] Running command[zk]: bin/run-zk conf
[Tue Nov 29 16:31:06 2022] Running command[broker]: bin/run-druid broker /apache-druid-31.0.0/conf/druid/single-server/quickstart '-Xms1187m -Xmx1187m -XX:MaxDirectMemorySize=791m'
[Tue Nov 29 16:31:06 2022] Running command[router]: bin/run-druid router /apache-druid-31.0.0/conf/druid/single-server/quickstart '-Xms128m -Xmx128m'
[Tue Nov 29 16:31:06 2022] Running command[coordinator-overlord]: bin/run-druid coordinator-overlord /apache-druid-31.0.0/conf/druid/single-server/quickstart '-Xms1290m -Xmx1290m'
[Tue Nov 29 16:31:06 2022] Running command[historical]: bin/run-druid historical /apache-druid-31.0.0/conf/druid/single-server/quickstart '-Xms1376m -Xmx1376m -XX:MaxDirectMemorySize=2064m'
[Tue Nov 29 16:31:06 2022] Running command[middleManager]: bin/run-druid middleManager /apache-druid-31.0.0/conf/druid/single-server/quickstart '-Xms64m -Xmx64m' '-Ddruid.worker.capacity=2 -Ddruid.indexer.runner.javaOptsArray=["-server","-Duser.timezone=UTC","-Dfile.encoding=UTF-8","-XX:+ExitOnOutOfMemoryError","-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager","-Xms256m","-Xmx256m","-XX:MaxDirectMemorySize=256m"]'
[Tue Nov 29 16:31:06 2022] Running command[broker]: bin/run-druid broker /apache-druid-31.0.1/conf/druid/single-server/quickstart '-Xms1187m -Xmx1187m -XX:MaxDirectMemorySize=791m'
[Tue Nov 29 16:31:06 2022] Running command[router]: bin/run-druid router /apache-druid-31.0.1/conf/druid/single-server/quickstart '-Xms128m -Xmx128m'
[Tue Nov 29 16:31:06 2022] Running command[coordinator-overlord]: bin/run-druid coordinator-overlord /apache-druid-31.0.1/conf/druid/single-server/quickstart '-Xms1290m -Xmx1290m'
[Tue Nov 29 16:31:06 2022] Running command[historical]: bin/run-druid historical /apache-druid-31.0.1/conf/druid/single-server/quickstart '-Xms1376m -Xmx1376m -XX:MaxDirectMemorySize=2064m'
[Tue Nov 29 16:31:06 2022] Running command[middleManager]: bin/run-druid middleManager /apache-druid-31.0.1/conf/druid/single-server/quickstart '-Xms64m -Xmx64m' '-Ddruid.worker.capacity=2 -Ddruid.indexer.runner.javaOptsArray=["-server","-Duser.timezone=UTC","-Dfile.encoding=UTF-8","-XX:+ExitOnOutOfMemoryError","-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager","-Xms256m","-Xmx256m","-XX:MaxDirectMemorySize=256m"]'
```

Druid may use up to 80% of the total available system memory.
To explicitly set the total memory available to Druid, pass a value for the memory parameter. For example, `./bin/start-druid -m 16g`.

Druid stores all persistent state data, such as the cluster metadata store and data segments, in `apache-druid-31.0.0/var`.
Each service writes to a log file under `apache-druid-31.0.0/log`.
Druid stores all persistent state data, such as the cluster metadata store and data segments, in `apache-druid-31.0.1/var`.
Each service writes to a log file under `apache-druid-31.0.1/log`.

At any time, you can revert Druid to its original, post-installation state by deleting the entire `var` directory. You may want to do this, for example, between Druid tutorials or after experimentation, to start with a fresh instance.

Expand Down
4 changes: 2 additions & 2 deletions docs/latest/tutorials/tutorial-batch-hadoop.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ For this tutorial, we've provided a Dockerfile for a Hadoop 3.3.6 cluster, which

This Dockerfile and related files are located at `quickstart/tutorial/hadoop/docker`.

From the apache-druid-31.0.0 package root, run the following commands to build a Docker image named "druid-hadoop-demo" with version tag "3.3.6":
From the apache-druid-31.0.1 package root, run the following commands to build a Docker image named "druid-hadoop-demo" with version tag "3.3.6":

```bash
cd quickstart/tutorial/hadoop/docker
Expand Down Expand Up @@ -110,7 +110,7 @@ docker exec -it druid-hadoop-demo bash

### Copy input data to the Hadoop container

From the apache-druid-31.0.0 package root on the host, copy the `quickstart/tutorial/wikiticker-2015-09-12-sampled.json.gz` sample data to the shared folder:
From the apache-druid-31.0.1 package root on the host, copy the `quickstart/tutorial/wikiticker-2015-09-12-sampled.json.gz` sample data to the shared folder:

```bash
cp quickstart/tutorial/wikiticker-2015-09-12-sampled.json.gz /tmp/shared/wikiticker-2015-09-12-sampled.json.gz
Expand Down
2 changes: 1 addition & 1 deletion docs/latest/tutorials/tutorial-batch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ Once the spec is submitted, you can follow the same instructions as above to wai

Let's briefly discuss how we would've submitted the ingestion task without using the script. You do not need to run these commands.

To submit the task, POST it to Druid in a new terminal window from the apache-druid-31.0.0 directory:
To submit the task, POST it to Druid in a new terminal window from the apache-druid-31.0.1 directory:

```bash
curl -X 'POST' -H 'Content-Type:application/json' -d @quickstart/tutorial/wikipedia-index.json http://localhost:8081/druid/indexer/v1/task
Expand Down
2 changes: 1 addition & 1 deletion docs/latest/tutorials/tutorial-ingestion-spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -582,7 +582,7 @@ We've finished defining the ingestion spec, it should now look like the followin

## Submit the task and query the data

From the apache-druid-31.0.0 package root, run the following command:
From the apache-druid-31.0.1 package root, run the following command:

```bash
bin/post-index-task --file quickstart/ingestion-tutorial-index.json --url http://localhost:8081
Expand Down
Loading