Skip to content

Commit

Permalink
feat(hadoop): upgrade to hadoop 3.2.1 (#307)
Browse files Browse the repository at this point in the history
  • Loading branch information
lyogev authored May 4, 2020
1 parent 18abcf9 commit 83faf0d
Show file tree
Hide file tree
Showing 24 changed files with 289 additions and 488 deletions.
6 changes: 2 additions & 4 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,9 @@ if: NOT commit_message =~ /^Setting version to.*SNAPSHOT$/
env:
global:
- SPARK_VERSION=2.4.5
- HADOOP_VERSION=2.9.2
- HIVE1_VERSION=1.2.2
- HUDI_HIVE1_VERSION=0.4.7
- HADOOP_VERSION=3.2.1
- HIVE_VERSION=2.3.3
- HUDI_VERSION=0.5.1-incubating
- HUDI_VERSION=0.5.2-incubating
before_script:
- export -f travis_fold
- export -f travis_time_start
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -455,7 +455,7 @@ Metorikku supports reading/writing with [Apache Hudi](https://github.com/apache/
Hudi is a very exciting project that basically allows upserts and deletes directly on top of partitioned parquet data.

In order to use Hudi with Metorikku you need to add to your classpath (via ```--jars``` or if running locally with ```-cp```)
an external JAR from here: https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark-bundle_2.11/0.5.1-incubating/hudi-spark-bundle_2.11-0.5.1-incubating.jar
an external JAR from here: https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark-bundle_2.11/0.5.2-incubating/hudi-spark-bundle_2.11-0.5.2-incubating.jar

To run Hudi jobs you also have to make sure you have the following spark configuration (pass with ```--conf``` or ```-D```):
```properties
Expand Down
2 changes: 1 addition & 1 deletion build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ libraryDependencies ++= Seq(
"org.influxdb" % "influxdb-java" % "2.14",
"org.apache.kafka" %% "kafka" % "2.2.0" % "provided",
"za.co.absa" % "abris_2.11" % "3.1.1" % "provided" excludeAll(excludeAvro, excludeSpark),
"org.apache.hudi" %% "hudi-spark-bundle" % "0.5.1-incubating" % "provided" excludeAll excludeFasterXML,
"org.apache.hudi" %% "hudi-spark-bundle" % "0.5.2-incubating" % "provided" excludeAll excludeFasterXML,
"org.apache.parquet" % "parquet-avro" % "1.10.1" % "provided",
"org.apache.avro" % "avro" % "1.8.2" % "provided",
"org.apache.hive" % "hive-jdbc" % "2.3.3" % "provided" excludeAll(excludeNetty, excludeNettyAll)
Expand Down
2 changes: 1 addition & 1 deletion docker/hive/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ ENV MYSQL_CONNECTOR_VERSION=5.1.47
RUN wget -q https://repo1.maven.org/maven2/mysql/mysql-connector-java/$MYSQL_CONNECTOR_VERSION/mysql-connector-java-$MYSQL_CONNECTOR_VERSION.jar \
&& mv mysql-connector-java-$MYSQL_CONNECTOR_VERSION.jar $HIVE_HOME/lib

ENV HUDI_VERSION=0.5.1-incubating
ENV HUDI_VERSION=0.5.2-incubating
RUN wget -q https://repo1.maven.org/maven2/org/apache/hudi/hudi-hive-bundle/$HUDI_VERSION/hudi-hive-bundle-$HUDI_VERSION.jar \
&& mv hudi-hive-bundle-$HUDI_VERSION.jar $HIVE_HOME/lib
RUN wget -q https://repo1.maven.org/maven2/org/apache/hudi/hudi-hadoop-mr-bundle/$HUDI_VERSION/hudi-hadoop-mr-bundle-$HUDI_VERSION.jar \
Expand Down
89 changes: 0 additions & 89 deletions docker/hive1/Dockerfile

This file was deleted.

156 changes: 0 additions & 156 deletions docker/hive1/atlas/import_hive.sh

This file was deleted.

4 changes: 0 additions & 4 deletions docker/hive1/log4j.json.properties

This file was deleted.

Loading

0 comments on commit 83faf0d

Please sign in to comment.