Metabase搭建手册:使用SparkSQL连接Hive

1.下载jar包

mkdir -p /usr/local/metabase
wget http://downloads.metabase.com/v0.32.10/metabase.jar -O /usr/local/metabase/metabase.jar

2. 下载sparksql依赖包

CDH5.16.x版本以下必须手动编译, 官方提供的依赖包只支持CDH6以上版本

2.1 官方下载(CDH6.0+)

mkdir -p /usr/local/metabase/plugins
wget https://s3.amazonaws.com/sparksql-deps/metabase-sparksql-deps-1.2.1.spark2-standalone.jar -O /usr/local/metabase/plugins/metabase-sparksql-deps-1.2.1.spark2-standalone.jar

2.2 源码编译(CDH5.X)

2.2.1 下载sparksql依赖项目的源码

cd /export/git && git clone https://github.com/metabase/sparksql-deps.git
cd sparksql-deps

2.2.2 修改project.clj文件

vim project.clj

将 第8行的 org.spark-project.hive/hive-jdbc "1.2.1.spark2" 替换成以下内容

org.spark-project.hive/hive-jdbc "0.13.1"

将第一行的 (defproject metabase-sparksql-deps "1.2.1.spark2" 替换成以下内容:

(defproject metabase-sparksql-deps "0.13.1"

【参考】

https://github.com/metabase/metabase/issues/7528
https://github.com/metabase/sparksql-deps

2.2.3 打包

lein uberjar

2.2.4 打包完成后,在target目录下有metabase-sparksql-deps-0.13.1-standalone.jar文件,复制到服务器上metabase/plugins 目录下

scp /export/git/sparksql-deps/target/metabase-sparksql-deps-0.13.1-standalone.jar cdh1:/usr/local/metabase/plugins/

3. 配置使用MySQL作为元数据库

可以先不做,参考https://www.liudon.org/1344.html 迁移也可

3.1 创建metabase库和用户

CREATE DATABASE metabase DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON metabase.* TO 'metabase'@'%' IDENTIFIED BY 'metabase'; 

3.2 配置环境变量

cat >> /etc/profile << EOF
#For Metabase 
export MB_DB_TYPE=mysql
export MB_DB_DBNAME=metabase
export MB_DB_PORT=3306
export MB_DB_USER=metabase
export MB_DB_PASS=metabase
export MB_DB_HOST=localhost
EOF
source /etc/profile

3.3 配置https

3.5 配置Hadoop账号和Yarn队列(重要)

(不做的话会报return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask错误,观察/var/log/hive/hadoop-cmf-hive-HIVEMETASTORE-cdh2.log.out 可知是账号权限问题)

4 启动metabase

nohup java -jar /usr/local/metabase/metabase.jar >> /var/log/metabase/metabase.log 2>&1 &

4.1 日志

tail -f /var/log/metabase/metabase.log

4.2 停止

ps -ef | grep 'metabase.jar' | grep -v 'grep' | awk '{print $2}' | xargs kill -9

5. 进入metabase, 配置hive server2的连接

;transportMode=http&useUnicode=true&characterEncoding=utf8&autoReconnect=true&tinyInt1isBit=false&useSSL=false
添加新评论