Impala 安裝

1 準(zhǔn)備

1.1 新增Impala用戶并加入sudo權(quán)限

chmod u+w /etc/sudoers

vi /etc/sudoers

impala?? ALL=(ALL) NOPASSWD: NOPASSWD: ALL

chmodu-w /etc/sudoers

1.2準(zhǔn)備rpm安裝文件和jar包

1.2.1 見附件內(nèi)的Hadoop5.0.1Impala1.3.1\1.3.1_RPM(impala需要的rpm包)

1.2.2 見附件內(nèi)的Hadoop5.0.1Impala1.3.1\1.3.1Lib_ALL(impala需要的jar包)

1.2.3 比如放置在 /home/impala/ Hadoop5.0.1Impala1.3.1內(nèi)

1.3服務(wù)規(guī)劃

Impala有三個(gè)服務(wù),impala-server,impala-catalog,impala-state-store.

Impla-server需要安裝在hadoop的每個(gè)數(shù)據(jù)節(jié)點(diǎn)上

impala-catalog,impala-state-store可以安裝在某一個(gè)數(shù)據(jù)節(jié)點(diǎn)或則根集群通的機(jī)器.

2 安裝

2.1安裝impala-server

sudo rpm -ivh bigtop-utils-0.7.0+cdh5.0.1+0-1.cdh5.0.1.p0.31.el6.noarch.rpm

sudo rpm -ivh --nodeps impala-1.3.1+cdh5.0.1+0-1.cdh5.0.1.p0.42.el6.x86_64.rpm

sudo rpm -ivh impala-server-1.3.1+cdh5.0.1+0-1.cdh5.0.1.p0.42.el6.x86_64.rpm

sudo rpm -ivh impala-shell-1.3.1+cdh5.0.1+0-1.cdh5.0.1.p0.42.el6.x86_64.rpm?

2.2安裝impala-catalog impala-state-store

sudo rpm -ivh bigtop-utils-0.7.0+cdh5.0.1+0-1.cdh5.0.1.p0.31.el6.noarch.rpm

sudo rpm -ivh impala-1.3.1+cdh5.0.1+0-1.cdh5.0.1.p0.42.el6.x86_64.rpm

sudo rpm -ivh impala-catalog-1.3.1+cdh5.0.1+0-1.cdh5.0.1.p0.42.el6.x86_64.rpm

sudo rpm -ivh impala-state-store-1.3.1+cdh5.0.1+0-1.cdh5.0.1.p0.42.el6.x86_64.rpm

3 配置

3.1 本地包確認(rèn)

先刪除/usr/lib/impala/lib/*.so.*

rm -rf /usr/lib/impala/lib/ libhadoop.so.1.0.0

rm -rf /usr/lib/impala/lib/ libhadoop.so

rm -rf /usr/lib/impala/lib/ libhdfs.so.1.0.0

rm -rf /usr/lib/impala/lib/ libhdfs.so

?

再確認(rèn) $HADOOP_HOME/lib/native的是否存在libhadoop, libhdfs文件,如果不存在需要拷貝,反之不需要操作該步.

Hadoop5.0.1Impala1.3.1\1.3.1_RPM\libhadoop.so.1.0.0

Hadoop5.0.1Impala1.3.1\1.3.1_RPM\libhdfs.so.0.0.0

并創(chuàng)建創(chuàng)建鏈接

ln -s libhadoop.so.1.0.0 libhadoop.so

ln -s libhdfs.so.0.0.0 libhdfs.so

?

3.2 拷貝so文件至impala lib下

sudo cp $HADOOP_HOME/lib/native/*.so*?/usr/lib/impala/lib/

?

3.3 拷貝依賴的第三方j(luò)ar至impala lib下

刪除前最好備份到一個(gè)目錄下

sudo rm -rf /usr/lib/impala/lib/hive*.jar &&

sudo rm -rf /usr/lib/impala/lib/hbase*.jar &&

sudo rm -rf /usr/lib/impala/lib/hadoop*.jar &&

sudo rm -rf /usr/lib/impala/lib/sentry*.jar &&

sudo rm -rf /usr/lib/impala/lib/zookeeper*.jar &&

sudo rm -rf /usr/lib/impala/lib/avro*.jar &&

sudo rm -rf /usr/lib/impala/lib/parquet-hadoop-bundle.jar

?

?

?

?

sudo cp /home/impala/Hadoop5.0.1Impala1.3.1/1.3.1Lib_ALL/*/usr/lib/impala/lib/

?

3.4 修改hadoop相關(guān)配置

3.4.1 建立短路讀的相關(guān)目錄和權(quán)限

sudomkdir -p /var/run/hadoop-hdfs &&

sudochmod u+x /var/run/hadoop-hdfs &&

sudochmod g+x /var/run/hadoop-hdfs &&

sudochmod g+w /var/run/hadoop-hdfs &&

sudochown -R impala:root /var/run/hadoop-hdfs &&

sudo usermod -a -G root impala

?

3.4.2 追加如下內(nèi)容到hadoop的hdfs-site.xml

<property>?

<name>dfs.client.read.shortcircuit</name>?

<value>true</value>?

</property>?

<property>?

<name>dfs.domain.socket.path</name>?

<value>/var/run/hadoop-hdfs/dn._PORT</value>?

</property>?

<property>?

<name>dfs.client.file-block-storage-locations.timeout</name>?

<value>3000</value>?

</property>?

<property>?

<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>?

<value>true</value>?

</property>

<property>
????<name>dfs.client.file-block-storage-locations.timeout.millis</name>
????<value>10000</value>
</property>

?

?

3.4.3 拷貝配置文件至impala

sudo cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml /etc/impala/conf

sudo cp $HADOOP_HOME/etc/hadoop/core-site.xml /etc/impala/conf

sudo cp $HIVE_HOME/conf/hive-site.xml /etc/impala/conf

?

?

3.5 修改 impala catalog和state store參數(shù)

sudo vi /etc/default/impala

IMPALA_CATALOG_SERVICE_HOST=wxdb01

IMPALA_STATE_STORE_HOST=wxdb01

備注: 這里需要寫impala catalog和state store服務(wù)所在的主機(jī)名

3.6 拷貝hive元數(shù)據(jù)的驅(qū)動(dòng)

示例:

sudo cp $HIVE_HOME/lib/mysql-connector-java-5.1.26-bin.jar/var/lib/impala/

3.7.1 啟動(dòng) impala-catalog impala-state-store

sudoservice impala-catalog start

sudoservice impala-state-store start

sudo service impala-catalog status

sudo service impala-state-store status

?

3.7.2 啟動(dòng) impala-server

sudoservice impala-server start

sudo service impala-server status

?

service impala-server status &&service impala-catalog status && service? impala-state-store status

4 問題排查

4.1 impala日志

目錄在 /var/log/impala內(nèi)

4.2 hadoop日志

目錄在 $HADOOP_HOME/logs內(nèi)

?

4.3 hadoop日志

?

Hdfs賦權(quán) /hive/warehouse777.

?

?

?

-----------------Impala安裝問題記錄:

Unable to findJava. JAVA_HOME should be set in /etc/default/bigtop-utils

?

解決辦法:采用rpm方式安裝jdk,

rpm -ivh oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

安裝路徑為: /usr/java/jdk1.7.0_67-cloudera

?

?

?

與50位技術(shù)專家面對面20年技術(shù)見證，附贈(zèng)技術(shù)全景圖

總結(jié)

以上是生活随笔為你收集整理的Impala手动安装的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。