SafeModeException: Name node is in safe mode

classorg_1_1apache_1_1hadoop_1_1hdfs_1_1server_1_1namenode_1_1_safe_mode_exception

org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /user/...../test. Name node is in safe mode.

In order to forcefully let the namenode leave safemode, following command should be executed:

$ bin/hadoop dfsadmin -safemode leave

-safemode isn’t a sub-command for hadoop fs, but it is of hadoop dfsadmin.

Run following command so that any inconsistencies in the hdfs might be sorted out.

$ hadoop fsck

Reference:

http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Safemode

 https://www.inkling.com/read/hadoop-definitive-guide-tom-white-3rd/chapter-10/hdfs

Sqoop:Exporting Data From HDFS to MySQL

Step 1: Install and start MySQL if you have not already done so

MySQL Installation Tutorial for instructions of how to install MySQL.

Step 2: Configure the MySQL Service and Connector

Downloadmysql-connector-java-5.0.5.jar file and copy it to $SQOOP_HOME/lib directory.

Step 3: Sqoop Installation

Sqoop Installation Tutorial for instructions of how to install Sqoop.

  • Database and table creation in MySQL

First connect to MySQL

$ mysql -u root -p

Enter password:

Create database ‘testDb’ and use ‘testDb’ database as a current database.

mysql> create database testDb;

mysql> use testDb;

Create table ‘stud1’

mysql> create table stud1(id integer,name char(20)); 

mysql> exit; 
  • HDFS File ‘student’
$hadoop dfs -cat /user/hduser/student 

1,Archana 

2,XYZ 

Sqoop Export

$sqoop export --connect jdbc:mysql://localhost/testDb --table stud1 -m 1 --export-dir /user/hduser/student

This example takes the files in /user/hduser/student and injects their contents in to the “stud1” table in the testDb” database. The target table must already exist in the database.

Note :

If you will get this

Error

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Access denied for user ''@'localhost' to database 'testDb' 

Solution

Grant all privileges on testDb database to user:

mysql> grant all privileges on testDb.* to ''@localhost ;

Table Contents in MySQL

mysql> use testDb; 

mysql> select * from stud1; 

+------+----------+ 
| id   | name     | 
+------+----------+ 
| 1    | Archana  | 
| 2    | XYZ      | 
+------+----------+ 
2 rows in set (0.00 sec) 

Reference:

http://sqoop.apache.org/docs/1.4.0-incubating/SqoopUserGuide.html

Note: The information provided here is best of my knowledge and experiences if at all any modifications are to be made please help me with your valuable suggestions which are always welcome….  🙂

Installing Pseudo- Distributed HBase on Ubuntu

HBase run modes: Standalone and Distributed

Standalone mode: By default HBase runs in standalone mode. In standalone mode, HBase does not use HDFS. 

Distributed mode: Distributed mode can be subdivided into distributed but all daemons run on a single node is pseudo-distributed— and fully-distributed where the daemons are spread across all nodes in the cluster.

Hadoop version support matrix

HBase-0.92.x HBase-0.94.x HBase-0.95
Hadoop-0.20.205

S

X

X

Hadoop-0.22.x

S

X

X

Hadoop-1.0.0-1.0.2[a]

S

S

X

Hadoop-1.0.3+

S

S

S

Hadoop-1.1.x

NT

S

S

Hadoop-0.23.x

X

S

NT

Hadoop-2.x

X

S

S

[a] HBase requires hadoop 1.0.3 at a minimum; there is an issue where we cannot find KerberosUtil compiling against earlier versions of Hadoop.

Where

S = supported and tested,
X = not supported,
NT = it should run, but not tested enough.

Pseudo- Distributed Installation

The hbase-0.94.8 installation is done in below versions of Linux, Java and Hadoop respectively.

UBUNTU 13.4

JAVA 1.7.0_25

HADOOP 1.1.2

I have hduser as a dedicated hadoop system user. I had installed my Hadoop in /home/hduser/hadoop folder. Now I am going to install hbase in /usr/lib/hbase folder.

  • Download hbase<version>.tar.gz stable version from here
  • Enter into the directory where the stable version is downloaded. By default it downloads in “Downloads” directory
$ cd Downloads/
  • Unzip the tar file.
$ tar -xvf hbase-0.94.8.tar.gz
  • Create directory
$ sudo mkdir /usr/lib/hbase
  • move  hbase-0.94.8 to hbase
$ mv hbase-0.94.8 /usr/lib/hbase/hbase-0.94.8
  • Open your hbase/conf/hbase-env.sh and modify these lines
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_25

export HBASE_REGIONSERVERS=/usr/lib/hbase/hbase-0.94.8/conf/regionservers

export HBASE_MANAGES_ZK=true
  • Set the HBASE_HOME path in bashrc file

To open bashrc file use this command

$ gedit ~/.bashrc

In bashrc file append the below 2 statements

export HBASE_HOME=/usr/lib/hbase/hbase-0.94.8

export PATH=$PATH:$HBASE_HOME/bin

 

  • Update hbase-site.xml in HBASE_HOME/conf folder with required properties.

    hbase-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>hbase.rootdir</name>

<value>hdfs://localhost:9000/hbase</value>

</property>

<property>

<name>hbase.cluster.distributed</name>

<value>true</value>

</property>

<property>

<name>hbase.zookeeper.quorum</name>

<value>localhost</value>

</property>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>hbase.zookeeper.property.clientPort</name>

<value>2181</value>

</property>

<property>

<name>hbase.zookeeper.property.dataDir</name>

<value>/home/hduser/hbase/zookeeper</value>

</property>

</configuration>
  • Now check Hadoop version support matrix. If Hadoop is not supported your hbase version then you will get some exception. To fix this simply copy hadoop-core-*.jar from your HADOOP_HOME and commons-collections-*.jar from HADOOP_HOME/lib folder into your HBASE_HOME/lib folder.
  • Extra steps

In /etc/hosts there are two entries:127.0.0.1 and 127.0.1.1.Change the second entry 127.0.1.1 to 127.0.0.1  otherwise it gives error: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing

  • To start Hbase [ First start hadoop ]
hduser@archana:~$ start-hbase.sh

localhost: starting zookeeper, logging to /usr/lib/hbase/hbase-0.94.8/bin/../logs/hbase-hduser-zookeeper-archana.out
 starting master, logging to /usr/lib/hbase/hbase-0.94.8/logs/hbase-hduser-master-archana.out
 localhost: starting regionserver, logging to /usr/lib/hbase/hbase-0.94.8/bin/../logs/hbase-hduser-regionserver-archana.out

jps command list down all currently running processes

hduser@archana:~$ jps

 4334 HQuorumPeer
 2882 SecondaryNameNode
 4867 Jps
 3207 TaskTracker
 2460 NameNode
 4671 HRegionServer
 4411 HMaster
 2977 JobTracker
 2668 DataNode

Hbase Shell

hduser@archana:~$ hbase shell

HBase Shell; enter 'help<RETURN>' for list of supported commands.
 Type "exit<RETURN>" to leave the HBase Shell
 Version 0.94.8, r1485407, Wed May 22 20:53:13 UTC 2013

hbase(main):001:0> create 't1','c1'
  • To stop HBase
HBASE_PATH$ bin/stop-hbase.sh

stopping hbase...............

To use the web interfaces

http://localhost:60010 for master
http://localhost:60030 for region server

  • Reference :

http://hbase.apache.org/book/standalone_dist.html

http://hbase.apache.org/book/standalone_dist.html#confirm

Note:The information provided here is best of my knowledge and experience if at all any modifications are to be made please help me with ur valuable suggestion which are always welcome…. :)

Install Kettle 4.4.0 on Ubuntu 13.04

images

Kettle is Pentaho’s ETL tool, which is also called Pentaho Data Integration (PDI).

  • The Kettle 4.4.0 installation is done in below versions of Linux, Java and Hadoop  respectively.

UBUNTU 13.4

JAVA 1.7.0_25

HADOOP 1.1.2

  • Download kettle stable version from here
  • Enter into the directory where the stable version is downloaded. By default it downloads in “Downloads” directory

     cd /Downloads

  • Unzip the tar file.

    tar -xzf pdi-ce-4.4.0-stable.tar.gz

  • Move data-integration to /bin/pdi-ce-4.4.0

    mv data-integration/ /bin/pdi-ce-4.4.0

  • Create a symlink

    cd  /bin

    ln -s pdi-ce-4.4.0 data-integration

  • To run Spoon:

    cd  /bin/data-integration

    ./spoon.sh

Apache Hadoop 1.1.2 is not compatible with the Apache Hadoop 0.20.x line, and thus PDI doesn’t work with 1.1.2.  Follow following steps to make it compatible 🙂

  • Create Folder “hadoop-112” in hadoop-configuration directory [data-integration /plugins/ pentaho-big-data-plugin/hadoop-configurations].
  • Copy “hadoop-20” folder to “hadoop-112” folder.
  • Replace the following JARs in the client/ subfolder [data-integration /plugins/ pentaho-big-data-plugin/hadoop-configurations/hadoop-112 /lib/client] with the versions from the Apache Hadoop 1.1.2 distribution:
  1.  commons-codec-<version>.jar
  2.  hadoop-core-<version>.jar    
  • Add the following JAR from the Hadoop 1.1.2  distribution to the client/ subfolder as well:

       commons-configuration-<version>.jar

  • Change the property in plugins.properties [ data-integration /plugins/ pentaho-big-data-plugin/] to point to my new folder:

active.hadoop.configuration=hadoop-112

  • Start PDI

    ./spoon.sh

    Reference:

http://funpdi.blogspot.in/2013/03/pentaho-data-integration-44-and-hadoop.html

Note:The information provided here is best of my knowledge and experience if at all any modifications are to be made please help me with ur valuable suggestion which are always welcome….

Hive Installation On Ubuntu

The hive-0.10.0 installation is done in below versions of Linux, Java and Hadoop respectively.

UBUNTU 13.4

JAVA 1.7.0_25

HADOOP 1.1.2

I have hduser as a dedicated hadoop system user. I had installed my Hadoop in /home/hduser/hadoop folder. Now I am going to install hive  in /usr/lib/hive folder.

  • Download hive stable version from this link

http://mirror.tcpdiag.net/apache/hive/stable/

  • Enter into the directory where the stable version is downloaded. By default it downloads in “Downloads” directory
$ cd ~/Downloads
  • Unzip the tar file.

[go to root user by using command: su ]

# tar xzf hive-0.10.0.tar.gz
  • Create directory
# mkdir /usr/lib/hive
  • move  hive-0.10.0 to hive
 # mv hive-0.10.0 /usr/lib/hive/hive-0.10.0

[Exit from root to hduser by using command: su hduser or exit ]

  • Set the HIVE_HOME path in bashrc file

To open bashrc file use this command

hduser@system_name:~$ gedit ~/.bashrc

            In bashrc file append the below 2 statements

export HIVE_HOME=/usr/lib/hive/hive-0.10.0

export PATH=$PATH:$HIVE_HOME/bin
  •  Type hive in command line and now you can see hive shell.
$ hive

hive>
  • Now you can play with Hive 🙂
%d bloggers like this: