Getting started with apache flume, retrieve Twitter tweets data into HDFS using flume

This post will explain you about, flume installation , retrieve tweets to HDFS, Twitter app creation for development.

1.Download latest flume
2.Untar the downloaded tar file in which ever location you want.
sudo tar -xvzf apache-flume-1.6.0-bin.tar.gz
3.Once you have done the above steps, u can start the ssh localhost, if not connected to ssh server
4.Start the dfs ./start-dfs.sh
5.Start the ./start-yarn.sh
6.Go up to bin folder where flume has been extracted
7.Here I am extracted flume under /usr/local/flume-ng/apache-flume-1.6.0-bin
8.First Download the flume-sources-1.0-snapshot.jar and move that jar into inside /usr/local/flume-ng/apache-flume-1.6.0-bin/lib/

9.Once we have done this, Now we need to set the java path and snapshot.jar path details in flume-env.sh

/usr/local/flume-ng/apache-flume-1.6.0-bin/conf>sudo cp flume-env.sh.template flume-env.sh
   /usr/local/flume-ng/apache-flume-1.6.0-bin/conf> sudo gedit flume-env.sh

10.Now we need to register our application with twitter dev
11.open the twitter , if you are not having sign in details, then please signup the same. Once you have signed up then use the twitter apps

12. Click on Create New App Enter the required details

13. Check the I Agree checkbox

14.Once application has been created then twitter page will be look like this.


15.Click on the Keys and Access Tokens tab and copy consumer key and consumer secret key and paste any notepad

16.Click Create my access token

17.It will generate access token and access token secret, copy these 2 values and place it notepad

18.Create flume.conf file under /usr/local/flume-ng/apache-flume-1.6.0-bin/conf and paste the below details.

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.


# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per agent, 
# in this case called 'TwitterAgent'

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = joeTPv3pjfc471vfMH0lmP
TwitterAgent.sources.Twitter.consumerSecret = PydW6v8aYoiHOm1gOe0qdQUboHua9HaTYzo1Vg3muu4xJhF
TwitterAgent.sources.Twitter.accessToken = 714023179098857474-4ZaCUhAxbcZCKdnvijGvyuWQteEv
TwitterAgent.sources.Twitter.accessTokenSecret = yhMgQrmrUZht2nMn6Ts1NbclmzuBda2xvtIIvVoneQ 
TwitterAgent.sources.Twitter.keywords = hadoop, big data, analytics, bigdata, cloudera, data science, data scientiest, business intelligence, mapreduce, data warehouse, data warehousing, mahout, hbase, nosql, newsql, businessintelligence, cloudcomputing

TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/user/flume/tweets/%Y/%m/%d/%H/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000

TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

19. Once that is done , then we need to run the twitter agent in flume.

/usr/local/flume-ng/apache-flume-1.6.0-bin>./bin/flume-ng agent -n TwitterAgent -c conf -f /usr/local/apache-flume-1.6.0-bin/conf/flume.conf

20. once it is started , wait for some time and click the ctrl+C and now it's time to see the tweets in HDFS file.
21. Open the browser which is there in unix machine and browse the same. go to /user/flume/tweets and see the tweets

http://localhost:50075

22. we can see data similar as shown in twitter, then the unstructured data has been streamed from twitter on to HDFS successfully. Now we can do analytics on this twitter data using Hive.

This is how we can bring live tweets data into HDFS and we can do the analytics using hive.
Thank you very much for viewing this post

Zookeeper Basics, HBase Zookeeper

This post will explain you basics about Zookeeper.
Why Zookeeper
Zookeeper is coordinating mechanism for HBase.

It is mainly used in cluster environment.
Target market for Zookeeper


Zookeeper Data Model
1. Hierarchal namespace (like File System)
2. Each Z node as data and children
3. Data is read and write in its entity

Zookeeper will provide services like, if any one server failure , then another server will be accessible, without any delay.
Zookeeper will provide
1. Wait free
2. Simple , Robust, Good Performance
3. Turned for Read Dominant Workloads
4. Familiar Models and interfaces
5. Need to be able to wait efficiency
Zookeeper and Hbase
Master failover
Region servers and master discovery via zookeeper
1. HBase clients connect zookeeper to find configuration details
2. Region server and master failure detection.
How HBase and Zookeeper will work?


Master
If more than one master, then they fight
Root Region Server
1. This Z node holds the location of the server hosting the root of all the tables in HBase.
2. A directory in which there is a znode per HBase region server
3. Region servers register themselves with zookeeper when they come online.
On region server failure (detected via ephemeral znodes and notification via zookeeper), the
master splits the edits out per region.

These are the basic details about zookeeper.
Thank you very much for viewing this post.

Getting started with HBase, HBase Compactions, Load data into HBase Using Sqoop

This post will explain you abou HBase Compactions, how to install HBase and start the Hbase, HBase Basic operations.
How to load data into HBase using sqoop.
HBase Compactions

1. HBase writes out immutable files as data is added
a). Each store consists rowkey-ordered files.
b).Immutable- more files accumulated over time.
2. Compaction rewrite several files into one
a).Lesser files – Faster reads
3. Major compaction rewrites all files in a store into one
a).Can drop deleted records and older versions
4. In a minor compaction, files to compact are selected based on a heuristic.

How to install HBase and start the same.
1. First download latest version HBase from http://www.apache.org/dyn/closer.cgi/hbase/ or https://hbase.apache.org/
2. Once Downloaded, then try to un tar the same.
3. tar –xvzf hbase-1.0.1.1-hadoop1-bin.tar.gz
4. Go to /usr/local/hbase/hbase-1.0.1.1/
5. ./bin/start-hbase.sh
6. Once it is started , then
7. ./bin/hbase shell



We can see the shell window to work with. Try to enter list. It will show you list of existing tables.
If we are able to execute this command means our hbase started successfully without any issue.

Hbase>list

Now we will see sql operations through HBase.
HBase Basic operations
Create a table syntax
Create ‘table_name’ , ‘column_family’

HBase>Create ‘htest’,’cf’

Insert data

put ‘table_name’ ,’row_key1’,’column_family:columnname’,’v1’

Update data

put ‘table_name’ ,’row_key1’,’column_family:columnname’,’v2’

Select few rows

get  ‘table_name’ ,’row_key1’

Select whole table

scan ‘table_name’

Delete particular row value

delete  ‘table_name’ ,’row_key1’,’column_family:columnname’

Alter existing table
Before alter the table, first we need to disable the same table

disable ''
alter '' ,{NAME=''}

Drop the table
First disable the existing table, which we supposed to be drop

Hbase>Disable  ‘testdrop1’
Hbase>drop  ‘testdrop1’

How to create table from java and insert the data to the same in HBase table ?
First open eclipse-> create a new project ->class->HBaseTest.java
Copy and paste the below code. If any compilation errors then add the respective Hbase jars the same

Public class HBaseTest {
  Public static vaoid main(String args[]) throws  IO Exception{
 //We need Configuration object to tell the client where to connect.
//when we create a HBaseConfiguration , it reads whatever we have set into our hbase-site.xml, and //hbase-default.xml, as long as these can be found in the classpath
    Configuration config = HBaseConfiguration.create();
 //Instantiate HTable  object, that connects the testHBaseTable
//Create a table with name  testHBaseTable,  if it is not available.
   HTable table = new HTable(config,” testHBaseTable”);
//To Add a row use Put, Put constructor takes the name of the row which we want to insert into a //byte array, in HBase , the Bytes class has utility to converting all kinds of java types to byte arrays.
Put p = new Put(“testRow”);

//to set the value to row , we would like to update in the row testRow .
//Specify the column family. Column qualifier and value of the table.
//cell we would like to update then the column family must already exist.
//in our table schema the qualifier can be anything
//All must be specified as byte arrays as hbase is all about byte arrays.
p.add(Bytes.toBytes(“littleFamily”),Bytes.toBytes(“littleQualifier”),Bytes.toBytes(“little Value”));
//Once we have updated all the values for Put instance. Then HTable#put method takes Put instance  //we have building and pushes the change we made into HBase.
table.put(p);
//Now, to retrieve the data which we have just wrote the table;
Get   g = new Get(Bytes.toBytes(“testRow”)
Result   r = table.get(g);
byte [] value = r.getValue(Bytes.toBytes(“littleFamily”),Bytes.toBytes(“littleQualifier”));
String ValueString = Bytes.toString(value);
System.out.println(“GET:”+valueString);
//Some times we don’t know about row name, then we can use the scan to retrieve all the data from //the table
Scan s = new Scan();
s.addColumn(Bytes.toBytes(“littleFamily”), Bytes.toBytes(“littleQualifier”));
ResultScanner scanner = table.getScanner(s);
try{
   for (Result rr = scanner.next();rr!=null;rr=scanner.next()){
    System.out.println(“Found Row record:”+ rr);
  } 
}
finally{
scanner.close();
}
 }
 }

Different ways to load the data into HBase
1. HBase Shell
2. Using Client API
3. Using PIG
4. Using SQOOP

How to load data into HBase using SQOOP?
Sqoop can be used directly import data from RDBMS to HBase.
First we need to install sqoop.
1. Download sqoop http://www.apache.org/dyn/closer.lua/sqoop/1.4.6
2. Untar the Sqoop

tar -xvzf sqoop-1.4.6.bin__hadoop-0.23.tar.gz  

3. Go upto bin. then run the executing below command.

sqoop import
               --connector jdbc:mysql://\
                --username  --password 
                --table
                --hbase-table 
                --column-family 
                --hbase-row-key 
                --hbase-create-table

This is how we will work with HBase.
Thank you very much for viewing this post.

HBase Basics, HBase Architecture, Getting started with No SQL Database HBase, HBase Components

This post will explain you about History of Hbase and HBase Architecture,Basic details about HBase, Different types of No SQL Databases
History of HBase
Started in Google.
GFS -> HDFS
MapReduce-> MapReduce
Big Table -> Apache HBASE

Any SQL system – RDBMS
1. Users data is increasing, then we will implement cache mechanism to improve performance.
2. Cache mechanism also having certain limlits.
3. Remove indexing.
4. Avoiding joins
5. Materialized view .
If we use above, then advantages of RDBMS has gone.
Google also faced same problem, then they started with Big Table.
For faster performance we use HBase.
What ever the features hive will not support like crud operations, we can do with HBase.
If anything need to be updated in real time access ,HBase if very useful.
Ad Targeting in real time is very faster.
What is Common problem with existing data processing with Hadoop or Hive?
1. Huge Data
2. Fast Random Access
3. Structured Data
4. Variable Schema- will support to enhance or increase the column names at runtime, which is RDBMS is not supported.
5. Need of compression
6. Need of Distribution(Shading)
How Traditional System(RDBMS) will solve this?
Case: If we want to design Linkedin database to maintain connections?
There 2 tables
1. Users – id,Name,Sex,age
2. Conenctions- User_id,Connection_id,type
But in case of HBase, we can save all the details about users and connections in same column family.
Characteristics of Probable
1. Distributed Database
2. Sorted Data
3. Sparse Data Store
4. Automatic Sharding.

Sorted Data
Example : How data stored in sorted way?
1. www.abc.com
2. www.ghf.com
3. Mail.abc.com
When ever user try’s to access abc.com , then mail.abc.com will not be returned in case of normal storage.

If we use sorted storage then data will be stored like below.
com.abc.www
com.abc.mail
com.ghf.www

If we store like above, then it is easy to access the same.
Sparse Data store
This is mathematical term. If there is null value for particular column , then it will not store.

No SQL Landscape


1.Each No SQL databases as mentioned above is same, they have developed for their purpose.
2.Dynamo is developed by Amazon and it available in Cloud. We can access the same.
3.Cassandra developed by Facebook and they will be using the same. It is combination of Dynamo and HBase, all the features available in Cassandra.

Any No SQL database will have all the characteristics.
It will satisfy only two property at the same time.

HBase Definition
It is a non -relational (NoSQL)database, which stores data in key value pair and it is also called as hadoop database.
1. It Sparse
2. Distributed
3. Multi –dimensional (table name,column name,timestamp) etc..
4. Sorted Map
5. Consistent

Difference between HBase and RDBMS


When to use HBase


When not to use HBase?
1. When you have only few thousand or millions records then it is not advisable to use HBase.
2. Lacks RDBMS commands, if our database requires sql commands then also not go for Hbase
3. When we have hardware less than 5 Data Nodes when replication factor is less than 3, then no need of HBase. It will overhead for system

HBase can run in local system –but this should be considered for a development configuration.
How face book uses HBase as their Message System

1.facebook monitored their usage and figured our what they really needed.
What they needed was a system that could handle two types of data pattern
1. A short set of temporal data that tends to be volatile
2. An ever growing set of data that can be accessed rarely.

1. Real Time
2. Key Value
3. Linearly
4. Big Data
5. Column oriented
6. Distributed
7. Robust
8. Scalable
9. Open source
These are the characteristics of HBase.
HBase is using not only facebook.But also twitter,yahoo etc… they will use to process their large volume data.

Major components of HBase
1. The HBase Master
It will store all the Hbase table and it will coordinate
2. The HRegion server
Actual data will be stored in this server
3. The HBase Client
We will interact to do the crud operations and processing the data

It is same like name node in HDFS
How data distribution will happen in HBase?

We are having data rows from  A to Z
     Rows                                               Servers
    A1,A2 –                Region  Null - A3                  Region server1
    B2,B3,B23,B43-         Region  B2 – B43                   Region server2      
    K1,K2,Z30 -            Region K1 – Z30                    Region server3

How HBase will write data to the file?

1.Every HBase requires confirmation from both Write Ahead Log (WAL) and the MemStore.
2.The two steps ensures that every write to HBase happens as fast as possible while maintaining durability.
3.The Memstore is flushed to a new HFile when it fills up.
4.Usally Memstore default size 256MB, once it is filled up then , it move that information to HFile it's default size is 64 KB.
5.It will be act as a immutable object.
HBase Read File
1.Data is reconciled from the block cache, The Mem-Store and the HFiles to give the client an up to date view of the rows which client requested for.
2.HFiles contain a snapshot of the Memstore at the point when it was flushed. Data for a complete row can be stored across multiple HFiles.
3. In order to read complete row, HBase must read across all HFiles that might contain information for that row in order to compose the complete record.




HFile Compaction

All HFiles will be compacted and put as Compacted HFile.
HBase Components
1. Region – a range of rows stored together
2. Region servers- serves one or more regions
a. A region served by only one region server
3. Master Server – Daemon responsible for managing HBase cluster.
4. HBase stores its data into HDFS- Relies on HDFS’s High availability and fault tolerance.
HBase Architecture

This architecture will explain you about how Hbase will work.

This is Basics about HBase. My next post you can see How to install and work with HBase.
Thank you very much for viewing this post.