Getting started Hadoop with oracle or vmware virtual box and Ubuntu

Hadoop installation with Single DataNode( VMware or Oracle virtual box)
Download latest version VM ware from the below link
http://www.traffictool.net/vmware/
Download Oracle virtual box from the below site and install the same in local system.
http://www.oracle.com/technetwork/server-storage/virtualbox/downloads/index.html
Run the Virtual box(VirtualBox.exe) Application
click on new ->

And click on Next->Next-And create virtual box
Once that’s done virtual box will look like this. Select the Ubuntu downloaded package.




Start the Virtual box, then provide password from which user you want to start.

Once virtual box started then screeb will look like this


Open the terminal, by right click on the screen or search for terminal and open the same.



Command:to update the ubuntu
1. sudo apt-get update
Once update is complete

Command: install openssh server
2. sudo apt-get install openssh–server
Command: create a hadoop directory
3. mkdir /usr/local/hadoop
Download the hadoop latest version from below link
http://hadoop.apache.org/releases.html
copy to virtual box and extract the tar file
Here I extracted under /usr/local/hadoop/
Command: to extract the tar file
4. tar -xvf .tar.gz
After extracting enter this command ls –lrt , you can see the list of folders related to hadoop
Command: To add hadoop to the group
5. sudo addgroup hadoop
Command: create new user called hduser
6. sudo adduser --ingroup hadoop hduser

Command: assign hduser to sudo
7. sudo adduser hduser sudo
Command: change the owner for hadoop as hduser
8. sudo chown –R hduser:hadoop /usr/local/hadoop
Command: switch to hduser
9. su – hduser

Command: install ssh
10. sudo apt-get install ssh
Command: generate a ssh key
11. ssh-keygen -t rsa –P ""
/home/hduser/.ssh/id_rsa
Command: copy id_rsa.pub key to authorized_keys
12. cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Command: install vim editor
13. sudo apt-get install vim
Command: Edit the sysctl.conf file to dispable few of the ipv6 realted configuration
14. sudo gedit /etc/sysctl.conf or sudo vi /etc/sysctl.conf
Add below lines

net.ipv6.conf.all.disable_ipv6=1
                   net.ipv6.conf.default_ipv6=1
                  net.ipv6.conf.io.disable_ipv6=1
              

Command:Start the ssh
15. ssh localhost
Command: get the updates
16. sudo apt-get update
Command: edit the bashrc file to add the path of java and hadoop
17. sudo vi ./bashrc or sudo gedit ./bashrc

export  HADOOP_HOME = /usr/local/hadoop
         export  JAVA_HOME=/usr   [or] where ever your java installed location

Command: Source the bashrc file
18. source .bashrc

Command: Now check the version of java and hadoop
19. java –version
20. hadoop version
Command:Create a data directory inside /usr/local/hadoop
21. mkdir /usr/loca/hadoop/data
Command: edit the hadoop_env.sh file to add the configuration
22. sudo gedit /usr/loca/hadoop/etc/hadoop/hadoop_env.sh

export JAVA_HOME=/usr 
        export HADOOP_OPTS=”$HADOOP_OPTS –Djava.net.preferIPv4Stack= true  -Djava.library.path=$HADOOP_PREFIX/lib”

Command: edit the yarn_env.sh file to add the configuration
23. sudo gedit /usr/loca/hadoop/etc/hadoop/yarn_env.sh

export HADOOP_CONF_LIB_NATIVE_DIR=${HADOOP_PREFIX:-“lib/native”}
        export HADOOP_OPTS=” Djava.library.path=$HADOOP_PREFIX/lib”

Now we need to edit the some of the hadoop related files, to start the single node
Go to /usr/local/hadoop/etc/hadoop$
Command: Edit the existing file and add the below configuration
24. sudo gedit core-site.xml

fs.default.name
hdfs://localhost:9000


hadoop.tmp.dir
/usr/local/hadoop/data

Command: Rename mapred-site.xml.template to mapred-site.xml
Go to /usr/local/hadoop/etc/hadoop
25. mv mapred-site.xml.template mapred-site.xml
26. sudo gedit mapred-site.xml

    
       mapreduce.framework.name
       yarn
 

Then close this file
Edit the hdfs-site.xml,
Command: to edit the hdfs-site.xml
27. sudo gedit hdfs-site.xml

dfs.replication
3

Command:Edit the yarn.xml
28. sudo gedit yarn.xml

   
       yarn.nodemanager.aux-services 
       mapreduce_shuffle
  
 
        yarn.nodemanager.aux-services.mapreduce_shuffle.class
       org.apache.hadoop.mapred.ShuffleHandler
  


       yarn.resourcemanager.resource-tracker.address
       localhost:8025
  


        yarn.resourcemanager.scheduler.address
       localhost:8030
  


        yarn.resourcemanager.address
       localhost:8050
  

Command: Need to format the namenode
29. /usr/local/hadoop/bin/hadoop namenode –format
After this format done then we need to start the dfs and yarn
30. /usr/local/hadoop/sbin/start-dfs.sh
31. /usr/local/hadoop/sbin/start-yan.sh
Command: to display all the running datanodes and namemodes
32. jps



This is how we can setup the hadoop using oracle/vmware virtual box.

Thank you for viewing this post.

Getting started with web2py using pythonanywhere web hosting service (cloud)

This post tells you, how to deploy and execute python code written using web2py framework in
pythonanywhere web hosting service kind of cloud for python.
First write any sample code using web2py framework. Sample codes you can check my previous posts like getting started with web2py , blog app using web2py

Once you have completed the simple project, then we need to deploy the same using the cloud.
Web2py can be deployed any web hosting services, now we will look how we can deploy using https://www.pythonanywhere.com/




Click on Signup here! (if you are not sign up yet)




Click on Create a Beginner account and provide the required details, it is enough to post any details in internet.




After successful sign up and Login then, you can redirected to pythonanywhere


Click on DashBoard Then Click on Web




Now we need to create new web app (Click on Add anew web app).




Pythonanywhere will support so many python frameworks, Select the web2py , since we are implementing application using web2py.





Provide the password , which is required to access the application. It will created directory (/home/siva82k/web2py/) with my username.
Click on Next, which will create url for you


Now you can check your application through internet, usually welcome application will be copied to your account.
My case my url will be http://siva82k.pythonanywhere.com, if you try to click on this you can redirected to your application.




Now it’s time to deploy our existing code into pythonanywhere site. First go to our application, where we have written our code, click on pack all as shown in below image.




Then save the code in local system.Once it is completed then go to your pythonanywhere site.
Click on Administrative Interface, and the provide the password, which you have given while creating web2py account.



After successful Login, it will redirected to below page



We need to upload the file into python anywhere site . Provide the details under upload and install packed application
I am providing Application name as sivaweb2py
Upload a package from your local system, Earlier where you have downloaded.
Then click on install, our application got installed on pythonanywhere machine.
Now what ever we did in local host machine same thing available in pythonanywhere internet.
You can check my previous posts related to web2py examples.
Earlier we checked Role based access, same thing we will check in pythonanywhere
Click on https://siva82k.pythonanywhere.com/sivaweb2py/blog/view

It will ask the username and password , in my case I have provided user only (siva82k@gmail.com), have access to post the blog.
https://siva82k.pythonanywhere.com/sivaweb2py/blog/post



If we provide correct user name and password, it will take us to post the blog.



If we provide other details, other than post access then it will say you are not authorized.



One more example which we have worked earlier, basics to add the 2 numbers
https://siva82k.pythonanywhere.com/sivaweb2py/basics/request_args/10/20



We can test whatever we did in our previous examples in local system, same thing available in internet.
This is how we can deploy our web2py code using pythonanywhere.
Thanks for viewing this post

State design pattern with java

State pattern, which allows objects to behave in different ways depending on internal state. State is used when you need a class to behave differently, such as performing slightly different computations, based on some arguments passed through to the class.

Problem: Vehicle different status details

1. VehicleNotificationStatus – having different status details about vehicle.

package com.siva;

public enum VehicleNotificationStatus {
 
 VEHICLE_ORDERED("Vehicle-Ordered"),
 VEHICLE_ORDER_SUBMITTED_FOR_BODY_BUILD("Vehicle-Order-Submitted-For-Body-Build"),
 VEHICLE_BODY_SHOP_COMPLETED("Vehicle-Body-shop_completed"),
 VEHICLE_DELIVERED("Vehicle-Delivered"),
 INVALID("Invalid");
 
 VehicleNotificationStatus(String _type){
  this.type=_type;
 }
 private String type;
 

 public String getType() {
  return type;
 }
 public void setType(String type) {
  this.type = type;
 }

}

2. VehicleStatus interface to evaluate the different statues.

package com.siva;

  public interface VehicleStatus {
 
 public VehicleNotificationStatus evaluateVehicleStatus(String status);

  }

3. Different statues classes to implement the VehicleStatus interface.

VehicleOrdered is one type of status class is to implement VehicleStatus

package com.siva.status;

import com.siva.VehicleNotificationStatus;
import com.siva.VehicleStatus;

public class VehicleOrdered implements VehicleStatus{

 @Override
 public VehicleNotificationStatus evaluateVehicleStatus(String status) {
  System.out.println("VehicleOrdered state start");
    if(VehicleNotificationStatus.VEHICLE_ORDERED.getType().equalsIgnoreCase(status)){
    return VehicleNotificationStatus.VEHICLE_ORDERED;
  }
  return VehicleNotificationStatus.INVALID;
    
 }
 

}

2. VehicleForBodyBuild is one type of status class is to implement VehicleStatus

package com.siva.status;

import com.siva.VehicleNotificationStatus;
import com.siva.VehicleStatus;

public class VehicleForBodyBuild implements VehicleStatus{

 @Override
 public VehicleNotificationStatus evaluateVehicleStatus(String status) {
  System.out.println("VehicleForBodyBuild state start");
    if(VehicleNotificationStatus.VEHICLE_ORDER_SUBMITTED_FOR_BODY_BUILD.getType().equalsIgnoreCase(status)){
    return VehicleNotificationStatus.VEHICLE_ORDER_SUBMITTED_FOR_BODY_BUILD;
  }
  return VehicleNotificationStatus.INVALID;
    
 }

}

3. VehicleDelivered is one type of status class is to implement VehicleStatus

package com.siva.status;

import com.siva.VehicleNotificationStatus;
import com.siva.VehicleStatus;

public class VehicleDelivered  implements VehicleStatus{

 @Override
 public VehicleNotificationStatus evaluateVehicleStatus(String status) {
  System.out.println("VehicleDelivered state start");
    if(VehicleNotificationStatus.VEHICLE_DELIVERED.getType().equalsIgnoreCase(status)){
     System.out.println();
    return VehicleNotificationStatus.VEHICLE_DELIVERED;
  }
  return VehicleNotificationStatus.INVALID;
    
 }
}

4. Now we need to write Factory class which can be implement all this status details.

package com.siva;

import com.siva.status.VehicleDelivered;
import com.siva.status.VehicleForBodyBuild;
import com.siva.status.VehicleOrdered;

public class VehicleStatusFactory {
 
 public static VehicleStatus createStatus(VehicleNotificationStatus status){
  if(status.equals(VehicleNotificationStatus.VEHICLE_ORDERED)){
   return new VehicleOrdered();
  }
  else if(status.equals(VehicleNotificationStatus.VEHICLE_ORDER_SUBMITTED_FOR_BODY_BUILD)){
   return new VehicleForBodyBuild();
  }
  else if(status.equals(VehicleNotificationStatus.VEHICLE_DELIVERED)){
   return new VehicleDelivered();
  }
  else
   return null;
 }

}

5. We have completed the all the classes related to State pattern, now we need to write test class to see the results.

package com.siva;

public class TestStateDesignPattern {
 
 public static void main(String[] args) {
  
  VehicleStatus vehicleStatus =VehicleStatusFactory.createStatus(VehicleNotificationStatus.VEHICLE_ORDERED);
  VehicleNotificationStatus notificationStatus = vehicleStatus.evaluateVehicleStatus(VehicleNotificationStatus.VEHICLE_ORDERED.getType());
  System.out.println(notificationStatus);
  vehicleStatus =VehicleStatusFactory.createStatus(VehicleNotificationStatus.VEHICLE_ORDER_SUBMITTED_FOR_BODY_BUILD);
  VehicleNotificationStatus notificationStatus1 = vehicleStatus.evaluateVehicleStatus(VehicleNotificationStatus.VEHICLE_ORDER_SUBMITTED_FOR_BODY_BUILD.getType());
  System.out.println(notificationStatus1);
  vehicleStatus =VehicleStatusFactory.createStatus(VehicleNotificationStatus.VEHICLE_DELIVERED);
  VehicleNotificationStatus notificationStatus2 = vehicleStatus.evaluateVehicleStatus(VehicleNotificationStatus.VEHICLE_DELIVERED.getType());
  System.out.println(notificationStatus2);
 }

}

Output:

VehicleOrdered state start
VEHICLE_ORDERED
VehicleForBodyBuild state start
VEHICLE_ORDER_SUBMITTED_FOR_BODY_BUILD
VehicleDelivered state start

VEHICLE_DELIVERED

This is how state pattern will work.
Thanks for viewing this post.

Role Based Access control using web2py framework

This post having how to provide Role based access to particular users. More information about web2py framework You can check my previous posts.
Getting started with web2py and blog app using web2py
Usually
Role Based Access control having

Account Registration
Providing Security
Adding to Groups
Access to Groups

and users can be

Administrator
Standard User
Manager

Now we can see how we will provide these roles using web2py

1. Start the web2py and provide the password an d start the server.

2. Click this link to open the web2py homepage http://127.0.0.1:8000/welcome/default/index




Click on the Admin link and provide the password.
Edit the sivaweb2py application




After edit, the application will be look like this





Click on the database administration tab which is under Models section, you can view the below page related to database table details




Now we are going to create role based access for exiting project, which is available in my previous post(blog app using web2py). Click the below link to open the exiting project results
http://127.0.0.1:8000/sivaweb2py/blog/view




Now we need to signup this blog and screen will look like as below.



After entering the details and click on signup then you will be redirected to view page. With message as Logged In




Now we have created user and logged in successfully, Now we can check in the database administrator for db details.





We can edit this record and we can see the details. Click on the 1 and see the details, password is encrypted.
Now we have to create auth group, allowing that only same group users only can post the topics in blog.
Go to Databaseadministrator under Models Section
Click on the db.auth_group
Click on the New Record and provide the Role as blog_users and description as you like




After submit the record table details look like below




Now we need to create authentication membership Click on db.auth_membership




Click on the New Record and assign the new memebership for this user and select the Group ID as blog_users





Now it’s time to provide access control
Edit the our existing blog.py.
If any user want to post requires authentication and who ever there under blog_users membership only can post the topics in blog.
@auth.requires_membership('blog_users')
Users are if login then only they can able to view the page.
@auth.requires_login()

# -*- coding: utf-8 -*-
# try something like
def index(): return dict(message="hello from blog.py")

@auth.requires_membership('blog_users')
def post():
    form =SQLFORM(db.blog).process()
    return locals()
@auth.requires_login()
def view():
    rows = db(db.blog).select(orderby=~db.blog.id)
    return locals()

def display_form():
   form = SQLFORM(db.blog)
   if form.process().accepted:
       response.flash = 'form accepted'
       redirect(URL('thanks'))
   elif form.errors:
       response.flash = 'form has errors'
   else:
       response.flash = 'please fill out the form'
   return locals()
def update():
    record = db.blog(request.args(0)) or redirect (URL(post))
    form = SQLFORM(db.blog,record)
    if form.process().accepted:
        response.flash = T('Record Updated')
    else:
        response.flash=T('Please complete the form')
    return locals()

After updating the blog.py with requires_membership and requires_login for post and view respectively, then

If we try to http://127.0.0.1:8000/sivaweb2py/blog/view or
http://127.0.0.1:8000/sivaweb2py/blog/post
Then it will redirected us to login page




Once we enter correct details after successful login , then it will redirected to blog post page or view page





Now I am going to create one more user, who does not have any privileges to post.




New user is not part of post_users group, If we try to click the below link
http://127.0.0.1:8000/sivaweb2py/blog/post

So it will redirect us to not authorized page.




This is how access control can be given to users and user groups using web2py.

Thanks for viewing this page....