Pivotal HD Single-Node VM

source: http://pivotalhd.cfapps.io/getting-started/pivotalhd-vm.html

Pivotal HD Single-Node VM

System requirements

• 8 GB RAM (The VM consumes 4 GB)
• At least 30 GB free disk space
• At least a dual-core CPU
• NAT networking configured in the VM
• 7z Archive utility

Pivotal HD VM contents

Pivotal HD VM is shipped with the following Pivotal products:

• Pivotal HD 2.0 - Hadoop 2.0.5, Zookeeper, HBase, Hive, Pig, Mahout, Madlib
• Pivotal HAWQ 1.1.4
• Pivotal Extension Framework (PXF) 1.1.3
• Pivotal GemFire XD 0.5 Beta
• Pivotal Command Center 2.2
• Product usage documentation

Other installed packages:

• JDK 7
• Spring Tool Suite (Eclipse-based IDE)
• Maven
• Git
• Retail Demo data for tutorials
• Sample Code

Credentials

The Pivotal HD Single-Node VM uses the following credentials:

Credential	Username	Password
Hadoop	gpadmin	password
VM Root user	root	password
DataLoader Web UI	gpadmin	password
Command Center Login	gpadmin	Gpadmin1
sudo	gpadmin user has partial sudo privileges	password

Notes: sudo is configured so that the gpadmin (default user for this VM) account can run commands via sudo without providing a password to access any of the other system accounts (mapred, hdfs, hadoop, etc.).

Scripts

You can use the following scripts to manage your Pivotal HD Single-Node VM:

• start_all.sh – Starts all Hadoop services.
• stop_all.sh – Stops all Hadoop services
• start_gfxd.sh – Starts GemFire XD (Note that this script also stops some Hadoop services that are not needed for the GemFire Tutorials)
• stop_gfxd.sh – Stops GemFire XD.

Installing the Pivotal HD Single-Node VM

1. Download the Pivotal HD Single-Node VM.
   Note: This tutorial is directed towards users of the Pivotal HD Single Node VM version 2.0 and who received an early-access version of the VM. The instructions in this document will not work correctly with other versions. Version 2.0 of the VM will be available for download when PHD 2.0 is released.
2. Enter the following command to extract the files using 7z:
   7za x ~/VMWare/PIVHDSNE_VMWARE_VM-2.0.0-40-2.7zs
   If you do not have a tool that can expand a 7z archive, you can download the 7z utility from: http://www.7-zip.org/
3. Start the VM by double-clicking the .vmx file within the newly-created PIVHDSNE_VMWARE_VM-2.0.0-40 directory.
4. After the VM starts, log in to the gpadmin user account using the password password (Note: this is also the root password.)
5. In the VM, launch the Firefox Web browser to see the Command Center Web UI at http://localhost:5443/.
6. Login to the Command Center with gpadmin as the username and Gpadmin1 as the password.
7. Start Pivotal HD and HAWQ using the ~/Desktop/start_all.sh script
   Note: You can stop Pivotal HD and HAWQ using the ~/Desktop/stop_all.sh script

How to use this VM:

1. Start the Hadoop services using the start_all.sh script located on the desktop.
2. Follow the tutorials in the Setting up the Pivotal HD Tutorial section of this site.

Setting Up the Pivotal HD Tutorial

The Getting Started with Pivotal HD Tutorial tutorial provides a set of examples that demonstrate loading, querying, and running MapReduce Applications on a Pivotal HD 2.0 distribution. The tutorial is organized into several groups of examples, each demonstrating specific tasks.

Prerequisites

To run this tutorial, you use the Pivotal Single-Node Virtual Machine. This VM contains the Pivotal HD software, sample data, scripts, documentation, Java, Ant, Maven, and the Spring Tool Suite (STS) IDE. See Pivotal HD Single-Node VM.

Loading Sample Data

All of the examples use a common data set. The raw data is already included on the file system of the Single-Node VM. You load this data to run the examples in this tutorial.

• See Example Data Set: Retail Demo.

Source code for the tutorials

The source code for the tutorials is available at https://github.com/gopivotal/pivotal-samples.git. This code is also included on the filesystem of the Pivotal HD Single-Node VM.

Running the Tutorial

The following examples demonstrate Pivotal HD features:

Map Reduce Examples

• Top Ten Postal Codes by Revenue
• First and Last Order ID/date of every customer

HAWQ Examples

• HAWQ Internal Tables
• PXF External Tables - HDFS
• PXF External Tables - Hbase
• PXF External Tables Predicate-push-down - Hbase
• PXF External Tables - Statistics

Hive

• Hive Tables

Pig

• Top Ten Postal Codes by Revenue
• First and Last Order Date / Order Id for each Customer

Spring JDBC with PivotalHD and Hawq

source: http://oracamp.com/spring-jdbc-pivotalhd-and-hawq

Spring JDBC with PivotalHD and Hawq

HAWQ enables SQL for Hadoop ensuring we can use something like Spring JDBC as shown below. In this example we use the PivotalHD VM with data from a HAWQ append only table as shown below.

view plainprint?

1.     
2. gpadmin=# \dt  
3.                              List of relations  
4.    Schema    |            Name             | Type  |  Owner  |   Storage     
5. -------------+-----------------------------+-------+---------+-------------  
6.  retail_demo | categories_dim_hawq         | table | gpadmin | append only  
7.  retail_demo | customer_addresses_dim_hawq | table | gpadmin | append only  
8.  retail_demo | customers_dim_hawq          | table | gpadmin | append only  
9.  retail_demo | date_dim_hawq               | table | gpadmin | append only  
10.  retail_demo | email_addresses_dim_hawq    | table | gpadmin | append only  
11.  retail_demo | order_lineitems_hawq        | table | gpadmin | append only  
12.  retail_demo | orders_hawq                 | table | gpadmin | append only  
13.  retail_demo | payment_methods_hawq        | table | gpadmin | append only  
14.  retail_demo | products_dim_hawq           | table | gpadmin | append only  
15. (9 rows)  
16.   
17. gpadmin=# select * from customers_dim_hawq limit 5;  
18.  customer_id | first_name | last_name | gender   
19. -------------+------------+-----------+--------  
20.  11371       | Delphine   | Williams  | F  
21.  5480        | Everett    | Johnson   | M  
22.  26030       | Dominique  | Davis     | M  
23.  41922       | Brice      | Martinez  | M  
24.  47265       | Iva        | Wilson    | F  
25. (5 rows)  
26.   
27. Time: 57.334 ms   

Code

Customer.java (POJO)

view plainprint?

1.     
2. package pivotal.au.hawq.beans;  
3.   
4. public class Customer {  
5.   
6.  public String customerId;  
7.  public String firstName;  
8.  public String lastName;  
9.  public String gender;  
10.    
11.  public Customer()   
12.  {  
13.  }  
14.   
15.  public Customer(String customerId, String firstName, String lastName,  
16.    String gender) {  
17.   super();  
18.   this.customerId = customerId;  
19.   this.firstName = firstName;  
20.   this.lastName = lastName;  
21.   this.gender = gender;  
22.  }  
23.   
24. ..... getters/setters etc ....    

DAO : Constants.java

view plainprint?

1.     
2. package pivotal.au.hawq.dao;  
3.   
4. public interface Constants   
5. {  
6.    public static final String SELECT_CUSTOMER = "select * from retail_demo.customers_dim_hawq where customer_id = ?";  
7.      
8.    public static final String SELECT_FIRST_FIVE_CUSTOMERS = "select * from retail_demo.customers_dim_hawq limit 5";  
9.      
10. }    

DAO : CustomerDAO.java

view plainprint?

1.     
2. package pivotal.au.hawq.dao;  
3.   
4. import java.util.List;  
5.   
6. import pivotal.au.hawq.beans.Customer;  
7.   
8. public interface CustomerDAO   
9. {  
10.    public Customer selectCustomer (String customerId);  
11.      
12.    public List firstFiveCustomers();  
13.      
14. }    

DAO : CustomerDAOImpl.java

view plainprint?

1.     
2. package pivotal.au.hawq.dao;  
3.   
4. import java.util.ArrayList;  
5. import java.util.List;  
6. import javax.sql.DataSource;  
7.   
8. import org.springframework.jdbc.core.BeanPropertyRowMapper;  
9. import org.springframework.jdbc.core.JdbcTemplate;  
10. import pivotal.au.hawq.beans.Customer;  
11.   
12. public class CustomerDAOImpl implements CustomerDAO   
13. {  
14.  private JdbcTemplate jdbcTemplate;  
15.    
16.  public void setDataSource(DataSource dataSource)  
17.  {  
18.      this.jdbcTemplate = new JdbcTemplate(dataSource);  
19.  }  
20.   
21.  public Customer selectCustomer(String customerId)   
22.  {  
23.      return (Customer) jdbcTemplate.queryForObject  
24.             (Constants.SELECT_CUSTOMER,   
25.               new Object[] { customerId },   
26.               new BeanPropertyRowMapper( Customer.class));  
27.  }  
28.    
29.  public List firstFiveCustomers()   
30.  {  
31.   List customers = new ArrayList();  
32.     
33.   customers = jdbcTemplate.query(Constants.SELECT_FIRST_FIVE_CUSTOMERS,   
34.                            new BeanPropertyRowMapper( Customer.class));   
35.     
36.   return customers;  
37.     
38.  }  
39.   
40. }    

application-context.xml

view plainprint?

1.     
2. xml version="1.0" encoding="UTF-8"?>  
3. <beans xmlns="http://www.springframework.org/schema/beans"  
4.  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
5.  xmlns:jdbc="http://www.springframework.org/schema/jdbc"  
6.  xmlns:context="http://www.springframework.org/schema/context"  
7.  xsi:schemaLocation="http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc-3.2.xsd  
8.   http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd  
9.   http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.2.xsd">  
10.   
11.  <context:property-placeholder location="classpath:/jdbc.properties"/>  
12.    
13.  <bean id="pivotalHDDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">  
14.   <property name="driverClassName" value="${jdbc.driverClassName}" />  
15.   <property name="url" value="${jdbc.url}" />  
16.   <property name="username" value="${jdbc.username}" />  
17.   <property name="password" value="${jdbc.password}" />  
18.  </bean>  
19.    
20.  <bean id="customerDAOImpl" class="pivotal.au.hawq.dao.CustomerDAOImpl">  
21.       <property name="dataSource" ref="pivotalHDDataSource" />  
22.    </bean>  
23. </beans>    

jdbc.properties

jdbc.driverClassName=org.postgresql.Driver
jdbc.url=jdbc:postgresql://172.16.62.142:5432/gpadmin
jdbc.username=gpadmin
jdbc.password=gpadmin

TestCustomerDAO.java

view plainprint?

1.     
2. package pivotal.au.hawq.dao.test;  
3.   
4. import java.util.List;  
5. import java.util.logging.Level;  
6. import java.util.logging.Logger;  
7.   
8. import org.springframework.context.ApplicationContext;  
9. import org.springframework.context.support.ClassPathXmlApplicationContext;  
10.   
11. import pivotal.au.hawq.beans.Customer;  
12. import pivotal.au.hawq.dao.CustomerDAO;  
13.   
14. public class TestCustomerDAO   
15. {  
16.  private Logger logger = Logger.getLogger(this.getClass().getSimpleName());  
17.  private ApplicationContext context;  
18.  private static final String BEAN_NAME = "customerDAOImpl";  
19.  private CustomerDAO customerDAO;  
20.    
21.  public TestCustomerDAO()   
22.  {  
23.      context = new ClassPathXmlApplicationContext("application-context.xml");  
24.      customerDAO = (CustomerDAO) context.getBean(BEAN_NAME);    
25.      logger.log (Level.INFO, "Obtained customerDAOImpl BEAN...");  
26.  }  
27.   
28.  public void run()  
29.  {  
30.   System.out.println("Select single customer from HAWQ -> ");  
31.   Customer customer = customerDAO.selectCustomer("59047");  
32.   System.out.println(customer.toString());  
33.     
34.   System.out.println("Select five customers from HAWQ -> ");  
35.     
36.   List customers = customerDAO.firstFiveCustomers();  
37.     
38.   for (Customer cust: customers)  
39.   {  
40.    System.out.println(cust.toString());  
41.   }  
42.     
43.  }  
44.    
45.  public static void main(String[] args)   
46.  {  
47.   // TODO Auto-generated method stub  
48.   TestCustomerDAO test = new TestCustomerDAO();  
49.   test.run();  
50.  }  
51.   
52. }    

Output

log4j:WARN No appenders could be found for logger (org.springframework.core.env.StandardEnvironment).

log4j:WARN Please initialize the log4j system properly.

Sep 16, 2013 9:59:09 PM pivotal.au.hawq.dao.test.TestCustomerDAO

INFO: Obtained customerDAOImpl BEAN...

Select single customer from HAWQ -> 

Customer [customerId=59047, firstName=Olivia, lastName=Anderson, gender=F]

Select five customers from HAWQ -> 

Customer [customerId=11371, firstName=Delphine, lastName=Williams, gender=F]

Customer [customerId=5480, firstName=Everett, lastName=Johnson, gender=M]

Customer [customerId=26030, firstName=Dominique, lastName=Davis, gender=M]

Customer [customerId=41922, firstName=Brice, lastName=Martinez, gender=M]

Customer [customerId=47265, firstName=Iva, lastName=Wilson, gender=F]

More Information

Here is the high level page describing the Pivotal HD & HAWQ technology.
http://blog.gopivotal.com/products/pivotal-hd-ga

This page dives deeper into the PHD VM with a walkthrough from data loading, map reduce and SQL queries.
http://pivotalhd.cfapps.io/getting-started/pivotalhd-vm.html

Finally, the following link is the direct download location of the VM discussed above above.
http://bitcast-a.v1.o1.sjc1.bitgravity.com/greenplum/pivotal-sw/pivotalhd_singlenodevm_101_v1.7z