Installing HBase on Windows using Cygwin
Introduction
HBase is a distributed, column-oriented store, modeled after Google's BigTable. HBase is built on top of Hadoop for itsMapReduce and distributed file system implementation. All these projects are open-source and part of the Apache Software Foundation.
As being distributed, large scale platforms, the Hadoop and HBase projects mainly focus on *nix environments for production installations. However, being developed in Java, both projects are fully portable across platforms and, hence, also to the Windows operating system. For ease of development the projects rely on Cygwin to have a *nix-like environment on Windows to run the shell scripts.
Purpose
This document explains the intricacies of running HBase on Windows using Cygwin as an all-in-one single-node installation for testing and development. The HBase Overview and QuickStart guides on the other hand go a long way in explaning how to setup HBase in more complex deployment scenario's.
Installation
For running HBase on Windows, 3 technologies are required: Java, Cygwin and SSH. The following paragraphs detail the installation of each of the aforementioned technologies.
Java
HBase depends on the Java Platform, Standard Edition, 6 Release.
Cygwin
- Make sure you have Administrator privileges on the target system.
- Choose and create you Root and Local Package directories. A good suggestion is to use C:\cygwin\root andC:\cygwin\setup folders.
- Download the setup.exe utility and save it to the Local Package directory.
- Run the setup.exe utility,
- Choose the Install from Internet option,
- Choose your Root and Local Package folders
- and select an appropriate mirror.
- Don't select any additional packages yet, as we only want to install Cygwin for now.
- Wait for download and install
- Finish the installation
- Optionally, you can now also add a shortcut to your Start menu pointing to the setup.exe utility in the Local Package folder.
- Add CYGWIN_HOME system-wide environment variable that points to your Root directory.
- Add %CYGWIN_HOME%\bin to the end of your PATH environment variable.
- Reboot the sytem after making changes to the environment variables otherwise the OS will not be able to find the Cygwin utilities.
- Test your installation by running your freshly created shortcuts or the Cygwin.bat command in the Root folder. You should end up in a terminal window that is running a Bash shell. Test the shell by issuing following commands:
- cd / should take you to thr Root directory in Cygwin;
- the LS commands that should list all files and folders in the current directory.
- Use the exit command to end the terminal.
- When needed, to uninstall Cygwin you can simply delete the Root and Local Package directory, and the shortcutsthat were created during installation.
SSH
HBase (and Hadoop) rely on SSH for interprocess/-node communication and launching remote commands. SSH will be provisioned on the target system via Cygwin, which supports running Cygwin programs as Windows services!
- Rerun the setup.exe utility.
- OpenSSH
- tcp_wrappers
- diffutils
- zlib
- Wait for the install to complete and finish the installation.
HBase
Download the latest release of HBase from the website. As the HBase distributable is just a zipped archive, installation is as simple as unpacking the archive so it ends up in its final installation directory. Notice that HBase has to be installed in Cygwin and a good directory suggestion is to use /usr/local/ (or [Root directory]\usr\local in Windows slang). You should end up with a /usr/local/hbase-
Configuration
There are 3 parts left to configure: Java, SSH and HBase itself. Following paragraphs explain eacht topic in detail.
Java
One important thing to remember in shell scripting in general (i.e. *nix and Windows) is that managing, manipulating and assembling path names that contains spaces can be very hard, due to the need to escape and quote those characters and strings. So we try to stay away from spaces in path names. *nix environments can help us out here very easily by usingsymbolic links.
- Create a link in /usr/local to the Java home directory by using the following command and substituting the name of your chosen Java environment:
LN -s /cygdrive/c/Program\ Files/Java/
- Test your java installation by changing directories to your Java folder CD /usr/local/
and issueing the command ./bin/java -version. This should output your version of the chosen JRE.
SSH
Configuring SSH is quite elaborate, but primarily a question of launching it by default as a Windows service.
- On Windows Vista and above make sure you run the Cygwin shell with elevated privileges, by right-clicking on the shortcut an using Run as Administrator.
- First of all, we have to make sure the rights on some crucial files are correct. Use the commands underneath. You can verify all rights by using the LS -L command on the different files. Also, notice the auto-completion feature in the shell using
is extremely handy in these situations. - chmod +r /etc/passwd to make the passwords file readable for all
- chmod u+w /etc/passwd to make the passwords file writable for the owner
- chmod +r /etc/group to make the groups file readable for all
- chmod u+w /etc/group to make the groups file writable for the owner
- chmod 755 /var to make the var folder writable to owner and readable and executable to all
- Edit the /etc/hosts.allow file using your favorite editor (why not VI in the shell!) and make sure the following two lines are in there before the PARANOID line:
- ALL : localhost 127.0.0.1/32 : allow
- ALL : [::1]/128 : allow
- Next we have to configure SSH by using the script ssh-host-config
- If this script asks to overwrite an existing /etc/ssh_config, answer yes.
- If this script asks to overwrite an existing /etc/sshd_config, answer yes.
- If this script asks to use privilege separation, answer yes.
- If this script asks to install sshd as a service, answer yes. Make sure you started your shell as Adminstrator!
- If this script asks for the CYGWIN value, just
as the default is ntsec. - If this script asks to create the sshd account, answer yes.
- If this script asks to use a different user name as service account, answer no as the default will suffice.
- If this script asks to create the cyg_server account, answer yes. Enter a password for the account.
- Start the SSH service using net start sshd or cygrunsrv --start sshd. Notice that cygrunsrv is the utility that make the process run as a Windows service. Confirm that you see a message stating that the CYGWIN sshd service was started succesfully.
- Harmonize Windows and Cygwin user account by using the commands:
- mkpasswd -cl > /etc/passwd
- mkgroup --local > /etc/group
- Test the installation of SSH:
- Open a new Cygwin terminal
- Use the command whoami to verify your userID
- Issue an ssh localhost to connect to the system itself
- Answer yes when presented with the server's fingerprint
- Issue your password when prompted
- test a few commands in the remote session
- The exit command should take you back to your first shell in Cygwin
- Exit should terminate the Cygwin shell.
HBase
HBase config[installation directory]- HBase uses the ./conf/hbase-env.sh
- export JAVA_HOME=/usr/local/
- export HBASE_IDENT_STRING=$HOSTNAME as this most likely does not inlcude spaces.
- export JAVA_HOME=/usr/local/
- HBase uses the ./conf/hbase-default.xml file for configuration. S
- hbase.rootdir must read e.g. file:///C:/cygwin/root/tmp/hbase/data
- hbase.tmp.dir must read C:/cygwin/root/tmp/hbase/tmp
- hbase.zookeeper.quorum must read 127.0.0.1 because for some reason localhost doesn't seem to resolve properly on Cygwin.
- Make sure the configured hbase.rootdir and hbase.tmp.dir directories exist and have the proper rights set up e.g. by issuing a chmod 777 on them.
Testing
This should conclude the installation and configuration of HBase on Windows using Cygwin. So it's time to test it.
- Start a Cygwin terminal, if you haven't already.
- Change directory to HBase installation using CD /usr/local/hbase-
, preferably using auto-completion. - Start HBase using the command ./bin/start-hbase.sh
- When prompted to accept the SSH fingerprint, answer yes.
- When prompted, provide your password. Maybe multiple times.
- When the command completes, the HBase server should have started.
- However, to be absolutely certain, check the logs in the ./logs directory for any exceptions.
- Next we start the HBase shell using the command ./bin/hbase shell
- We run some simple test commands
- Create a simple table using command create 'test', 'data'
- Verify the table exists using the command list
- Insert data into the table using e.g.
put 'test', 'row1', 'data:1', 'value1' put 'test', 'row2', 'data:2', 'value2' put 'test', 'row3', 'data:3', 'value3'
- List all rows in the table using the command scan 'test' that should list all the rows previously inserted. Notice how 3 new columns where added without changing the schema!
- Finally we get rid of the table by issuing disable 'test' followed by drop 'test' and verified by listwhich should give an empty listing.
- Leave the shell by exit
- To stop the HBase server issue the ./bin/stop-hbase.sh command. And wait for it to complete!!! Killing the process might corrupt your data on disk.
- In case of problems,
- verify the HBase logs in the ./logs directory.
- Try to fix the problem
- Get help on the forums or IRC (#hbase@freenode.net). People are very active and keen to help out!
- Stopr, restart and retest the server.
Conclusion
Now your HBase server is running, start coding and build that next killer app on this particular, but scalable datastore!
