What Is is About?
This guide is a continuation of this guide . Its goal is to describe how to set up a comfortable development environment for Hadoop in a Linux virtual machine with HDP 2.2.
Adding Users to Sudoers
chmod 640 /etc/sudoers
gedit /etc/sudoers &
and find the line
root ALL=(ALL) ALL
Add there one more line:
Ihor_Bobak ALL=(ALL) ALL
This will allow the user to launch commands with administrator privileges.
You will need VNC Server in the case if you’re installing everything on a remote machine, and you will have to perform development on it. In the case if this is a local VM, you don’t need it.
Run the following commands:
yum install tigervnc-server
sudo chkconfig vncserver on
gedit /etc/sysconfig/vncservers &
add the following lines to the end:
And save the file.
su – Ihor_Bobak
Then run the server
service vncserver start
netstat -antl | grep 590
It must give you the following output:
tcp 0 0 0.0.0.0:5901 0.0.0.0:* LISTEN
Now install UltraVNC on your client machine and connect to yourhostname:5901. You should see a GUI interface in the VNC window.
Java was installed by Ambari Server (JDK 1.7.0_67). We need to make so that everyone sees the JAVA_HOME directory, and that the java’s bin directory is on the path.
Put a file jaaRun commands:
and fill in the following lines into the shell file that you opened:
Save the file.
But this is not all. Run
it will produce you in the output that this is OpenJDK, but I need it to be Oracle JDK just because of one reason: OpenJDK is buggy.
– it will point you to the path from where the java is running. In my case this is a /usr/bin/java
ls /usr/bin/java –l
in my case it points that this link goes to
lrwxrwxrwx. 1 root root 22 May 15 04:38 /usr/bin/java -> /etc/alternatives/java
Now let us do
ls /etc/alternatives/java -l
It will output a link to
lrwxrwxrwx. 1 root root 46 May 15 04:38 /etc/alternatives/java -> /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
So, here it how to fix it: we need to simply create a new link to the necessary JRE. I would prefer putting this link
ln -s /usr/jdk64/jdk1.7.0_67/bin/java /etc/alternatives/java
And here you are:
[root@epmbigdata devtools]# java -version
java version “1.7.0_67″
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
– this is what we needed exactly.
Go to the page https://maven.apache.org/download.cgi and download apache-maven-3.3.3-bin.tar.gz (the filename may differ dependently on the current version)
tar -xvf apache-maven-3.3.3-bin.tar.gz
Do the same thing as with java – put a file maven.sh into a directory /etc/prodile.d with this contents:
REMOVE THE FILE maven.sh~ which gedit creates as a backup !!!
Go to https://www.jetbrains.com/idea/download/ and get the latest version of community IntelliJ IDEA.
Put the file to /devtools and run the command
tar -xvf ideaIC-14.1.3.tar.gz
and delete the file ideaIC-14.1.3.tar.gz
If you have BGR display, open this file in editor:
gedit /devtools/idea-IC-141.1010.3/bin/idea64.vmoptions &
and fix the last line to be
Download this file
and unpack to /devtools.
Edit squirrel-sql.sh – do there two changes in these lines:
if $macosx ; then
In the squirrel add the following driver:
The list of jar files to add in the “Extra Class Path” is next: add everything that is in the directories:
Then add a new alias:
Now you can connect to the Hive server, run SQL queries and browse the objects:
Go to this website http://dev.mysql.com/downloads/repo/yum/ and download the rpm file for your platform (in my case I have CentOS 6.6, therefore I download “Red Hat Enterprise Linux 6″.
Install this rpm file.
As a result, you got two files in the /etc/yum.repos.d which begin from “mysql”. Your repository is set. Now run the command:
yum repolist enabled | grep “mysql.*-community.*”
There must be the following output:
mysql-connectors-community MySQL Connectors Community 14
mysql-tools-community MySQL Tools Community 23
mysql56-community MySQL 5.6 Community Server 146
Now run the following command:
yum install mysql-workbench-community
Run the command
service mysql start
it will tell you that MySQL service is started.
Go to the top menu Applications / Programming -> MySQL Workbench:
It must tell you that it successfully established connection to MySQL.
Run OK and connect to this MySQL instance. Run the following SQL scripts:
select * from VERSION;
yum install mc
yum install epel-release
FileZilla FTP Client
You may need it to have a better FTP client – to exchange information with the external host.
yum install filezilla
Configuration of the connection on the Linux machine:
Settings on the windows site are as follows. In the filezilla server interface:
Firewall is set up in my case, therefore:
Press “advanced settings” and add the following rule:
And in the filezilla server configuration condifure it in the following way:
Now check the connection in the FileZilla server:
FTP Server on Linux
Run the following commands:
sudo yum install vsftpd
gedit /etc/vsftpd/vsftpd.conf &
Edit the following line: anonymous_enable=NO
Save the file. Run command
sudo service vsftpd restart
chkconfig vsftpd on
And then you may connect to this Linux machine from any other using FTP or FTPS with your Linux login and password.
Download https://www.libreoffice.org/download/libreoffice-fresh/ LibreOffice from here. You will get a file
LibreOffice_4.4.3_Linux_x86-64_rpm.tar.gz (the version may differ – depending on the current version).
Run the following commands:
tar -xvf LibreOffice_4.4.3_Linux_x86-64_rpm.tar.gz
yum localinstall RPMS/*.rpm
In our case
yum localinstall AdbeRdr9.5.5-1_i486linux_enu.rpm
For more comfortable work in CentOS I suggest to install Ubuntu fonts. Download this file
and extract it. You will get a folder ubuntu-fonts. Copy this folder to /usr/share/fonts
and run the following commands: