JULY SOFT .NET BLOG

July Soft Geysir Enterprise Search NEW Features:

1. Hadoop Indexer

Geysir Enterprise Search support now Hadoop Indexer capable of scanning any Hadoop root folder using WebHDFS Api. Hadoop indexer requires that WebHDF to be enabled. For more information how to install Hadoop in a cluster configuration using Ubuntu machines see this Hadoop 1 Master & 2 Slaves Setup article.

2. Enable OCR(Optical Character Recognition)

This setting is at extractor level. When is True and is allowed by the licence option this will allow Geysir to index text from Image or from PDF's files images using OCR. This setting is available only for Image and PDF extractors.
More about this OCR feature you can find at Geysir Enterprise Search – CRM Import Service from VC & Emails Free* article.

3. CRM module – import from visit cards and emails

This setting is for DMS / CRM module. This is a service to import your clients / contacts from 2 main sources: from your emails and/or from your visit cards.
More about this CRM import feature you can find at Geysir Enterprise Search version supports now OCR (Optical Character Recognition) for PDF and IMAGE Formats article.

4. Extranet module

Exranet module is web platform for sharing files, updates and allows your company to have a more structured, traceable communication with your partners / clients. It also allows you to have real time chat communication to and from your clients and also supports OCR and structured data extraction from images and non-OCR PDFs in order to gain more efficiency in your data input activities.
It is integrated with Geysir Enterprise Search as follows: you may index within Geysir by content all shared files to and from your customers
Extranet advantages: eliminate communication errors, fast response time, flexibility and control.
The advantages of an Extranet compared to email are: superior, scalable, no problems of spam, traceability, backup, more professional stance towards your customers, can be customized, can add charts, reports, search easy, can be integrated with other applications.
More about this module we can show you through a demo on your request.

For a Better Search,

July Soft Team - www.julysoft.net

Responsibility. Integrity. Passion.

Why Hadoop is important in handling Big Data?

Hadoop provides excellent big data management provision, supports the processing of large data sets in a distributed computing environment. It is designed to expand from single servers to thousands of machines, each providing computation and storage. Its distributed file system facilitates rapid data transfer rates among nodes and allows the system to continue operating uninterrupted in case of a node failure, which minimizes the risk of catastrophic system failure, even if a significant number of nodes become out of action. Hadoop is very valuable for large scale businesses.

Hadoop installation scenario on 3 Ubuntu machines:

ub1 is server node and ub2 and ub3 are the slaves nodes.

Steps:

We will install Hadoop on master node ub1
Hadoop is based on java framework, so we will install java first:

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install default-jdk
sudo apt-get install oracle-java8-installer

Last command will install java at "/usr/lib/jvm/java-8-oracle". In order to check if the installation was Ok use next command:
Create a hadoop group and "hduser" user as system user:
Install SSH for secure accessing one machine from another(used by Hadoop for acceing slaves nodes):
Configure SSH. Login with hduser:
Generate SSH key for hduser:
Copy id_rsa.pub to authorized keys from hduser:
Add "hduser" to sudoers:
Hadoop doesn’t work on IPv6, so Ipv6 must be disabled:
- Add into above file below settings:

# disable ipv6

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

net.ipv6.conf.lo.disable_ipv6 = 1

CRTL+X -> yes

Locate hadoop installation parent directory:
Download Hadoop:
Extract Hadoop sources:
Move hadoop-2.7.3 to hadoop folder:
Assign ownership of this folder to Hadoop user hduser:
Create Hadoop temp dirs for namenode and datanode:
Assign ownership of this Hadoop temp folder to Hadoop user:
Check JAVA_HOME path:
Edit hadoop configuration files. Edit ".bashrc" file:

# -- HADOOP ENVIRONMENT VARIABLES START -- #

export JAVA_HOME=/usr/lib/jvm/java-8-oracle

export HADOOP_HOME=/usr/local/hadoop

export PATH=$PATH:$HADOOP_HOME/bin

export PATH=$PATH:$HADOOP_HOME/sbin

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOMEi

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

# -- HADOOP ENVIRONMENT VARIABLES END -- #

Edit "hadoop-env.sh":
Edit "core-site.xml":
Edit "hdfs-site.xml":
- cd /usr/local/hadoop/etc/hadoop
- sudoedit hdfs-site.xml
- add into above file:

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>file:/usr/local/hadoop_tmp/hdfs/namenode</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>file:/usr/local/hadoop_tmp/hdfs/datanode</value>

</property>

Edit "yarn-site.xml":
- cd /usr/local/hadoop/etc/hadoop
- sudoedit yarn-site.xml
- Add into above file:

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

Copy template of mapred-site.xml.template file:
Edit "mapred-site.xml":
- cd /usr/local/hadoop/etc/hadoop
- sudoedit mapred-site.xml
- Add into above file it:

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

Reset the PC & open again the terminal with hduser. Format namenode:
Start all hadpop daemons:
Verify hadoop daemons:

Test resource manager: Http://localhost:8088

Test namenode: Http://localhost:50070

Now we will extend setup hadoop on slaves nodes.
Add all host names to /etc/hosts directory in all Machines (Master and Slave nodes). You can find each PC IP using ifconfig command
- on UB1 / then on UB2 / then on UB3:
  - sudo vim /etc/hosts
  - if vim is not installed you will intall it using:
  - Add into above file:

10.0.3.15 UB1

10.0.3.16 UB2

10.0.3.17 UB3

Create hadoop as group and hduser as user in all slaves Pcs
Install rsync for sharing hadoop source on all PCs
Edit core-site.xml on master PC:
Edit hdfs-site.xml on master and replace replication factor from 1 to 3
Edit yarn-site.xml on master:

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>UB1:8025</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>UB2:8035</value>

</property>

<property>

<name>yarn.resourcemanager.address</name>

<value>UB3:8050</value>

</property>

Edit mapred-site.xml on master and add new entry:

<property>

<name>mapreduce.job.tracker</name>

<value>UB1:5431</value>

</property>

Edit on master node the master:
Update slaves on master:
- cd /usr/local/hadoop/etc/hadoop
- sudo vim slaves

## Add name of slave nodes

UB2

UB3

Use rsync on master:
- First install SSH on each slave PC
- sudo rsync -avxP /usr/local/hadoop/ hduser@UB1:/usr/local/hadoop/
- sudo rsync -avxP /usr/local/hadoop/ hduser@UB2:/usr/local/hadoop/
On master:
On each slave node:
Execute on master:
Execute on master:
Excute on each slave:
Test:

In order to configure WebHDFS, we need to hdfs-site.xml as follows:

        <property>
           <name>dfs.webhdfs.enabled</name>
           <value>true</value>
        </property>

Copy local folder to hadoop:

hadoop fs -copyFromLocal /home/user/DataFolder /

Http://UB1:50070 :

For a Better Search,

July Soft Team - www.julysoft.net

Responsibility. Integrity. Passion.

July Soft Geysir Enterprise Search NEW Features

July Soft Geysir Enterprise Search NEW Features:

1. Hadoop Indexer

2. Enable OCR(Optical Character Recognition)

3. CRM module – import from visit cards and emails

4. Extranet module

Hadoop 1 Master & 2 Slaves Setup

Why Hadoop is important in handling Big Data?

Hadoop installation scenario on 3 Ubuntu machines:

What #Logistics, Software Development have in common?

Planning is 10x more efficient than action without planning!

Wanting to have average usage of your Warehouse's gates of 90%?

3PL - DockScheduling Solution

You can't do Efficient Cross-Docking without a #DockScheduling Solution!

#JulySoftNet can implement #DockScheduling

#DockScheduling Solution may free up to 50% of time spent by Managers

How to eliminate time spent by Warehouse Managers with scheduling?

La Multi Ani 2022!

July Soft HEKLA at #EECONNECTED 2021