Introducing Big Data and Hadoop

What is Big Data and where does Hadoop fit in

Two important Hadoop ecosystem components, namely, MapReduce and HDFS

In-depth Hadoop Distributed File System – Replications, Block Size, Secondary Name node

High Availability and in-depth YARN – resource manager and node manager

Learning the working mechanism of MapReduce

Understanding the mapping and reducing stages in MR

Various terminologies in MR like Input Format

Output Format, Partitioners, Combiners, Shuffle and Sort

Introducing Hadoop Hive, detailed architecture of Hive

Comparing Hive with Pig and RDBMS

Working with Hive Query Language

Creation of database, table

Group by and other clauses

Various types of Hive tables, HCatalog, storing the Hive Results, Hive partitioning and Buckets

Indexing in Hive, the Map Side Join in Hive, working with complex data types, the Hive User-defined Functions

Introduction to Impala

Comparing Hive with Impala

The detailed architecture of Impala

Apache Pig introduction

Its various features, various data types and schema in Hive

The available functions in Pig, Hive Bags, Tuples and Fields

Apache Sqoop introduction

Overview

Importing and exporting data

performance improvement with Sqoop, Sqoop limitations

Introduction to Flume and understanding the architecture of Flume and what is HBase and the CAP theorem

Using Scala for writing Apache Spark applications

Detailed study of Scala

The need for Scala, the concept of object-oriented programming, executing the Scala code, various classes in Scala like Getters, Setters, Constructors, Abstract, Extending Objects, Overriding Methods, the Java and Scala interoperability

The concept of functional programming and anonymous functions

Bobsrockets package and comparing the mutable and immutable collections

Scala REPL, Lazy Values

Control Structures in Scala

Directed Acyclic Graph (DAG)

First Spark application using SBT/Eclipse

Spark Web UI

Spark in Hadoop ecosystem.

Detailed Apache Spark, its various features

Comparing with Hadoop

Various Spark components

Combining HDFS with Spark

Scalding

Introduction to Scala and importance of Scala and RDD

Understanding the Spark RDD operations

Comparison of Spark with MapReduce

What is a Spark transformation

Loading data in Spark

Types of RDD operations viz. transformation and action and what is a Key/Value pair

The detailed Spark SQL

The significance of SQL in Spark for working with structured data processing

Spark SQL JSON support

Working with XML data and parquet files

Creating Hive Context

Writing Data Frame to Hive

How to read a JDBC file, significance of a Spark Data Frame

How to create a Data Frame

What is schema manual inferring

How to work with CSV files, JDBC table reading

Data conversion from Data Frame to JDBC

Spark SQL user-defined functions

Shared variable and accumulators

How to query and transform data in Data Frames

How Data Frame provides the benefits of both Spark RDD and Spark SQL and deploying Hive on Spark as the execution engine

Introduction to Spark MLlib

Understanding various algorithms

What is Spark iterative algorithm

Spark graph processing analysis, introducing Machine Learning

K-Means clustering

Spark variables like shared and broadcast variables

What are accumulators, various ML algorithms supported by MLlib

Linear Regression, Logistic Regression, Decision Tree, Random Forest

K-means clustering techniques, building a Recommendation Engine

Why Kafka, what is Kafka, Kafka architecture, Kafka workflow, configuring Kafka cluster, basic operations, Kafka monitoring tools

Integrating Apache Flume and Apache Kafka

Introduction to Spark streaming

The architecture of Spark streaming

Working with the Spark streaming program

Processing data using Spark streaming

Requesting count and DStream

Multi-batch and sliding window operations and working with advanced data sources

Introduction to Spark Streaming, features of Spark Streaming, Spark Streaming workflow,

Initializing StreamingContext, Discretized Streams (DStreams), Input DStreams and Receivers, transformations on DStreams, Output Operations on DStreams

Windowed Operators and why it is useful, important Windowed Operators, Stateful Operators.

Create a 4-node Hadoop cluster setup

Running the MapReduce Jobs on the Hadoop cluster

Successfully running the MapReduce code and working with the Cloudera Manager setup

The overview of Hadoop configuration

The importance of Hadoop configuration file

The various parameters and values of configuration

The HDFS parameters and MapReduce parameters

Setting up the Hadoop environment

The Include and Exclude configuration files

The administration and maintenance of name node

Data node directory structures and files

What is a File system image and understanding Edit log?

Introduction to the checkpoint procedure

name node failure and how to ensure the recovery procedure, Safe Mode, Metadata and Data Backup, various potential problems and solutions

What to look for and how to add and remove nodes

How ETL tools work in the Big Data industry

Introduction to ETL and data warehousing

Working with prominent use cases of Big Data in the ETL industry and end-to-end ETL PoC showing Big Data integration with the ETL tool