Omal Perera

How Data Pipelines Supports IoT Data Pre-Processing

Tue, 03 Jul 2018 00:00:00 +0000

Hello all! let’s have a productive discussion on How Data Pipelines Supports IoT Data Pre-Processing.

Today The Internet of Things (IoT) has the capability to change our world. And while we are starting to see its incredible impact, we are still very much at the beginning of the transformational journey. Soon every device you own – and nearly every element you can imagine – will be linked to the Internet. Whether it’s through your phone, wearable tech or everyday household objects, the Internet of Things (IoT) will connect us in ways we can’t even imagine yet.

There were more devices connected to the Internet in 2008, than there were people.
In 2015, there were 4.9 billion (4,9000,000,000+) connected Internet of Things. Smartphone users topped at over 6.1 billion (6,100,000,000+).
Predictions for the number of connected IoT by 2020, meets or goes over 50 billion (50,0000,000,000+). A quarter of a billion vehicles are expected to be connected to the Internet.

IoT plays a big role in each and every domain which essentialy needs computation on data records sent by them. But there are number of implementation challenges such as

Hardware Compatibility Issues
Incorrect Data Capture difficulties
Analysis Challenges
Data Security Issues and many more…

In order to cope with these challanges we need a different approach.

Here we are discussing a way to overcome the heterogenity of IoT devices when comes to the big data processing.

WHITE PAPER on How Data Pipelines Supports IoT Data Pre-Processing

Download White Paper from here

Image Source : http://datascience.esy.es/tag/mapa/

kerberos Simplified

Thu, 01 Feb 2018 00:00:00 +0000

Hello all of you! lets discuss what is kerberos in a simple manner. This article is not going into a deep level of kerberos technology. But you all can get a highlevel overview, with regards to its History, how it works, why is it needed, advantages as well as disadvantages etc.

Introduction

Kerberos is a popular authentication protocol that can be used for single sign-on (SSO). The main key concept behind SSO is getting the services that a user is authorized, with a single login (one time activity). So, user don’t want to login on each of those services in many different times. Kerberos systems offers protection against many network attacks and vulnerabilities. Following are some characteristics of Kerberos.

uses tickets to authenticate
avoids saving passwords locally
avoids sending passwords over the internet
involves a trusted 3rd-party
built on symmetric-key cryptography

History

Initially, Massachusetts Institute of Technology (MIT) developed Kerberos to protect network services provided by Project Athena in 1980. The current version of the protocol is version 5. Previous versions of the protocol were based on the earlier Needham–Schroeder symmetric key protocol. Version 4 of the Kerberos protocol was the first version that was publicly released, and is now deprecated for security reasons. In 1987, MIT released the Kerberos software freely available, under a certain copyright permissions and in 2007, MIT formed the Kerberos Consortium to encourage development. In 1993, It became an IETF Standard which is the premier Internet standard.

The name Kerberos arrived from the three-headed dog that guarded the entrance to Hades in Greek mythology. It’s pretty matching for the Kerberos concept since it takes a third-party (a Key Distribution Center) to authenticate between a client and a service or host machine.

Authenticating using Kerberos

Why Kerberos

Kerberos has two basic purposes namely security and authentication. On non-distributed computer systems, a secret key or a password can be used to prove a user’s identity. But on a distributed computer network system, scenario is different than standalone computer systems. If same theory applied for the distributed systems, password must be transmitted over the network, from one machine to another machine. For an example if a user requests a private file from a server, that typical user needs to send his secret key to the server in order to authenticate him. Because this password is the one secret piece of information that identifies a user, anyone in that network knowing a user’s password can access their files as that user.

Therefore, it is necessary to prevent anyone from intercepting or eavesdropping on the transmitted password. In addition, it is necessary to provide a means of authenticating users any time a user requests a service, they must prove their identity.

In order to fill above mentioned gap in the security in distributed systems, Kerberos has:

Strong mutual authentication.
- Secrets are not transmitted across the network.
- Critical authentication data is encrypted.
- The client (user) is authenticated to the server and the server is authenticated to the client. The client identity is used to authorize services on the server.
Single sign-on.
- A user convenience meaning a single identity and password can be used for many services with only one login sequence.

So what is the different between Firewall & Kerberos

A firewall is a kind of network security system. Firewalls can be presents in either hardware or software based. Those are basically depending on rules that controls incoming and outgoing network traffic. A firewall acts as a barrier between a trusted network and untrusted network and controls access to the resources of a network through a positive control model. Firewalls assumes, that the attackers are coming from the outside. But in distributed systems, attacks are frequently from within the system.

In contrast, Kerberos assumes that network connections are the weak link in network security rather than servers and work stations.

Kerberos Components

Key Distribution Center (KDC)
- Each Kerberos realm will have at least one Kerberos server. This server, the Key Distribution Center, contains the Authentication Service, the Ticket-Granting Service, and the master database for Kerberos. These services are implemented as a single daemon.
Authentication Service
- The authentication service handles user authentication, or the process of verifying that principals are correctly identified.
- Consists of the security servers in the KDC (or KDCs), and security clients.
- A security client communicates with a security server to request information and operations. The security server accesses the registry database to perform queries and updates and to validate user logins.
Ticket-Granting Service
- Once authenticated, a principal will be granted a TGT and a ticket session key, which gives the principal the right to use the ticket. This combination of the ticket and its associated key is known as your credentials.
- A principal’s credentials are stored in a credentials cache, which is often just a file in the principal’s local directory tree.
The Kerberos Database
- The Kerberos database contains all of the realm’s Kerberos principals, their passwords, and other administrative information about each principal.
- Each KDC contains its own copy of the Kerberos database. The master KDC contains the primary copy of the database, which it propagates at regular intervals to the slave KDCs.
Kerberos Utility Programs
- OpenVMS provides three different versions of each of the Kerberos user interface programs, the original UNIX style, a DCL version, and an X Windows version.
Kerberos Registry
- The Kerberos registry can be manipulated in several ways. It is initially created via the KRB$CONFIGURE command procedure. Other tools used to access the Kerberos information are:
- kadmin - Used for reading or updating the Kerberos registry.
- kinit - Creates credentials for a user.
- klist - Displays the existing credentials for a user.
- kdestroy - Deletes a user’s credentials.
- kpasswd - Changes a user’s Kerberos password.
- kdb5_util - Dumps or loads the Kerberos database for save and restore operations.

Authenticating using Kerberos

When a client logs in to the realm, an authentication request is sent to the Kerberos Key Distribution Center (KDC).
A Ticket-Granting Ticket (TGT) is returned as the result of authentication.
When the client application starts, the TGT is used to request an application ticket.
The application ticket is then sent to the application server, which verifies the application ticket with the KDC.
Service ticket for the required Service is sent to the server.
Server sends the requested data to the client

Advantages of Kerberos

There are many advantages of Kerberos than traditional securing methodologies.

Scalable
- Servers do not need to contact KDC to authenticate users
- Only users and machine account authenticate with the KDC, once per 10h of activity
Secure
- Passwords are not transmitted over the wire
- Ticket based authentication based on certificates trusts
- Short-term session keys (Long-term secrets used only to derive short-term keys)
- Separate session key for each user-server pair
Advanced Features
- Single Sign-On
- Delegation
- Cross Domain Authentication
- Interoperability
- Mutual authentication

Kerberos Applications

Local authentication
- login and su in OpenBSD
Authentication for network protocols
- Remote login
- Remote Shell
Secure windowing systems
Microsoft Windows (used in Security Support Provider(SSP))
For Kerberizing applications such as Email, FTP, network file systems

Weaknesses / Limitations

Single point of failure
- It requires continuous availability of a central Kerberos server. In a case of Knock out the Kerberos server and no one can log in
Password guessing
- no authentication is required to request a ticket, so attacker can gather equivalent of /etc/passwd by requesting many tickets.
Kerberos is vulnerable if the local machine is compromised and malware captures the password.
Ticket hijacking – Malicious user may steal the service ticket of another user on the same workstation and try to use it
server authentication – Attacker may misconfigure the network so that he receives messages addressed to a legitimate server

Image Copyrights : www.freepik.com .

Sending POST requests from Node.js

Fri, 19 Jan 2018 00:00:00 +0000

Hello all of you! from this article we are going to explore how to send POST requests from a Node.js Application. Node application using here is very much simple & very easy to understand.

For the easy coding we are using requests npm package to send the requests. In requests npm package you can find many easy http method implementations

request.get(): Defaults to method: “GET”.
request.post(): Defaults to method: “POST”.
request.put(): Defaults to method: “PUT”.
request.patch(): Defaults to method: “PATCH”.
request.del() / request.delete(): Defaults to method: “DELETE”.
request.head(): Defaults to method: “HEAD”.
request.options(): Defaults to method: “OPTIONS”.

So lets directly go into the code

var http       = require('http');
var express    = require('express');
var request    = require('request');
var app        = express();

var server = http.createServer(app);
var port = 9000;

//Starting the server
server.listen(port);
console.log('Server Started on port ' + port);

//POST request
request.post(
  'http://localhost:8090/api/users',
  { json: { key1: 'value1', key2: 'value2' } },
  function (error, response, body) {
    if (!error && response.statusCode == 200) {
      console.log(body);
    }
  }
);

To run the code snippet

Step 1 - run npm init to initiate a command line questionnaire that will conclude with the creation of a package.json in the directory

$ npm init

Step 2 - You first need to install related npm packages using npm install <package-name> command

$ npm install body-parser
$ npm install express
$ npm install request

Step 3 Run the js file

$ node file-name.js

Playing with GIT Branches

Fri, 29 Dec 2017 00:00:00 +0000

Once you clone a git repository from github, you may need to figure out what other branches really exist on the remote repository so you can pull them down and checkout them, merge them into your local branches. It is not a hard task if you are using GitHub Desktop. But much better to know if you interacting with command line.

For the sake of better understanding, lets look into an example of cloning a remote repository into your local machine.

$ git clone https://github.com/OmalPerera/sample-project.git
$ cd sample-project

To check the local branches in your repository

$ git branch

* master

To list all existing Branches

$ git branch -av

* master                cc30c0f Initial commit
  remotes/origin/HEAD   -> origin/master
  remotes/origin/dev    cc30c0f Initial commit
  remotes/origin/master cc30c0f Initial commit

Create a local tracking branch in order to work on it

$ git checkout <dev>

  Branch dev set up to track remote branch dev from origin.
  Switched to a new branch 'dev'

Fetching all remote branches at once

$ git pull --all

Below I have explained the difference between git pull & git fetch

Switching between branches

$ git checkout <branch>

Creating a new tracking branch based on a remote branch

$ git checkout --track <remote/branch>

Deleting local branch

$ git branch -d <branch>

`git pull` vs `git fetch`

git pull

Git tries to automate your work for you. Since it is context sensitive, git will merge any pulled commits into the branch you are currently working in. pull automatically merges the commits without letting you review them first. You may run into frequent conflicts if you are mot managing your branches in a proper way.

git fetch

Git gathers any commits from the particular branch that do not exist in your current branch & saves them in your local repository. But it doesn’t merge them with your current branch. This is particularly useful if you need to keep your repository up to date, but are working on something that might break if you update your files.

Explaining `Downstream` & `UpStream`

In terms of source control, you’re downstream when you copy (clone, checkout, etc) from a repository. Information flowed downstream to you.

When you make changes, you usually want to send them back upstream so they make it into that repository so that everyone pulling from the same source is working with all the same changes. Source

Install Maven on Mac OSX

Thu, 21 Dec 2017 00:00:00 +0000

First of all lets check for Maven in your system, because some versions of OS X came with Apache Maven 3 built in installed.

$ mvn -version

if you are getting a response as follows.

Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T13:28:13+05:30)
Maven home: /usr/share/maven
Java version: 1.8.0_60, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.12.5", arch: "x86_64", family: "mac"

Congratulations!! you don’t want to install Maven, You already have it.

But if you are getting a message like this, you can go ahead with the tutorial

-bash: mvn: command not found

Installing Maven

Step I - Downloading Maven

Download maven from maven.apache.org

Step II - Extract the tar.gz

Navigate to the directory where tar.gz file downloaded & extract it to /usr/share/maven. You can use following terminal commands to exact directly in to /usr/share/maven.

Remember to change the maven version in the command!.

sudo tar -xvf apache-maven-3.5.2.tar.gz -C /usr/share/
cd /usr/share/
sudo mv apache-maven-3.5.2 maven

Step III - Update the HOME & PATH variables

export M2_HOME=/usr/share/maven
export PATH=$PATH:$M2_HOME/bin

Okay! now check the previous command again

$ mvn -version

if you are getting a response as follows.

Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T13:28:13+05:30)
Maven home: /usr/share/maven
Java version: 1.8.0_60, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.12.5", arch: "x86_64", family: "mac"

Congratulations! 🎉 you have correctly configured the Maven in your Mac!

Up & Run Apache Spark within seconds with all dependencies

Sat, 16 Dec 2017 00:00:00 +0000

install-spark is a fully automated Spark installation bash script for linux Dabian based Servers.

Script will install following key components needes to install Real-time data processing with few clicks.

Full Update & Upgrade
install up-to-date JRE
install up-to-date JDK
install scala 2.11.11
install Apache kafka 2.10 0.10.2.1
install git
install sbt
install Apache spark spark-2.2.0 pre-built for Apache hadoop 2.7 & later
all the environmental + all the PATH variables

Using the Script

1. Granting the executing permission to the script

Navigate to the directory where the script is.

$ chmod +x install-spark.sh

2. Run the script

$ source install-spark.sh

Customizing the Script

In the script you can define the versions of libraries that you need to install.

eg: spark-2.2.0-bin-hadoop2.7 , scala-2.11.11

Under the # Default Application versions define your prefered version.

# Default Application versions

SCALA_VERSION="scala-2.11.11"
SPARK_VERSION="spark-2.2.0-bin-hadoop2.7"
KAFKA_VERSION="kafka_2.10-0.10.2.1"

Apache Spark - Kafka Integration for Real-time Data Processing with Scala

Thu, 30 Nov 2017 00:00:00 +0000

Real-time processing! kind of a trending term that techie people talks & do things. So actually what are the components do we need to perform Real-time Processing. Apache Spark Streaming, Apache Kafka are key two components out of many that comes in to my mind.

Spark Streaming is built-in library in Apache Spark which is micro-batch oriented stream processing engine. There are other alternatives such as Flink, Storm etc.

As we discussed in above paragraph, Spark Streaming reads & process streams. So who provides these Streams to Spark ?. In that case, we use Apache kafka to accomplish this task.

But why? can’t we use Direct streams via TCP sockets?. But the point is parallelism . Kafka enables parallel streaming with a support named "partition" which is highly compatible to use with Spark’s "partition".

I think, now it is clear why are we using spark with kafka. So let’s look in to integrate these two components. Consider this as a starting point.

My Development Environment

Spark version - 2.2.0
Scala version - 2.11.11
SBT version - 0.13.16
Kafka version - 0.10.2.0
OS - Mac OS (Unix based)

As mentioned the Spark docs “Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.2.0 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).” so you better use any 2.11.x version of scala in order to avoid dependency problems.

Before continuing the project, probably you better double check your scala & sbt installations in your machine

$ scala -version
Scala code runner version 2.11.11 -- Copyright 2002-2017, LAMP/EPFL

$ sbt sbtVersion
[info] Updated file /root/.../project/build.properties: set sbt.version to 0.13.16

Developing the Spark app with Scala

Note : Dependencies for this project can be definitely change with the time. And also basically those dependencies depend on the scala version, Spark version, SBT version etc that you have installed in your system. So try to stick with above mentioned development environment or else follow up the error log & adjust the dependencies according to your development environment.

So let’s start the journey to Real-time data processing with kafka , spark , & scala !

Directory structure

first of all we have to arrange our project’s directories & files in to a specific order which supports for sbt. Folder structure should be as follows.

./src
./src/main
./src/main/scala
./src/main/scala/kafkar.scala
./build.sbt

build.sbt file

As we are doing our project with SBT, here is the sbt build file build.sbt , where we include all the dependencies needed for our project.

name := "kafkar"

version := "0.1"

scalaVersion := "2.11.11"

retrieveManaged := true

fork := true

libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % "2.2.0"

libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.1.0"

Integrating Spark Streaming and Apache Kafka

Here we are going to fetch data from kafka topic to our spark app. if you are absolute newbie to these area, I’m recommending you to google on what is kafka?, how it works, what are kafka topics? what spark does? . I’m leaving it to you as a homework.

So this will be the code

import org.apache.log4j.{Level, Logger}
import org.apache.spark.SparkConf
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}


object kafkar {

  def main(args: Array[String]) {

    Logger.getLogger("org").setLevel(Level.OFF)
    Logger.getLogger("akka").setLevel(Level.OFF)

    println("program started")

    val conf = new SparkConf().setMaster("local[4]").setAppName("kafkar")
    val ssc = new StreamingContext(conf, Seconds(2))

    // my kafka topic name is 'test'
    val kafkaStream = KafkaUtils.createStream(ssc, "localhost:2181","spark-streaming-consumer-group", Map("test" -> 5))

    kafkaStream.print()
    ssc.start
    ssc.awaitTermination()

  }
}

You would need a multi-core machine (>= 2 cores) for spark-streaming to work while running in standalone mode. But it will work fine if you run it in local mode with master as local[4].

there are several Master Urls passed to Spark

local - Run Spark locally with one worker thread (no parallelism at all).
local[K] - Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine).
local[* ] - Run Spark locally with as many worker threads as logical cores on your machine.
spark://HOST:PORT - Connect to the given Spark standalone cluster master.

are some of them

so we are now done with the spark app. prior to run our app, we have to make sure we have Up & running kafka topic with name of test. If not Spark app wont be able to consume the streams.

Configuring up Kafka broker

If you don’t have kafka installed in your environment, you can refer my post Setting Up Apache Kafka locally to setup it from the scratch. if you have already installed kafka, we have to create a topic named test & start kafka producer.

Starting the Zookeeper Server

As you are now in the kafka_2.10-0.10.2.0 directory (can be differ depends on your kafka version), execute the following command

  sudo bin/zookeeper-server-start.sh config/zookeeper.properties

Starting the Kafka Server

  sudo bin/kafka-server-start.sh config/server.properties

Creating topic

Creating a topic located at zookeeper at the localhost:2181 named test with a single partition & single replica. (do it in a separate terminal).

  sudo bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

To ensure that our topic is created, execute following command

  bin/kafka-topics.sh --list --zookeeper localhost:2181

Running the Producer

This is for feed the test Topic (do it in a separate console)

  bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

You can send messages to the Kafka cluster from the console even except the standard file inputs. just type the message in the console. These messages will be consumed by our spark app.

  This is a Sample message

Running the Consumer

This is for listen to the producer at that port (do it in a separate terminal).
Following command will listens for the topic inputs and outputs

    bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test

If you type messages into the producer terminal you should see them appear in the consumer terminal.

Now Kafka broker is ready to go. Now its time to run our Spark application.

Running Spark Application with sbt

We are all set for running the application. So open a terminal window & navigate to the project directory. Now we just want to compile the code with sbt compile & run it with sbt run on sbt console.

$ sbt compile

[info] Loading project definition from /root/xc/project
[info] Loading settings from build.sbt ...
[info] Set current project to kafkar (in build file:/root/xc/)
[info] Executing in batch mode. For better performance use sbt's shell
[success] Total time: 6 s, completed Dec 1, 2017 12:36:52 PM

$ sbt run

[info] Loading project definition from /root/xc/project
[info] Loading settings from build.sbt ...
[info] Set current project to kafkar (in build file:/root/xcity/)
[info] Running (fork) kafkar
[info] program started
[info] -------------------------------------------
[info] Time: 1512131872000 ms
[info] -------------------------------------------
[info] -------------------------------------------
[info] Time: 1512131873000 ms
[info] -------------------------------------------
[info] -------------------------------------------
[info] Time: 1512131874000 ms
[info] -------------------------------------------

We are done! Spark Streaming is now connected to Apache Kafka and consumes messages every 2 seconds. Leave it running and switch to kafka Producer terminal and enter some messages

omal@ubuntu-london:~/kafka_2.10-0.10.2.1# sudo bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

OK,41,14,690,41,2014-11-13T09:15:00,3,32502142,158355
OK,36,19,690,36,2014-11-13T09:20:00,1,32502581,158355
OK,36,11,690,36,2014-11-13T09:25:00,0,32503023,158355

Switch to the terminal with Spark application running and see the message printed out.

[info] -------------------------------------------
[info] Time: 1512137948000 ms
[info] -------------------------------------------
[info] -------------------------------------------
[info] Time: 1512137949000 ms
[info] -------------------------------------------
[info] (null,OK,41,14,690,41,2014-11-13T09:15:00,3,32502142,158355)
[info] (null,OK,36,19,690,36,2014-11-13T09:20:00,1,32502581,158355)
[info] (null,OK,36,11,690,36,2014-11-13T09:25:00,0,32503023,158355)
[info] -------------------------------------------
[info] Time: 1512137950000 ms
[info] -------------------------------------------
[info] -------------------------------------------
[info] Time: 1512137951000 ms
[info] -------------------------------------------

Additionally you can visit Apache Spark web UI (most of the time http://localhost:4040/) to get more details, histograms on your Spark job.

Congratulations! 🎉 you have completed the mission

Setting Up Apache Kafka locally

Fri, 10 Nov 2017 00:00:00 +0000

This Guide is for the linux based OS & directories in the Kafka setup can be slightly different in the windows setup. Mainly we deal with the /bin folder.

for mac - kafka_2.10-0.10.2.0/bin/
for Windows - kafka_2.10-0.10.2.0\bin\windows\

Note : Every command must be executed in a separate terminal or console except for creating the topic and ending a server.

Step 1 - Download the source code & extract it

Download Apache kafka from here
Navigate to the downloaded directory & untar the kafka_2.10-0.10.2.0.tar

  tar -xzf kafka_2.10-0.10.2.0.tgz
  cd kafka_2.10-0.10.2.0

Step 2 - Starting the Server

You first need to start a ZooKeeper serve, since Kafka uses Zookeeper for the tasks such as Electing a controller, Topic configuration, ACLs & for many more.
As you are now in the kafka_2.10-0.10.2.0 directory execute the following command

  bin/zookeeper-server-start.sh config/zookeeper.properties

Starting the Kafka Server

  bin/kafka-server-start.sh config/server.properties

Step 3 - Creating a Topic

Creating a topic located at zookeeper at the localhost:2181 named mytopic with a single partition & single replica.

  bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic mytopic

To ensure that our topic is created, execute following command

  bin/kafka-topics.sh --list --zookeeper localhost:2181

Step 4 - Running the Producer

This is for feed the mytopic Topic (do it in a separate console)

  bin/kafka-console-producer.sh --broker-list localhost:9092 --topic mytopic

You can send messages to the Kafka cluster from the console even except the standard file inputs. just type the message in the console.

  This is a Sample message

Step 5 - Running the Consumer

This is for listen to the producer at that port (do it in a separate terminal).
Following command will listens for the topic inputs and outputs

    bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic mytopic

If you type messages into the producer terminal you should see them appear in the consumer terminal.

Step 6 - Stopping the Kafka Server

bin/kafka-server-stop.sh

Step 7 - Stopping the Zookeeper Server

bin/zookeeper-server-stop.sh

Omal Perera

How Data Pipelines Supports IoT Data Pre-Processing

WHITE PAPER on How Data Pipelines Supports IoT Data Pre-Processing

kerberos Simplified

Introduction

History

Authenticating using Kerberos

Why Kerberos

So what is the different between Firewall & Kerberos

Kerberos Components

Authenticating using Kerberos

Advantages of Kerberos

Kerberos Applications

Weaknesses / Limitations

Sending POST requests from Node.js

So lets directly go into the code

To run the code snippet

Playing with GIT Branches

To check the local branches in your repository

To list all existing Branches

Create a local tracking branch in order to work on it

Fetching all remote branches at once

Switching between branches

Creating a new tracking branch based on a remote branch

Deleting local branch

git pull vs git fetch

git pull

git fetch

Explaining Downstream & UpStream

Install Maven on Mac OSX

Installing Maven

Step I - Downloading Maven

Step II - Extract the tar.gz

Step III - Update the HOME & PATH variables

Up & Run Apache Spark within seconds with all dependencies

Using the Script

1. Granting the executing permission to the script

2. Run the script

Customizing the Script

Apache Spark - Kafka Integration for Real-time Data Processing with Scala

My Development Environment

Developing the Spark app with Scala

Directory structure

build.sbt file

Integrating Spark Streaming and Apache Kafka

Configuring up Kafka broker

Starting the Zookeeper Server

Creating topic

Running the Producer

Running the Consumer

Running Spark Application with sbt

Congratulations! 🎉 you have completed the mission

Setting Up Apache Kafka locally

Step 1 - Download the source code & extract it

Step 2 - Starting the Server

Step 3 - Creating a Topic

Step 4 - Running the Producer

Step 5 - Running the Consumer

Step 6 - Stopping the Kafka Server

Step 7 - Stopping the Zookeeper Server

`git pull` vs `git fetch`

Explaining `Downstream` & `UpStream`