It is really One Microsoft experience to work closely with Azure Security Center and Azure Key Vault team to launch this service. Also personally, I grew up a lot after going through the Machine Learning algorithm improvement, infrastructure refactoring, BCDR and privacy policy compliance, cost reduce, monthly business review(MBR), customer feedback investigation.

It is indeed a challenging and inspiring work to wake me up every day.

What is Azure Defender for Key Vault

https://docs.microsoft.com/en-us/azure/security-center/defender-for-key-vault-introduction

Customers are using Azure Key Vault to store the most sensitive information in their Azure environment: keys, passwords, secrets and certificates for all of their Azure resources. By achieving this data, attackers may be able to perform lateral movement and breach other resources in the customers Azure environment.

Azure Defender for Key Vault is a cloud-native, breadth threat protection suite – gives customers additional layer of protection for the precious secretes stored in the Key Vault by helping the SOC team to detect suspicious activities in their Key Vaults and protect the entire Azure environment.

How to Enable Azure Defender for Key Vault

1. Enable it from Azure Key Vault

   In Key Vault’s Security page, click “try it for the first 30 days”

   

2. Enable it from Azure Security Center

   https://docs.microsoft.com/en-us/azure/security-center/security-center-pricing#enable-azure-defender

   1. From Security Center’s main menu, select Pricing & settings.
   2. Select the subscription that you want to upgrade.
   3. Select Azure Defender on to upgrade.
   4. Select Save.

   Below is the pricing page for an example subscription. You’ll notice that each plan in Azure Defender is priced separately and can be individually set to on or off. Make sure it is on for Azure Key Vault.

   

Azure Defender for Key Vault Alerts

https://review.docs.microsoft.com/en-us/azure/security-center/alerts-reference?branch=master#alerts-azurekv

Current Status

We just releasing to GA and we already have:

• 30G Azure Key Vault logs processed per month
• 1.2M Azure Key Vaults protected
• 63K Azure subscriptions protected

And expecting these numbers to raise dramatically in the current months.

General Availability Announcement at Ignite 2020

Azure Defender for Key Vault is generally available: https://docs.microsoft.com/en-us/azure/security-center/release-notes#azure-defender-for-key-vault-is-generally-available

What’s new in Azure Key Vault: https://techcommunity.microsoft.com/t5/video-hub/azure-key-vault-what-s-new/m-p/1698834

Introducing Azure Defender: https://myignite.microsoft.com/sessions/764ff397-97ff-4841-ad62-493f1da51d40

What’s new in Azure Security Center: https://myignite.microsoft.com/sessions/d40bd0a5-485e-455d-ac28-882b85de8dfb

Last month, I passed two Azure exams and earned the Azure Data Engineer Certification.

I take data engineer certificate as an example, to explain how to prepare and what the Azure certification exams look like.

Overview

Microsoft has made big changes to its Azure certifications at Ignite conference 2018. The new certifications are role-based, more practical and have a narrower focus for each certification.

Currently, there are 9 role based azure certifications. To earn each certification, it is required to clear one or two certification exams. I list the current certifications and also the required exams in the following:

• Associate
  • Data Engineer (DP-200, DP-201)
  • Data Scientist (DP-100)
  • Administrator (AZ-104)
  • Security Engineer (AZ-500)
  • Database Administrator (DP-300)
  • AI Engineer (AI-100)
  • Developer (AZ-204)
• Expert
  • Solution Architect (AZ-303, AZ-304)
  • DevOps Engineer (AZ-400)

Which certification is right for me?

Should you go with AWS, Azure, or Google Cloud? Whether you are looking to move into a high paying cloud career or just looking to declare your existing cloud skills, Azure certifications are a great choice. Besides all the benefits that will bring to your career, all the Azure certs can be obtained remotely, which is awesome when working from home to fight against COVID-19.

Once you decided which cloud provider path is right for you, your next step is to figure out which certification path is right for you. From my experience, since Microsoft’s Azure certifications are role-based, it is better to select the job title and also go through the skills measured for each certification.

Certification Learning Path

Data Engineer Certification: validates the skills and knowledge to design and implement the management, monitoring, security, and privacy of data using the full stack of Azure data services to satisfy business needs.

Job role: Data Engineer

Prerequisites: None

Required exams($165 each):

• DP-200 Implementing an Azure Data Solution
• DP-201 Designing an Azure Data Solution

Skills measured: skills outline

• Implement data storage solutions
• Manage and develop data processing
• Monitor and optimize data solutions
• Design Azure data storage solutions
• Design data processing solutions
• Design for data security and compliance

What the Azure certification online exams look like?

All the exams are held by Pearson VUE, the exam appointments appear on the dashboard after registration.

All Azure role based certifications can be taken online, but there are extra policies need to be follow.

Before the exam:

• Perform a system test
• prepare 2 Government issued personal ID
• prepare your phone for identity verification and room scans. Also the exam proctor might contact you if there is an issue during the exam. Make sure your mobile phone is outside of the immediate testing space, but within extended arms reach with the screen visible.
• prepare a work area:
  • Additional monitors (must be unplugged and turned away from you)
  • Additional computers (must be turned off and monitors must be dark)
  • clear of all materials, including the following items that are not allowed within arm’s reach: books, notepads, Post-it notes, typed notes/papers, or writing instruments such as pens, markers, whiteboards, or pencils.

Start the exam:

• log in 15 minutes early to start the check-in process.
• Choose Start a previously scheduled online proctored exam on dashboard.
• Select the exam under Purchased Online Exams.
• Select Begin exam and proceed through the self-check-in process and wait for a proctor to connect with you.

During the exam:

• No breaks/eating/drinking
• No personal belongings
• No exam assistance
• use facial comparison technology to verify identity during the testing process
• the proctor will continuously monitor you by video and audio, and your face, voice, the physical room where you are seated, and the location during exam delivery will be recorded.

After exam:

• when your exam ends, you should see your exam results(pass or fail) immediately before exiting the exam app.
• Also your sore report will be available on dashboard after several hours.
• You will receive an email about claiming your exams and certifications. Click the link to claim on cclaim, and then you can share your earned badge anywhere.

Reschedule policy: at least 6 business days prior to your appointment. 12.5% reschedule fee.

Cancelation policy: at least 24 hours prior to your appointment.

Exam question types

The exam may contain several question types: active screen, best answer, build list, case studies, drag and drop, hot area, multiple choice, repeated answer choices, short answer, labs, mark review, and review screen.

For security reason, exam formats or exact question types are not identified before the exam. You can view all question samples here to prepare the exam.

Learning materials

• Azure docs for mentioned products.
• Azure learn
• Pluralsight paths
• Cloud academy

A stateful structured stream processing job is suddenly stuck at the 1st micro-batch job. Here are the notes about that issue, how to debug Spark stateful streaming job, and also how I fix it.

Stateful Structured Streaming Processing Job

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

 ## specify data source, read data from Azure Event Hub
(spark.readStream.format("eventhubs") 
 .options(**self.config.ehConfig)
 .load()  
 ## use watermark to control state size
 .withWatermark(processTimeCol, waterMarkTime)
 ## transformations. 
 .withColumn(eventTimeCol, col(eventTimeCol).cast('timestamp'))
 ## aggregation by event time windows
 .groupBy(col(key), window(eventTimeCol, "15 mins"))
 .agg()
 ...
 .select(*(cols+['windowStart', 'firstEventTime', 'lastEventTime', 'count']))
 ## specify data sink, write transformed output to Azure blob storage
 .writeStream
 .format("parquet")
 .option('path', outputPath)
 .outputMode("append")
 ## Processing details--Trigger: when to process data
 .trigger(processingTime="2 seconds")  
 ## Processing details--Checkpoint: for tracking the progress of the query
 .option("checkpointLocation", checkpointPath)
 .start())

Spark SQL converts batch-like query to a series of incremental execution plans operating on new micro-batches of data.

Environment

This Streaming job is running on Databricks clusters triggered by Azure Data Factory pipeline.

Data source is from Azure Event Hub, and this job store the aggregated output to Azure Blob Storage mounted on Databricks.

Databricks instance has Vnet injections and have NSG associated with the Vnet.

Databricks cluster version is 5.5 LTS which use Scala 2.11, Spark 2.4.3 and Python 3.

We also use PySpark 2.4.4 for this streaming job.

Issue Symptom

This streaming job is scheduled to run for 4 hours every time. When the current job stops, the next one will start to run. It means the max number of concurrency job is 1.

It was running well before. Suddenly, when a new streaming job starts, it seems to stuck at the 1st micro-batch like the following picture. You can see the job is stuck for batch=0 and it fails because of time out.

We have 3 regions and this issue happened to every region one by one in a week.

If you check the checkpoint folder:

compare with the normal checkpoint:

The difference is obvious: there is no commits in the checkpoint path. It means the streaming job didn’t succeed in processing even a single micro-batch.

Possible Causes

It is really tough to debug this issue because no error message shown up and just several misleading warning messages in the executor’s error logs.

Usually, there are several possible reasons to cause streaming processing job stuck, such as:

• Total size of state per worker is too large which leads to higher overheads of snapshotting and JVM GC pauses.

• Number of shuffle partitions is too high, so the cost of writing state to HDFS will increase which cause the higher latency.

• NSG rules added to Databricks Vnet might block some ports and thus infect worker to worker communication.

• Databricks mounted blobs are expired and need to rotate the storage connection string and databricks access token.

• Databricks cluster version is deprecated and not supported any more.

• Spark .metadata directory is messed up. We need to delete the metadata and let the pipeline recreate a new one. but for this one, it would complete micro-batch, and it just do nothing in the process.

But none of them work this time. We struggle to figure out the root cause is:

• Azure blob storage has too many files in the checkpoint folder which slow down the read and write speed.

I compare the DAG visualization with normal job’s, it is shown that the StateStoreSave stage takes much longer (16 hours) than the normal one (21 seconds). StateStoreSave is the stage when spark store current streaming process status in checkpoint. Thus the issue exists in checkpoints. More info for StateStoreSave can be found here

From this detailed stage information, we can get:

1. number of total state rows is not the concern. we cannot solve the issue by reduce the watermark threshold or recreate checkpoint.

2. memory used by state total is lower than the normal state. so it is not a JVM GC pause issue.

3. time to update total is the pain point. It takes longer even the number of updated state rows is less, which point the issue to the write speed in blob storage.

   In the checkpoint folder, we stored around 17 million checkpoints for each region. After I delete the whole checkpoint folder and restart the streaming job, the issue is fixed. I am not 100% sure about the reason for it. One possible reason for it is Azure blob storage doesn’t support hierarchical namespace, and it just mimic hierarchical directory structure by using slashes in the name.

Solutions

Solution 1 - migrate Azure blob storage to Azure Data Lake Gen2 which supports hierarchical namespace.

Solution 2 - delete checkpoint folder and decrease retention period.

Step1: Stop current streaming job

Step2: Delete .metadata directory and checkpoint folder

Step3: Add a failover mechanism so that the streaming job will resume from where the streaming job stopped in the last successful data persistence.

Here is my failover mechanism if no checkpoint found, so we can delete the checkpoint folder without losing state.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

try:
       # if the checkpoint exists, continue to use it without refreshing
       config.dbutils.fs.ls(config.checkpointPath)
       print('Continue streaming job with checkpoint path, %s' %
             config.checkpointPath)
   except:
       # remove _spark_metadata folder when use a new checkpoint
       config.dbutils.fs.rm(config.outputPath + '_spark_metadata', True)
       print('removed _spark_metadata for last checkpoint')
       print('New streaming job checkpoint path is %s' %
             config.checkpointPath)
       # set the streaming start time to catch up from where the streaming job stoped in last data persistence
       timeKey = 'windowStart'
       try:
           ts = [int(p.path.split(timeKey + "=")[1][:-1])
                 for p in config.dbutils.fs.ls(config.outputPath) if timeKey in p.path]
           if ts:
               lookbackTs = int(dt.datetime.now().replace(minute=0, second=0, microsecond=0).timestamp()) - defualtLookbackTime
               # Set stream start time to the maximum of (15min aggregation output timestamp or one day back from current time)
               streamStartTime = np.max([np.max(ts), lookbackTs])
               # Add 15min to start time
               streamStartTime = dt.datetime.fromtimestamp(
                   streamStartTime) + dt.timedelta(minutes=15)
               streamStartTime = streamStartTime.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
               # Create the positions
               startingEventPosition = {
                   "offset": None,
                   "seqNo": -1,
                   "enqueuedTime": streamStartTime,
                   "isInclusive": True
               }
               config.ehConfig["eventhubs.startingPosition"] = json.dumps(
                   startingEventPosition)
        except:
           pass

Step 4: restart the streaming job.

Step 5: Add retention policy to the checkpoint folder to decrease the checkpoints lifetime.

When you need to migrate an old Databricks to a new Databricks, all of the files, jobs, clusters, configurations and dependencies are supposed to move. It is time consuming and also easy to omit some parts. I document the detailed migration steps, and also write several scripts to automatically migrate folders, clusters and jobs.
In this chapter, I will show you how to migrate Databricks.

0. Prepare all scripts

Navigate to https://github.com/xinyeah/Azure-Databricks-migration-tutorial, fork this repository, and download all needed scripts.

1. Install databricks-cli

1

pip3 install databricks

2. Set up authentication for two profiles

Set up authentication for two profiles for old databricks and new databricks. This CLI authentication need to done by a personal access token.

2.1 Generate a personal access token.

Here is step by step guide to generate it.

2.2 Copy the generated token and store it as a secret in Azure Key Vault.

On the Key Vault properties pages, select Secrets.
Click on Generate/Import.
On the Create a secret screen choose the following values:

• Upload options: Manual.
• Name:
• Value: paste the generated token here
• Leave the other values to their defaults. Click Create.

2.3 Set up profiles

In this case, the profile primary is for the old Databricks, and the profile secondary is for the new one.

1
2

databricks configure --profile primary --token
databricks configure --profile secondary --token

Every time set up a profile, you need to provide the Databricks host url and the personal access token generated previously.

2.4 Validate the profile

1
2

databricks fs ls --absolute --profile primary
databricks fs ls --absolute --profile secondary

3. Migrate Azure Active Directory users

3.1 Navigate to the old Databricks UI, expand Account in the right corner, then click Admin Console. You can get a list of users as admin in this Databricks.

3.2 Navigate to the new Databricks portal, click Add User under Users tag of Admin Console to add admins.

4. Migrate the workspace folders and notebooks

Solution 1
Put the migrate-folders.py in a separate folder (it will export files in this folder), and then run the migrate-folders.py script to migrate folders and notebooks. Libraries are not included using this scripts. It is shown in Step 5 to migrate libraries.
Remember to replace the profile variables in this script to your customized profile names:

1
2

EXPORT_PROFILE = "primary"
IMPORT_PROFILE = "secondary"

Solution 2
Also, you can do it manually: Export as DBC file and then import.

5. Migrate libraries

There is no external API for libraries, so need to reinstall all libraries into new Databricks manually.

5.1 List all libraries in the old Databricks.

5.2 Install all libraries.

Maven libraries:

PyPI libraries:

6. Migrate the cluster configuration

Run migrate-cluster.py to migrate all interactive clusters. This script will skip all job source clusters.
Remember to replace the profile variables in this script to your customized profile names:

1
2

EXPORT_PROFILE = "primary"
IMPORT_PROFILE = "secondary"

7. Migrate the jobs configuration

Run migrate-job.py to migrate all jobs, schedule information will be removed so job doesn’t start before proper cutover.
Remember to replace the profile variables in this script to your customized profile names:

1
2

EXPORT_PROFILE = "primary"
IMPORT_PROFILE = "secondary"

8. Migrate Azure Key Vaults secret scopes

There are two types of secret scope: Azure Key Vault-backed and Databricks-backed.
Creating an Azure Key Vault-backed secret scope is supported only in the Azure Databricks UI. You cannot create a scope using the Secrets CLI or API.

List all secret scopes:

1

databricks secrets list-scopes --profile primary

Generate key vault-backed secret scope:

1. Go to https://<databricks-instance>#secrets/createScope. This URL is case sensitive; scope in createScope must be uppercase.
2. Enter the name of the secret scope. Secret scope names are case insensitive.
3. These properties are available from the Properties tab of an Azure Key Vault in your Azure portal.
4. Click the Create button.

9. Migrate Azure blob storage and Azure Data Lake Storage mounts

There is no external API to use, have to manually remount all storage.

9.1 List all mount points in old Databricks using notebook.

1

dbutils.fs.mounts()

9.2 Remount all blob storage following the official docs using notebook.

1
2
3
4

dbutils.fs.mount(
  source = "wasbs://<container-name>@<storage-account-name>.blob.core.windows.net",
  mount_point = "/mnt/<mount-name>",
  extra_configs = {"<conf-key>":dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>")})

where

• <mount-name> is a DBFS path representing where the Blob storage container or a folder inside the container (specified in source) will be mounted in DBFS.
• <conf-key> can be either fs.azure.account.key.<storage-account-name>.blob.core.windows.net or fs.azure.sas.<container-name>.<storage-account-name>.blob.core.windows.net
• dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>") gets the key that has been stored as a secret in a secret scope.

10. Migrate cluster init scripts

Copy all cluster initialization scripts to new Databricks using DBFS CLI.

1
2
3
4
5

// Primary to local
dbfs cp -r dbfs:/databricks/init ./old-ws-init-scripts --profile primary

// Local to Secondary workspace
dbfs cp -r old-ws-init-scripts dbfs:/databricks/init --profile secondary

11. ADF config

For Databricks jobs scheduled by Azure Data Factory, navigate to Azure Data Factory UI. Create a new Databricks linked service linked to the new Databricks by the personal access key generated in step 2.

Reference

https://docs.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/vnet-inject
https://docs.microsoft.com/en-us/azure/azure-databricks/howto-regional-disaster-recovery#detailed-migration-steps
https://docs.microsoft.com/en-us/azure/databricks/dev-tools/cli/
https://docs.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes

Application Insights is one of Azure Monitoring solutions, it monitors the availability, performance, and usage of your web application.

How it works

Before you can use the Application Insights, you need to install an Application Insights SDK (instrumentation package) in your app. This instrumentation monitors your app and sends out the telemetry data to an Azure Application Insights resource identified by an instrumentation key (a unique GUID).

It support Java, C#, Node.js, python and so on

Flush data

The official docs says, the SDK sends out data at fixed intervals (typically 30 secs) or whenever the buffer is full (typically 500 items).

However, from my personal experience, it won’t send the data if you don’t flush.

Here is a C# code example to flush the telemetry.

1
2
3
4
5
6
7
8
9
10
11
12

// Set up some properties and metrics:
var properties = new Dictionary <string, string>
    {{"game", currentGame.Name}, {"difficulty", currentGame.Difficulty}};
var metrics = new Dictionary <string, double>
    {{"Score", currentGame.Score}, {"Opponents", currentGame.OpponentCount}};

// Send the event:
telemetry.TrackEvent("WinGame", properties, metrics);
// Flush the buffer
telemetry.Flush();
// Allow some time for flushing before shutdown.
System.Threading.Thread.Sleep(5000);

Reference

https://docs.microsoft.com/en-us/azure/azure-monitor/app/api-custom-events-metrics

https://docs.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview

Object Serialization is the process of converting an object into a stream of bytes to store or transmit the object between machines.

The reverse process is called deserialization to use the byte stream to recreate the object.

Issue for Java Serialization

The main concern to use Java Serialization is security issue. There is a so called Java deserialization vulnerability affect all apps that receives serialized Java objects which can be used by attackers to gain complete remote control of an app service. Also, the attack surface is so big and even if you adhere to all best practice, your app is still be vulnerable.

What’s the vulnerability?

Many apps that accept serialized bytes stream do not validate or check untrusted input before deserialization. The attackers can insert a malicious code into the bytes stream and have it execute on the app. They can easily mount a denial-of-service attack by causing the deserialization takes forever, which is called deserialization bomb.

Solutions

The best way to avoid Java serialization vulnerability is never to use Java serialization!

There are other mechanisms to store and transmit between objects and bytes sequences which avoid Java serialization vulnerability, such as JSON and Protocol Buffers(Protobuf).

JSON vs Protocol Buffers

I summarize the differences between them:

But they are both good serialization mechanisms:

• They are simpler than Java serialization.
• They don’t support auto serialization or deserialization
• They only support a few primitive and array data types to avoid deserialization issue.

Reference

Effective Java, Third Edition

https://www.darkreading.com/informationweek-home/why-the-java-deserialization-bug-is-a-big-deal/d/d-id/1323237

I struggled to deploy a Spark .NET app on Databricks scheduled by Azure Data Factory pipeline. Here are the notes on the solutions how I finally figured out.

From this chapter, you can step-by-step create a Spark .NET app and deploy it either on Databricks directly or scheduled by an Azure Data Factory pipeline.

Prepare a Spark .NET application

This doc teaches you how to run a Spark .NET app using .NET Core. If you are familiar with .NET, we can simplify the process as:

1. Prepare environment.

   1.1 Install the following dependencies: .NET, Java, compression software, Apache Spark, .NET for Apache Spark, WinUtils.

   1.2 Set DOTNET_WORKER_DIR environment variable.

   1.3 Verify you have all dependencies: you are good if you run dotnet, java,mvn,spark-shellfrom command line successfully.

2. Code a demo app to count words.

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

   using Microsoft.Spark.Sql; namespace MySparkApp { class Program { static void Main(string[] args) { // Create a Spark session. SparkSession spark = SparkSession .Builder() .AppName("word_count_sample") .GetOrCreate(); // Create initial DataFrame. DataFrame dataFrame = spark.Read().Text("input.txt"); // Count words. DataFrame words = dataFrame .Select(Functions.Split(Functions.Col("value"), " ").Alias("words")) .Select(Functions.Explode(Functions.Col("words")) .Alias("word")) .GroupBy("word") .Count() .OrderBy(Functions.Col("count").Desc()); // Show results. words.Show(); // Stop Spark session. spark.Stop(); } } }

3. Build your app.

   1	dotnet build

4. Locally submit your app to run on Apache Spark.

   1 2 3 4 5

   spark-submit \ --class org.apache.spark.deploy.dotnet.DotnetRunner \ --master local \ microsoft-spark-2.4.x-<version>.jar \ dotnet HelloSpark.dll

5. If it is successful, you can see the word count data written on the console.

Prepare dependencies on Databricks

1. Download Microsoft.Spark.Worker which helps Apache Spark execute your app.

2. Download install-worker.sh which copys .NET for Apache Spark dependencies into your cluster’s nodes.

3. Download db-init.sh which installs dependencies on your Databricks cluster.

4. Publish your Spark .NET app.

   1	dotnet publish -c Release -f netcoreapp3.1 -r ubuntu.16.04-x64

5. Compress the published app files in the previous step. Navigate to mySparkApp/bin/Release/netcoreapp3.1/ubuntu.16.04-x64, compress Publish folder as a zip file.

6. Upload files to DBFS.

   1 2 3 4 5 6 7 8 9 10

   databricks fs cp db-init.sh dbfs:/spark-dotnet/db-init.sh databricks fs cp install-worker.sh dbfs:/spark-dotnet/install-worker.sh databricks fs cp Microsoft.Spark.Worker.netcoreapp3.1.linux-x64-0.6.0.tar.gz dbfs:/spark-dotnet/ Microsoft.Spark.Worker.netcoreapp2.1.linux-x64-0.6.0.tar.gz cd mySparkApp databricks fs cp input.txt dbfs:/input.txt cd mySparkApp\bin\Release\netcoreapp3.1\ubuntu.16.04-x64 directory databricks fs cp mySparkApp.zip dbfs:/spark-dotnet/publish.zip databricks fs cp microsoft-spark-2.4.x-0.6.0.jar dbfs:/spark-dotnet/microsoft-spark-2.4.x-0.6.0.jar

7. Then all the dependencies are ready. We can deploy it on Databricks.

How to deploy

We can run .NET for Apache Spark apps on Databricks, but it is not what we usually do for Python or Scala jobs. For the Python or Scala jobs, we can just start a Notebook task for them. But for Spark .NET job, we need to use the “spark-submit” or “Jar” tasks.

Scheduled by Azure Data Factory pipeline

Deploy using Set Jar

1. Generate a Databricks access token for Azure Data Factory to access.

   1.1 In Databricks workspace, select your user profile in the upper right, and select User Settings.

   

   1.2 Select Generate New Token under the Access Tokens tab.

   

   1.3 Save the access token for later use in creating a Databricks linked service. Usually save it in Azure Key Vault for security.

2. Navigated to the Pipelines page on Azure Data Factory, create a new pipeline, search for Databricks activities, drag the Jar task to panel.

   

3. In the Jar activity Demo, updates the paths and settings as needed. Databricks linked service should be created using access token generated on Databricks previously. Remember to add init script for cluster settings.

   

4. Check the Jar settings. Main class name is org.apache.spark.deploy.dotnet.DotnetRunner. Parameters will pass to the main class. it must have your app publish.zip and your app name as the first two parameters. The rest parameters are what your app need. Append libraries are microsoft-spark-2.4.x-0.10.0.jar on dbfs.

   

5. Click Debug to run a test for the current pipeline.

6. Save the newly added pipeline by click Publish all.

Directly on databricks

1. Deploy using Spark-submit

1. Navigate to Databricks Workspace and create a job. Select Task as spark-submit. Set job parameters。

   

2. When configure Cluster, need to add init script located on DBFS (Databricks Filesystem).

   

3. select Run Now to test the job. Once the job’s cluster is created, your Spark job will be submitted.

   

2. Deploy using Set Jar

We can also use the Jar task to deploy on Databricks. The settings should be the same with the one triggered by Azure Data Factory.

Reference

https://docs.microsoft.com/en-us/dotnet/spark/how-to-guides/databricks-deploy-methods

https://docs.microsoft.com/en-us/azure/data-factory/solution-template-databricks-notebook

https://docs.microsoft.com/en-us/azure/data-factory/transform-data-databricks-jar

https://docs.microsoft.com/en-us/dotnet/spark/tutorials/get-started

https://dotnet.microsoft.com/learn/data/spark-tutorial/intro

首要原因：没钱。这是一套免费的组合拳。

在众多站点选择中，最终选择了Github Pages。主要还是因为熟悉Github的版本控制，以及Github对其他平台很好的集成。官方推荐的静态站点生成器(static site generator)是Jekyll。还可以在项目仓库Settings页面中的Github Pages部分，选择Jekyll的theme。

Hexo也是一款静态站点生成框架(static site generator)，基于Node.js 。通过Hexo可以使用Markdown来写文章，不用太关注排版和格式。而且Hexo比较成熟，有很多稳定的好看的主题。

Travis CI 是持续集成(continuous integration)的平台，可以监控repo具体分支上的代码变动，自动触发build和test。帮助实现频繁的merge小段代码的Best practice。有了自动部署，就可以不受开发平台限制，不需要搭建环境也可以发布文章。

安装环境

1. 安装并配置github

2. 安装Node.js 和 npm

   npm(Node Package Manager), 是用来开发和分享Javascript代码的工具。在这里https://nodejs.org/en/download/下载Node.js的最新版本，就包含了NPM。

   打开Command Prompt，验证安装成功了Node.js和NPM。

   1 2 3 4 5

   $ node –v v12.17.0 $ npm –v 6.14.4

3. 安装Hexo

   1 2 3 4 5 6 7

   $ npm install -g hexo-cli $ hexo -v hexo-cli: 3.1.0 os: Windows_NT 10.0.18363 win32 x64 node: 12.17.0 ...

4. 安装Hexo-deployer-git

   1	$ npm install hexo-deployer-git --save

初始化Hexo+Github Pages项目

1. 初始化.github.io为Hexo项目

   1 2 3 4 5 6 7 8 9 10 11

   $ mkdir <YourName>.github.io $ cd <YourName>.github.io $ hexo init INFO Copying data to ~/***/<YourName>.github.io INFO You are almost done! Don't forget to run 'npm install' before you start blogging with Hexo! $ npm install $ git init

2. 初始化后的目录如下：

   .
   ├── _config.yml #站点的配置文件
   ├── package.json #应用的基本信息和依赖应用
   ├── scaffolds #模板文件夹。新建文章时候，默认填充的内容模板。
   ├── source #markdown和html文件会被解析存放在public文件夹中
   | ├── _drafts #新建的draft会保存在这里
   | └── _posts #新建post的时候会保存在这里
   └── themes #主题文件夹，根据主题来生成静态页面

3. Github上创建一个.github.io为名的公开的代码库。其中Yourname应该跟你的Github用户名保持一致。

   为了防止错误，不要用 README, license, 或者 gitignore文件初始化项目.

   代码库Settings中查看Github Pages相关设置，你就拥有了自己的站点：https://.github.io。对于个人站点，只能将master分支设置为发布来源。

   

4. 复制代码库的URL。

2. 在本地代码库添加remote upstream

   1 2 3 4

   $ git remote add origin remote-repository-URL # Sets the new remote $ git remote -v # Verifies the new remote URL

3. 根据文档，修改_config.yml文件中关于站点的配置信息。

4. 执行以下命令， 验证效果

   1 2 3 4

   $ hexo clean $ hexo generate $ hexo server INFO Hexo is running at http://0.0.0.0:4000/. Press Ctrl+C to stop.

添加博客主题

1. Fork hexo-theme-next 项目到自己的仓库.

2. 运行以下命令将 Fork 出来的仓库 pull 到本地子模块

   1 2	cd <YourName>.github.io git submodule add https://github.com/<YourName>/hexo-theme-next.git themes/next

运行该命令后会在项目根目录生成 .gitmodules 文件，文件内容如下：

1
2
3

[submodule "themes/next"]
    path = themes/next
    url = https://github.com/sugartxy/hexo-theme-next

3. 对主题进行个性化配置后，先要 check in子模块，在 theme/next 目录下依次执行：

   1 2 3 4

   cd theme/next git add . git commit -m "update config file" git push origin master

4. 切换到项目根目录，打开站点配置文件(.github.io/_config.yml)，修改theme字段, 使得主题修改生效。

   1	theme: next

5. 执行以下命令， 验证效果

   1 2	$ hexo server INFO Hexo is running at http://0.0.0.0:4000/. Press Ctrl+C to stop.

6. 在项目根目录下，将代码check in到项目仓库下：

   1 2 3 4

   cd <YourName>.github.io git add . git commit -m "add submodule" git push origin master

生成博客并部署

1. 执行以下命令生成新的博客

   1 2	$ hexo new post <title> INFO Created: ~/<YourName>/<YourName>.github.io/source/_posts/<title>.md

   将博客内容写在新创建的markdown文件里。

2. 如果themes/next路径下的内容做了改变，在themes/next路径下，将更改的代码check in到刚刚Fork的repo中。

3. 在YourName.github.io项目路径下，将更改的代码check in到YourName.github.io repo 的master分支.

4. 在本地部署。使用后续Travis CI配置后，可以省略此步骤。

   4.1 修改配置文件_config.yml中关于部署的字段

   1 2 3 4

   deploy: type: git repository: https://git@github.com/<YourName>/<YourName>.github.io.git branch: master

   4.2 执行以下命令部署站点，当执行 hexo deploy 时，Hexo 会将 public 目录中的文件和目录推送至 _config.yml 中指定的远端仓库和分支中，并且完全覆盖该分支下的已有内容。

   1 2 3

   $ hexo clean # 清除缓存文件（db.json）和已经生成的静态文件（public） $ hexo generate # 生成静态文件 $ hexo deploy # 部署网站

使用Travis CI自动化部署

Travis CI对于开源的Repository是免费的，只需要拥有Github账户和至少一个项目，在项目中增加.travis.yml文件，就可以使用Travis CI。Hexo文档中详细说明了如何使用Travis CI将Hexo自动部署到Github Pages。只需要做如下修改:

1. 修改.travis.yml文件

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

   sudo: required language: node_js node_js: - 10 # use nodejs v10 LTS branches: only: - master # build master branch only # Start: Build Lifecycle install: - npm install -g hexo-cli - npm install - npm install hexo-deployer-git --save # 设置git提交名，邮箱 - git config user.name "<YourName>" - git config user.email "<YourEmail>" # 替换同目录下的_config.yml文件中gh_token字符串为travis后台刚才配置的变量，注意此处sed命令用了双引号。单引号无效！ - sed -i "s/gh_token/${GH_TOKEN}/g" ./_config.yml script: - hexo clean - hexo generate # generate static files after_success: # 只有前面步骤成功了才会触发 - hexo deploy # End: Build LifeCycle

2. 修改_config.yml文件

   1 2 3 4 5

   deploy: type: git # 下方的gh_token会被.travis.yml中sed命令替换 repo: https://gh_token@github.com/<YourName>/<YourName>.github.io.git branch: master

这样每一次更新博客，只需要check in Markdown文件到master 分支，就会自动部署。在Travis CI网站中可以看到部署的状态。

其他问题

1. 添加评论系统-gitalk

参考文献：https://www.standbyside.com/2018/12/04/add-comment-function-to-next/

1.1 进入github新建一个认证application

创建完后会生成这个application对应的 Client ID 和 Client Secret

1.2 在自己的github中创建一个同名的repository

以后每篇文章都会对应这里的一个issue，这篇文章的comments和like都会记录到对应的issue里。

1.3 Next主题v7.6.0中已经集成了gitalk，只需要进入主题的_config.yml里修改comments相关属性

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

comments:
    # Available values: tabs | buttons
    style: tabs
    # Choose a comment system to be displayed by default.
    # Available values: changyan | disqus | disqusjs | gitalk | livere | valine
    active: gitalk
    # Setting `true` means remembering the comment system selected by the visitor.
    storage: true
    # Lazyload all comment systems.
    lazyload: false
    # Modify texts or order for any navs, here are some examples.
    nav:
        #disqus:
        #  text: Load Disqus
        #  order: -1
        #gitalk:
        #  order: -2
    
gitalk:
    enable: true # 启用gitalk
    github_id: # 你的github用户名
    repo: # 刚才你创建的repository的名字，只要名字，不要全链接
    client_id: # 你的 Client ID
    client_secret: # 你的 Client Secret
    admin_user: # 联系人, 页面显示联系**初始化评论
    distraction_free_mode: true  # Facebook-like distraction free mode
    # Gitalk's display language depends on user's browser or system environment
    # If you want everyone visiting your site to see a uniform language, you can set a force language value
    # Available values: en | es-ES | fr | ru | zh-CN | zh-TW
    language:

2. 本地图片无法显示

参考文献：https://merrier.wang/20190111/image-skills-in-hexo.html

2.1 在路径 yourName.github.io/source下创建images文件夹，将图片全部放在这个文件夹下。

2.2 Markdown访问图片方式：

1

![](/images/image_name.jpg)

参考文献

Next中文教程：https://theme-next.iissnan.com/getting-started.html#description-setting

Hexo中文教程：https://hexo.io/zh-cn/docs/

Github Pages中文教程：https://help.github.com/cn/github/working-with-github-pages

Travis官方文档：https://docs.travis-ci.com/user/tutorial/