Introduction

H&M Group is a family of brands and businesses with 53 online markets and approximately 4,850 stores. The online store offers shoppers an extensive selection of products to browse through. But with too many choices, customers might not quickly find what interests them or what they are looking for, and ultimately, they might not make a purchase. To enhance the shopping experience, product recommendations are key. More importantly, helping customers make the right choices also has a positive implications for sustainability, as it reduces returns, and thereby minimizes emissions from transportation.

The goal of this data science challenge is to develop product recommendations based on data from previous transactions, as well as from customer and product meta data. The available meta data spans from simple data, such as garment type and customer age, to text data from product descriptions, to image data from garment images. Here we’re not going to use the images.

This project is divided in 2 parts:

• the first one (this notebook) is an EDA in order to gain insights from the available datasets, and to know how to prepare the dataset for the 2nd step
• in a second notebook: we’ll use the python library LightFM to build different recommendations models.

First insight

Let’s start by importing all the libraries we’re going to use, and load the three datasets relative to the customers, articles and transactions:

import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px

# plotly as pandas backend
# pd.options.plotting.backend = "plotly"



import os
from scipy import sparse

ENV = "COLAB"  # "LOCAL"  #

if ENV == "COLAB":
    from google.colab import drive
    drive.mount('/content/drive')
    dir_path = "drive/MyDrive/recomm/projet/"
else:
    dir_path = "../../../dataset/"


file_customers = "customers.csv"
file_articles = "articles.csv"
file_transactions = "transactions_train.csv"


df_customers = pd.read_csv(dir_path + file_customers)
df_articles = pd.read_csv(dir_path + file_articles)
df_transactions = pd.read_csv(dir_path + file_transactions)

Usually informations on the customers are more used for marketings (clustering / segmentation & KYC) purpose rather than for building the recommendation system. In order to get a big picture, we can start by a description of each datasets’ features (type, number & percentage of missing values, number & percentage of unique values and so on):

Metadata for each customer_id in dataset

def describe_df(df):
    list_item = []
    for col in df.columns:
        list_item.append([
            col,
            df[col].dtype,
            df[col].isna().sum(),
            round(df[col].isna().sum()/len(df[col])*100, 2),
            df[col].nunique(),
            round(df[col].nunique()/len(df[col])*100, 2),
            list(df[col].unique()[:5])
        ])
    return pd.DataFrame(
        columns=['feature', 'type', '# null', '% null', '# unique', '% unique', 'sample'],
        data = list_item
    )


assert df_customers.customer_id.nunique() == df_customers.shape[0]
describe_df(df_customers)

Detailed metadata for each article_id available for purchase:

This database contains information about the assortiment of H&M shops.

Unique indentifier of an article:

• article_id (int64) - an unique 9-digit identifier of the article, 105 542 unique values (as the length of the database)

5 product related columns:

• product_code (int64) - 6-digit product code (the first 6 digits of article_id, 47 224 unique values
• prod_name (object) - name of a product, 45 875 unique values
• product_type_no (int64) - product type number, 131 unique values
• product_type_name (object) - name of a product type, equivalent of product_type_no
• product_group_name (object) - name of a product group, in total 19 groups

2 columns related to the pattern:

• graphical_appearance_no (int64) - code of a pattern, 30 unique values
• graphical_appearance_name (object) - name of a pattern, 30 unique values

2 columns related to the color:

• colour_group_code (int64) - code of a color, 50 unique values
• colour_group_name (object) - name of a color, 50 unique values

4 columns related to perceived colour (general tone):

• perceived_colour_value_id - perceived color id, 8 unique values
• perceived_colour_value_name - perceived color name, 8 unique values
• perceived_colour_master_id - perceived master color id, 20 unique values
• perceived_colour_master_name - perceived master color name, 20 unique values

2 columns related to the department:

• department_no - department number, 299 unique values
• department_name - department name, 299 unique values

4 columns related to the index, which is actually a top-level category:

• index_code - index code, 10 unique values
• index_name - index name, 10 unique values
• index_group_no - index group code, 5 unique values
• index_group_name - index group code, 5 unique values

2 columns related to the section:

• section_no - section number, 56 unique values
• section_name - section name, 56 unique values

2 columns related to the garment group:

• garment_group_n - section number, 56 unique values
• garment_group_name - section name, 56 unique values

1 column with a detailed description of the article:

• detail_desc - 43 404 unique values

assert df_articles.article_id.nunique() == df_articles.shape[0]
describe_df(df_articles)

the training data, consisting of the purchases each customer for each date, as well as additional information. Duplicate rows correspond to multiple purchases of the same item. Our task is to predict the article_id each customer will purchase :

df_transactions.t_dat = pd.to_datetime(df_transactions.t_dat, infer_datetime_format=True)
describe_df(df_transactions)

Focus on Customers

The metadata for each customer_id in the dataset consists of club member status, whether they subscribe to fashion news or not, and age. Some types are wrong due to missing values:

print(f"there are {(df_customers.age != df_customers.age.round()).sum()} customers whose age is not an integer")
print(f"there are {df_customers.age.isnull().sum()} ages missing")

there are 15861 customers whose age is not an integer
there are 15861 ages missing

Missing ages are arbitrarily replaced by 0 so that unknow values can be distinguished:

mapping = {"FN": 0, "Active": 1, "club_member_status": "N.C", "fashion_news_frequency": "N.C", "age": 0}

df_customers.fillna(value=mapping, inplace=True)
df_customers.drop(columns="postal_code", inplace=True)

for col in ["FN", "age", "Active"]:
    df_customers[col] = df_customers[col].astype(np.int8)

Now let’s see the ratios of the number of customers for each feature:

cols = ["FN", "Active", "club_member_status", "fashion_news_frequency"]
fig, axes = plt.subplots(nrows=1, ncols=len(cols), figsize=(12, 6), tight_layout=True)

for i, c in enumerate(cols):
  df_customers[c].value_counts().plot.pie(ax=axes[i], title=c)

cols = ["FN", "Active", "club_member_status", "fashion_news_frequency"]
fig, axes = plt.subplots(nrows=1, ncols=len(cols), figsize=(12, 3), tight_layout=True)

for i, c in enumerate(cols):
  df_customers[c].value_counts().plot.bar(ax=axes[i], title=c)

By visualizing the distribution of the age, we can clearly see that:

• there are two types of clients: between 20 & 40, and older than 40 yrs old.
• the number of missing values isn’t too high

plt.figure(figsize=(6, 2))
sns.histplot(data=df_customers, x='age', bins=50)

The shape of the age distribution remains quite the same for each category of customer:

for c in cols:
  plt.figure(figsize=(6, 2))
  sns.histplot(data=df_customers, x='age', bins=50, hue=c, element="poly")
  plt.show()

Focus on Articles

With a sunburst chart, we visualize the hierarchical structures of the different clothes’ categories:

• the “sport” category is the least represented
• while the “ladieswear” & the “baby/children” seem to be the most important
• unlike the “ladieswear” composed mostly of “Garment Upper Body”, the “baby/children” clothes are more diversified:

cols = ["index_group_name", "index_name",  "product_group_name"]
df_temp = pd.DataFrame(df_articles[cols].value_counts()).rename(columns={0: "counts"}).reset_index()
px.sunburst(
    df_temp,
    path=cols,
    values='counts',
)

If we don’t group those clothes by index_group by rather by product type, we can see that the most represented product are: dresses, sweaters, swim wear bodies & trousers:

def plot_bar(df, column):
    long_df = pd.DataFrame(df.groupby(column)['article_id'].count().reset_index().rename({'article_id': 'count'}, axis=1))
    fig = px.bar(long_df, x=column, y="count", color=column, title=f"bar plot for {column} ", width=900, height=550)
    fig.show()

def plot_hist(df, column):
    fig = px.histogram(df, x=column, nbins=10, title=f'{column} distribution ')
    fig.show()


plot_bar(df_articles,'product_type_name')

plot_bar(df_articles,'product_group_name')

plot_bar(df_articles,'graphical_appearance_name')

Focus an Transactions

Most item catalogs exhibit the long tail effect (popularity bias): very few items are demanded & sold, whereas most of the articles aren’t sold that much.

df_transactions['article_id'].value_counts().reset_index().drop(columns=["index"]).plot(figsize=(6, 4))

Curiously, the same is also true for customers. This can be explained by the fact that few customers are in fact societies or resellers that buy high volumnes of items, where as the vast majority of the customers have only bought 1 or 2 items:

df_transactions['customer_id'].value_counts()\
  .reset_index().sort_index().plot(figsize=(6, 4))

The history of transactions spans three years:

df_transactions["t_dat"].dt.year.unique()

array([2018, 2019, 2020])

Few spikes can be observed from the total daily sales, and can be explained by the shopping events or highly promoted sales at discounted price such as the “black friday”:

df_transactions['month'] = df_transactions["t_dat"].dt.month
df_transactions['year'] = df_transactions["t_dat"].dt.year
df_transactions['dow'] = df_transactions["t_dat"].dt.day_name()

df_temp = df_transactions.groupby('t_dat')['price'].agg(['sum', 'mean']).sort_values(by = 't_dat', ascending=False).reset_index()
px.line(df_temp, x='t_dat', y='sum', title='Total Sales daily', width=900, height=450).show()

The count of monthly sells shows that more items are sold during the summer (remember the swim wear body as most demanded):

df_temp = df_transactions.groupby(['year', 'month']).count()["article_id"].reset_index().rename(columns={"article_id": "count"})
px.line(df_temp, x="month", y="count", color='year', width=900, height=350, markers=True)

2018 is an incomplete year in our dataset, that’s why there are fewer sells monthly. But there are also fewer monthly sells in 2020 compared to 2019 for the month

df_temp = df_transactions.groupby(["year", "month"]).agg({"price": "sum"}).reset_index()

px.histogram(
    df_temp,
    x="month",
    y="price",
    title='Monthly sells for each year',
    color='year',
    barmode='group',
    nbins=12,
    width=900,
    height=450
).show()

df_temp = df_transactions.groupby(["year", "dow"]).agg({"price": "sum"}).reset_index()

px.histogram(
    df_temp,
    x="dow",
    y="price",
    title='Daily sells for each year',
    color='year',
    barmode='group',
    nbins=10,
    width=900,
    height=450
).show()

Transactions analysis for different customers or articles categories

Let’s keep only few months for the sake of simplicity (and also because it’s quite hard to process huge amount of data with limited ressources):

df = df_transactions[(df_transactions.t_dat.dt.year == 2019) & (df_transactions.t_dat.dt.month.isin([5, 6, 7, 9]))]
df.shape

(6501193, 8)

df_transactions.head()

df = df.merge(df_articles[["article_id", "index_group_name", "index_name", "section_name"]], on='article_id')
df.drop(columns=["article_id"], inplace=True)#, "month", "year"])
del df_articles


df = df.merge(df_customers, on='customer_id')
df.drop(columns=["customer_id"], inplace=True)
del df_customers


# df.drop(columns=["postal_code"], inplace=True)
df['month'] = df.t_dat.dt.month
# df['year'] = df.t_dat.dt.year
df['dow'] = df.t_dat.dt.day_name
print(f"Total Memory Usage: {df.memory_usage(deep=True).sum() / 1024**2:.2f} MB")
df.head()

Total Memory Usage: 2751.01 MB

df['dow'] = df["t_dat"].dt.day_name()


def plot_var_accross_time(var):
  for time_scale in ['month', 'dow']:
    df_temp = df.groupby([time_scale, var]).count()["t_dat"].reset_index().rename(columns={"t_dat": "count"})
    px.line(df_temp, x=time_scale, y="count", color=var, width=900, height=350, markers=True, title=f'Evolution of transactions for different {var} for each {time_scale} over 2019').show()


plot_var_accross_time("index_group_name")

plot_var_accross_time("index_name")

plot_var_accross_time("Active")

Introduction

In the 1st part of this project we’ve analyzed in depth the different available datasets. Now, in this second and final step we’re going to build a product recommendations system based on data from previous transactions, by using the LightFM python library.

!pip install lightfm

As usual let’s import all that we need:

import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px

import itertools
import pickle

import os
from scipy import sparse

from lightfm import LightFM

from lightfm.data import Dataset
from lightfm.cross_validation import random_train_test_split
from lightfm.evaluation import precision_at_k, recall_at_k, auc_score

pd.set_option('mode.chained_assignment', None)


RANDOM_STATE = 42
ENV = "COLAB"  # "LOCAL"  #


if ENV == "COLAB":
    from google.colab import drive
    drive.mount('/content/drive')
    dir_path = "drive/MyDrive/recomm/projet/"
else:
    dir_path = "../../../dataset/"


file_customers = "customers.csv"
file_articles = "articles.csv"
file_transactions = "transactions_train.csv"


df_customers = pd.read_csv(dir_path + file_customers)
df_articles = pd.read_csv(dir_path + file_articles)
df_transactions = pd.read_csv(dir_path + file_transactions)

What is LightFM?

It’s a hybrid matrix factorisation model representing users and items as linear combinations of their content features’ latent factors. The model seems to outperforms both collaborative and content-based models in cold-start or sparse interaction data scenarios (using both user and item metadata), and performs at least as well as a pure collaborative matrix factorisation model where interaction data is abundant.

In LightFM, like in a collaborative filtering model, users and items are represented as latent vectors (embeddings). However, just as in a CB model, these are entirely defined by functions (in this case, linear combinations) of embeddings of the content features that describe each product or user.

How LightFM works?

The LightFM paper describes its inner working: a lightFM model learns embeddings (latent representations in a high-dimensional space) for users and items in a way that encodes user preferences over items. When multiplied together, these representations produce scores for every item for a given user; items scored highly are more likely to be interesting to the user.

Data Preparation

For recommendation models, we have to deal with sparse datasets. Here we’re going to keep a subset as dense as possible, by keeping only one full year (2019) of transactions:

assert df_articles.article_id.nunique() == df_articles.shape[0]
Hands-On Machine Learning with Scikit-Learn and TensorFlow

print(f"Nb of transactions before filtering: {df_transactions.shape[0]}")
df_transactions.t_dat = pd.to_datetime(df_transactions.t_dat, infer_datetime_format=True)
df = df_transactions[(df_transactions.t_dat.dt.year == 2019)] # & (df_transactions.t_dat.dt.month.isin([5, 6, 7]))] # DEBUG
print(f"Nb of transactions after filtering:  {df.shape[0]}")

df = df.merge(df_articles[["article_id", "index_group_name", "index_name", "section_name"]], on='article_id')

# del df_articles
# df = df.merge(df_customers, on='customer_id') # not needed
# del df_customers

f"Total Memory Usage: {df.memory_usage(deep=True).sum() / 1024**2:.2f} MB"

Nb of transactions before filtering: 31788324
Nb of transactions after filtering:  5274015
'Total Memory Usage: 1870.28 MB'

We keep only customers with at least 10 transactions, and drop customers with more than 100 purchases (resellers):

customers_count = df['customer_id'].value_counts()
customers_count[(customers_count > 10) & (customers_count < 50)].shape

(154282,)

We also filter customer_id aged above 38 as it seems to be one of the target of H&M according to our EDA:

df_customers.customer_id.nunique(), df_customers[(df_customers.age > 16) & (df_customers.age < 38)].customer_id.nunique(),

(1371980, 803696)

print(f"Nb of customers before filtering: {df.customer_id.nunique()} and nb_transactions {df.shape[0]}")

customers_count = df['customer_id'].value_counts()

# 1st selection based on the nb of transactions
customers_kept = customers_count[(customers_count > 10) & (customers_count < 100)].index.values
df = df[df.customer_id.isin(customers_kept)]

# 2nd selection based on the customers' ages
customers_kept = df_customers[(df_customers.age > 16) & (df_customers.age < 38)].customer_id.unique()
df = df[df.customer_id.isin(customers_kept)]

print(f"Nb of customers after filtering: {df.customer_id.nunique()} and nb_transactions {df.shape[0]}")

f"Total Memory Usage: {df.memory_usage(deep=True).sum() / 1024**2:.2f} MB"

Nb of customers before filtering: 559625 and nb_transactions 5274015
Nb of customers after filtering: 99603 and nb_transactions 2127829
'Total Memory Usage: 754.88 MB'

For the sake of simplicity (and because training recommendation models on huge amount of data required many computation ressources), we’re also going to keep only the main clothes’ target: “Ladieswear”

sns.countplot(y=df["index_group_name"])

df_temp = df.groupby(["index_group_name", "index_name"]).count()['t_dat'].reset_index().rename(columns={"t_dat": "count"})
px.bar(
    df_temp, x="count", y="index_group_name",
    color='index_name', barmode='group',
    width=700, height=400
).show()

For now, we’re not going to use the items features. Usually, the customers informations are more used for marketing purpose rather than features for the recommendation model. At the end, the dataset is only composed of customer_id & article_id, with many duplicated rows when items are purchased multiple times:

df = df.loc[df.index_name == "Ladieswear"]

# if we want to restore the original dataset without loading it again
df_backup = df.copy()

df.drop(columns=[
    't_dat',
    'price',
    'sales_channel_id',
    'index_group_name',
    'index_name',
    'section_name'], inplace=True)

print(df.shape)
print(df.customer_id.nunique(), df.article_id.nunique())
df.head()

(837769, 2)
92444 10962

Feedback matrix

Firstly, we have to create lightFM Dataset for our model. LightFM Dataset class makes it really easy for us for creating interection matrix, weights and user/item features.

• interection matrix: It is a matrix that contains user/ item interections or professional/quesiton intereactions.
• weights: weight of interection matrix. Less weight means less importance to that interection matrix.
• user/item features: user/item features supplied as like this (user_id, [‘feature_1’, ‘feature_2’, ‘feature_3’])

The LightFM libary can only be trained on sparse matrix: this is the types of dataset return by the build_interactions method based on our initial dataset:

dataset = Dataset()

# mapping creation
dataset.fit(
  users=df.customer_id.unique(),
  items=df.article_id.unique(),
  user_features=None,
  item_features=None
)

interactions, weights = dataset.build_interactions([(x[0], x[1]) for x in df.values])

int_dense = interactions.todense()
print(int_dense.shape)
int_dense

(92444, 10962)





matrix([[1, 0, 0, ..., 0, 0, 0],
        [1, 0, 0, ..., 0, 0, 0],
        [1, 0, 0, ..., 0, 0, 0],
        ...,
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0]], dtype=int32)

Split dataset

Let’s create separated train and test datasets:

train, test = random_train_test_split(
  interactions = interactions,
  test_percentage = 0.2,
  random_state = np.random.RandomState(seed=RANDOM_STATE)
)
train.todense().shape, test.todense().shape

((92444, 10962), (92444, 10962))

Recommendation systems

Recommender Systems are powerful, successful and widespread applications for almost every business selling products or services. It’s especially useful for companies with a wide offer and diverse clients. Ideal examples are retail companies as well as these selling services or digital products.

You’ll notice that after creating an account on Netflix or Spotify for example, the service will start to recommend you other products, movies or songs that the algorithm thinks will suit you the best. It’s their way to personalise the offer and who doesn’t like to get such care. That’s why these systems are precious for business owners. The more you buy, watch and listen the better it gets. Also, the more users the better it gets.

Recommender Systems usually are classified into three groups:

• Collaborative-filtering:

Collaborative filtering is a method of making automatic predictions (filtering) about the interests of a user by collecting preferences or taste information from many users (collaborating). The underlying assumption of the collaborative filtering approach is that if a person A has the same opinion as a person B on an issue, A is more likely to have B’s opinion on a different issue than that of a randomly chosen person.

Though collaborative filtering One major problem of collaborative filtering is “cold start”. As we’ve seen, collaborative-filtering can be a powerful way of recommending items based on user history, but what if there is no user history? This is called the “cold start” problem, and it can apply both to new items and to new users. Items with lots of history get recommended a lot, while those without never make it into the recommendation engine, resulting in a positive feedback loop. At the same time, new users have no history and thus the system doesn’t have any good recommendations. Potential solution: Onboarding processes can learn basic info to jump-start user preferences, importing social network contacts.

• Content-based filtering

These filtering methods are based on the description of an item and a profile of the user’s preferred choices. In a content-based recommendation system, keywords are used to describe the items; besides, a user profile is built to state the type of item this user likes. In other words, the algorithms try to recommend products which are similar to the ones that a user has liked in the past. The idea of content-based filtering is that if you like an item you will also like a ‘similar’ item. For example, when we are recommending the same kind of item like a movie or song recommendation.

One major problem of this approach is the diversity. Relevance is important, but it’s not all there is. If you watched and liked Star Wars, the odds are pretty good that you’ll also like The Empire Strikes Back, but you probably don’t need a recommendation engine to tell you that. It’s also important for a recommendation engine to come up with results that are novel (that is, stuff the user wasn’t expecting) and diverse (that is, stuff that represents a broad selection of their interests).

• Hybrid recommender system:

Hybrid recommender system is a special type of recommender system that combines both content and collaborative filtering method. Combining collaborative filtering and content-based filtering could be more effective in some cases. Hybrid approaches can be implemented in several ways: by making content-based and collaborative-based predictions separately and then combining them; by adding content-based capabilities to a collaborative-based approach (and vice versa). Several studies empirically compare the performance of the hybrid with pure collaborative and content-based methods and demonstrate that hybrid methods can provide more accurate recommendations than pure approaches. These methods can also be used to overcome some of the common problems in recommender systems such as cold start and the sparsity problem.

Building models with LightFM

We start building our LightFM model using LightFM class. LightFM class makes it really easy for making lightFM model. After that we will fit our model on our train dataset.

Baseline

model = LightFM(
    no_components=50,
    learning_rate=0.05,
    loss='warp',
    random_state=RANDOM_STATE)


model.fit(
    train,
    item_features=None,
    user_features=None,
    sample_weight=None,
    epochs=5,
    num_threads=4,
    verbose=True
)

Epoch: 100%|██████████| 5/5 [00:10<00:00,  2.17s/it]
<lightfm.lightfm.LightFM at 0x7ca41bcb2f20>

Evaluation

Evaluation metrics to consider:

• AUC : It measure the ROC AUC metric for a model: the probability that a randomly chosen positive example has a higher score than a randomly chosen negative example. A perfect score is 1.0.
• Precision at K : Measure the precision at k metric for a model: the fraction of known positives in the first k positions of the ranked list of results.A perfect score is 1.0.
• Recall at K : Measure the recall at k metric for a model: the number of positive items in the first k positions of the ranked list of results divided by the number of positive items in the test period. A perfect score is 1.0.
• Mean Reciprocal rank : Measure the reciprocal rank metric for a model: 1 / the rank of the highest ranked positive example. A perfect score is 1.0.

Here we’re going to use the preicision at k:

precision_train = precision_at_k(model, train, k=10, num_threads=4).mean()
precision_test = precision_at_k(model, test, k=10, num_threads=4).mean()
# recall_train = recall_at_k(model, train, k=10).mean()
# recall_test = recall_at_k(model, test, k=10).mean()

print(precision_train, precision_test)

0.17519069 0.024261154

Hyperparameter Tuning using Random Search

Taken from this blog post with adjustments to include or not the weights.

Side note: usually it’s better to perfom hyperparameter tuning on a validation dataset using k-folds.

def sample_hyperparameters():
    while True:
        yield {
            "no_components": np.random.randint(16, 64),
            "learning_schedule": np.random.choice(["adagrad", "adadelta"]),
            "loss": np.random.choice(["bpr", "warp", "warp-kos"]),
            "learning_rate": np.random.exponential(0.05),
            "item_alpha": np.random.exponential(1e-8),
            "user_alpha": np.random.exponential(1e-8),
            "max_sampled": np.random.randint(5, 15),
            "num_epochs": np.random.randint(5, 50),
        }


def random_search(train_interactions, test_interactions, num_samples=50, num_threads=1, weights=None):
    for hyperparams in itertools.islice(sample_hyperparameters(), num_samples):
        num_epochs = hyperparams.pop("num_epochs")

        model = LightFM(**hyperparams)
        model.fit(
            interactions,
            item_features=None,
            user_features=None,
            sample_weight=weights,
            epochs=num_epochs,
            num_threads=num_threads,
            verbose=True
        )

        score = precision_at_k(
            model=model,
            test_interactions=test,
            train_interactions=None,
            k=10,
            num_threads=num_threads
        ).mean()
        weights_ = "No" if weights is None else "Yes"
        print(f"score: {score:.4f}, weights: {weights_}, hyperparams: {hyperparams}")
        hyperparams["num_epochs"] = num_epochs
        yield (score, hyperparams, model)


optimized_dict={}

score, hyperparams, model = max(random_search(
    train_interactions = train,
    test_interactions = test,
    num_threads = 4
    ), key=lambda x: x[0])


print(f"WITHOUT WEIGHTS: best score {score} obtained with the following hyper parameters {hyperparams}")

with open(dir_path + 'model_without_weights.pkl', 'wb') as f:
    pickle.dump(model, f)

Epoch: 100%|██████████| 20/20 [01:16<00:00,  3.81s/it]


score: 0.1184, weights: No, hyperparams: {'no_components': 60, 'learning_schedule': 'adadelta', 'loss': 'warp-kos', 'learning_rate': 0.008489458350985983, 'item_alpha': 5.2786918388645496e-11, 'user_alpha': 3.5578101760199264e-08, 'max_sampled': 8}


Epoch: 100%|██████████| 15/15 [00:28<00:00,  1.91s/it]


score: 0.0157, weights: No, hyperparams: {'no_components': 26, 'learning_schedule': 'adagrad', 'loss': 'bpr', 'learning_rate': 0.011351236451160842, 'item_alpha': 2.3757447922172716e-09, 'user_alpha': 2.576221612240835e-08, 'max_sampled': 8}

[...]

Epoch: 100%|██████████| 44/44 [02:46<00:00,  3.79s/it]


score: 0.1304, weights: No, hyperparams: {'no_components': 54, 'learning_schedule': 'adadelta', 'loss': 'warp-kos', 'learning_rate': 0.009802596101768535, 'item_alpha': 1.1405022588283016e-08, 'user_alpha': 1.1784362838916816e-08, 'max_sampled': 11}


Epoch: 100%|██████████| 43/43 [01:38<00:00,  2.30s/it]


score: 0.0916, weights: No, hyperparams: {'no_components': 46, 'learning_schedule': 'adagrad', 'loss': 'bpr', 'learning_rate': 0.030843895830248953, 'item_alpha': 8.392851304125428e-09, 'user_alpha': 1.0179184395176035e-08, 'max_sampled': 13}
WITHOUT WEIGHTS: best score 0.13038481771945953 obtained with the following hyper parameters {'no_components': 54, 'learning_schedule': 'adadelta', 'loss': 'warp-kos', 'learning_rate': 0.009802596101768535, 'item_alpha': 1.1405022588283016e-08, 'user_alpha': 1.1784362838916816e-08, 'max_sampled': 11, 'num_epochs': 44}

So, without considering the weights (i.e the number of times an itmen is bought by the same customer), the best precision score 0.13 on the test set, obtained with the following hyper parameters:

• no_components: 54
• learning_schedule: ‘adadelta’
• learning_rate’: 0.0098
• item_alpha: 1.14e-08
• user_alpha: 1.17e-08
• max_sampled’: 11
• num_epochs: 44

Using Weights

Let’s try the same thing but this time with the weights:

# overriden because the k-OS loss with sample weights is not implemented.
def sample_hyperparameters():
    while True:
        yield {
            "no_components": np.random.randint(16, 64),
            "learning_schedule": np.random.choice(["adagrad", "adadelta"]),
            "loss": np.random.choice(["bpr", "warp"]), #, "warp-kos"]),
            "learning_rate": np.random.exponential(0.05),
            "item_alpha": np.random.exponential(1e-8),
            "user_alpha": np.random.exponential(1e-8),
            "max_sampled": np.random.randint(5, 15),
            "num_epochs": np.random.randint(5, 50),
        }

score_w, hyperparams_w, model_w = max(random_search(
    train_interactions = train,
    test_interactions = test,
    num_threads = 4,
    weights=weights,
    ), key=lambda x: x[0])


print(f"WITH WEIGHTS: best score {score_w} obtained with the following hyper parameters {hyperparams_w}")

with open(dir_path + 'model_with_weights.pkl', 'wb') as f:
    pickle.dump(model_w, f)

Epoch: 100%|██████████| 21/21 [00:36<00:00,  1.74s/it]


score: 0.0658, weights: Yes, hyperparams: {'no_components': 35, 'learning_schedule': 'adagrad', 'loss': 'warp', 'learning_rate': 0.03793522958313662, 'item_alpha': 1.526228884795337e-09, 'user_alpha': 1.4820473052576738e-08, 'max_sampled': 10}


Epoch: 100%|██████████| 41/41 [00:59<00:00,  1.44s/it]


score: 0.0558, weights: Yes, hyperparams: {'no_components': 35, 'learning_schedule': 'adagrad', 'loss': 'warp', 'learning_rate': 0.020399551978036737, 'item_alpha': 2.462376731983936e-08, 'user_alpha': 4.266936687426811e-09, 'max_sampled': 6}

[...]

Epoch: 100%|██████████| 27/27 [00:47<00:00,  1.74s/it]


score: 0.0537, weights: Yes, hyperparams: {'no_components': 19, 'learning_schedule': 'adadelta', 'loss': 'bpr', 'learning_rate': 0.08636897698001998, 'item_alpha': 4.230409864246658e-10, 'user_alpha': 3.649947540870082e-08, 'max_sampled': 7}


Epoch: 100%|██████████| 15/15 [00:31<00:00,  2.11s/it]


score: 0.0630, weights: Yes, hyperparams: {'no_components': 42, 'learning_schedule': 'adagrad', 'loss': 'bpr', 'learning_rate': 0.03273940087847153, 'item_alpha': 3.791919344957527e-09, 'user_alpha': 2.02930754839095e-08, 'max_sampled': 14}
WITH WEIGHTS: best score 0.11721404641866684 obtained with the following hyper parameters {'no_components': 62, 'learning_schedule': 'adadelta', 'loss': 'bpr', 'learning_rate': 0.005779964110324431, 'item_alpha': 5.8232841799619904e-09, 'user_alpha': 1.7112869692085306e-09, 'max_sampled': 10, 'num_epochs': 34}

The best precision score on the test set is 0.117, which unfortunately not better.

Predictions

To get recommendation for a particular customer, we can use the predict method and the different mappings, that way we can also have the score :

user_id_mapping, user_feature_mapping, item_id_mapping, item_feature_mapping = dataset.mapping()
n_users, n_items = df.customer_id.nunique(), df.article_id.nunique()


def get_top_k_recommendations_with_scores(customer_id, k=10):
  item_id_mapping_reverse = {v:k for k, v in item_id_mapping.items()}

  # the top recommendation is the item with the highest predict score, not the lowest.
  recommendation_scores_for_pairs = model.predict(user_id_mapping[customer_id], np.arange(n_items))
  recommendations = pd.DataFrame({"scores": recommendation_scores_for_pairs})
  recommendations["article_id"] = pd.Series(recommendations.index.values).apply(lambda x: item_id_mapping_reverse[x])
  recommendations = recommendations.merge(df_articles[["article_id", "prod_name", "product_type_name"]], on="article_id")

  display(recommendations.sort_values(by="scores", ascending=False).head(k))


get_top_k_recommendations_with_scores('c88e095d490d67ba66f57132759057247040570935ba21a447e64b782d20880c')

It also is possible to use the predict_rank method on a whole dataset:

ranks = model.predict_rank(
    test_interactions=test,
    train_interactions=None,
    item_features=None,
    user_features=None,
    num_threads=4,
    check_intersections=True
)

ranks_dense = ranks.todense()
assert ranks_dense.shape == (df.customer_id.nunique(), df.article_id.nunique())

References:

• The LightFM paper
• LightFM Hybrid Recommendation system
• The LightFM online documentation

Implementation of the “CCF: Fast and Scalable Connected Component Computation in MapReduce” paper with Spark. Study of its scalability on several datasets using various clusters’ sizes on Databricks and Google Cloud Platform (GCP)

Table of content

• Abstract
• Description of the CCF algorithm
  • Connected component definition
  • Global methodology
  • Differents steps - counting new pairs
• Spark Implementation
  • Spark Session and context
  • RDD & DataFrame
  • Explanation of each steps
• Scalability Analysis
  • Datasets
  • Computation with Databricks
  • Computation using Google Cloud Dataproc
  • Computation using the LAMSADE cluster
• Conclusion
• Appendix
• References

Abstract

A graph is a mathematical structure used to model pairwise relations between objects. It is made up of vertices (also called nodes or points) which are connected by edges (also called links or lines).
Many practical problems can be represented by graphs: they can be used to model many types of relations and processes in physical, biological, social and information systems. Finding connected components in a graph is a wellknown problem in a wide variety of application areas. For that purpose; in 2014, H. Kardes, S. Agrawal, X. Wang and A. Sun published “CCF: Fast and scalable connected component computation in MapReduce”. Hadoop MapReduce introduced a new paradigm: a programming model for processing big data sets in a parallel and in a distributed way on a cluster, it involves many read/write operations. On the contrary, by running as many operations as possible in-memory - few years later - Spark has proven to be much more faster and has become de-facto a new standard.
In this study, we explain the algorithm and main concepts behind CCF. Then we make a PySpark inplementatoin. And finally we analyze the scalability of our solution applied on datasets of increasing sizes. The computations are realised on a cluster also of an increasing number of nodes in order to see the evolution of the calculation time. We’ve used the Databricks community edition and Google Cloud Dataproc.

Description of the CCF algorithm

Connected component definition

First, let’s give a formal definition in graph theory context:

• V is the set of vertices
• and E is the set of edges
• G = (V,E) be an undirected graph

Properties

• C = (C1,C2, …,Cn) is the set of disjoint connected components in this graph
• (C1 U C2 U … U Cn) = V
• (C1 intersect C2 intersect … intersect Cn) = void.
• For each connected component Ci in C, there exists a path in G between any two vertices vk and vl where (vk, vl) in Ci.
• Additionally, for any distinct connected component (Ci,Cj) in C, there is no path between any pair vk and vl where vk in Ci, vl in Cj.

Thus, problem of finding all connected components in a graph is finding the C satisfying the above conditions.

Global methodology

Here is how CCF works:

• it takes as input a list of all the edges.
• it returns as an output the mapping (i.e a table) from each node to its corresponding componentID (i.e the smallest node id in each connected component it belongs to)

To this end, the following chain of operations take place:

Two jobs run iteratively till we don’t find any new connected peer attached to the existing components:

• CCF-Iterate

This job generates an adjacency lists AL = (a1, a2, …, an) for each node v i.e the list of new nodes belonging to the same connected component. Each time, the node id is eventually updated in case of a new peer node with an id that become the new minimum.
If there is only one node in AL, it means we will generate the pair that we have in previous iteration. However, if there is more than one node in AL, it means that the process is not completed yet: an other iteration is needed to find other peers.

• CCF-Dedup

During the CCF-Iterate job, the same pair might be emitted multiple times. The second job, CCF-Dedup, just deduplicates the output of the CCF-Iterate job in order to improve the algorithm’s efficiency.

Differents steps - counting new pairs

Let’s break the whole process piece by piece using the example illustrated below:

• For each edge, the CCT-Iterate mapper emits both (k, v) and (v, k) pairs so that a should be in the adjacency list of b and vice versa.
• In reduce phase, all the adjacent nodes are grouped together –> pairs are sorted by keys
• All the values are parsed group by group:
  • a first time to find the the minValue
  • a second time to emit a new componentID if needed
• The global NewPair counter is initialized to 0. For each component if a new peer node is found, the counter is incremented. At the end of the job, if the NewPair is still 0: it means that there is not any new edge that can be attached to the existing components: the whole computation task is over. Otherwise an other iteration is needed.

Then we just have to calculate the number of connected components by counting the distinct componentIDs.

Spark Implementation

Spark Session and context

First we create a Spark Session:

Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (called the driver program).

Spark Driver manage the whole application. It decides what part of job will be done on which Executor and also gets the information from Executors about task statuses.

Since earlier versions of Spark or Pyspark, SparkContext (JavaSparkContext for Java) is an entry point to Spark programming with RDD and to connect to Spark Cluster, Since Spark 2.0 SparkSession has been introduced and became an entry point to start programming with DataFrame and Dataset.

An accumulator is used as an excrementing variable to count new pairs:

# initialize nb_new_pair as a spark accumulator
nb_new_pair = sc.accumulator(0)

[...]

def iterate_reduce_df(df):
    [...]
    nb_new_pair += df.withColumn("count", size("v")-1).select(sum("count")).collect()[0][0]
    [...]
...

Accumulators are created at driver program by calling Spark context object. Then accumulators objects are passed along with other serialized tasks code to distributed executors. Task code updates accumulator values. Then Spark sends accumulators back to driver program, merges their values obtained from multuple tasks, and here we can use accumulators for whatever purpose (e.g. reporting). Important moment is that accumulators become accessible to driver code once processing stage is complete.

We use the SparkSession to load the dataset into a DataFrame and the SparkContext to use it with RDD.

As you can see the first few lines of headers starting with ‘#’ are not well interpreted, and so is the separator ‘\t’:

rdd_raw = load_rdd(path)
rdd_raw.take(6)

['# bla bla', '# header', '1\t2', '2\t3', '2\t4', '4\t5']

The same issue occurs for with a dataframe:

df_raw = load_df(path)
df_raw.show(6)

+---------+
|      _c0|
+---------+
|# bla bla|
| # header|
|      1	2|
|      2	3|
|      2	4|
|      4	5|
+---------+
only showing top 6 rows

RDD & DataFrame

The RDDs are defined as the distributed collection of the data elements without any schema operating at low level. The Dataframes are defined as the distributed collection organized into named columns with a schema (but without being strongly typed like the Datasets).

Here is a more exhaustive lists of the differences:

They are considered “resilient” because the whole lineage of data transformations can be rebuild from the DAG if we loose an executor for instance.

Before computation of connected components we prepare the datasets by removing multiline headers and split the two columns separated by a tabulation:

We also cast the informations to integers to get a clean list of (keys, values) in the RDD:

rdd = preprocess_rdd(rdd_raw)
rdd.take(10)

[(1, 2), (2, 3), (2, 4), (4, 5), (6, 7), (7, 8)]

and a ready-to-use table in a Dataframe:

df = preprocess_df(df_raw)
df.show(10)

+---+---+
|  k|  v|
+---+---+
|  1|  2|
|  2|  3|
|  2|  4|
|  4|  5|
|  6|  7|
|  7|  8|
+---+---+

Explanation of each steps

The mapper & reducer jobs illustrated in the picture seen previously (see “Differents steps - counting new pairs”) correspond to the first iteration of the following graph :

For the sake of clarity, we are going to replace the vertices A by 1, B by 2 and so on… And for each steps, let’s see both the RDD and DataFrame outputs. The computation part starts with the “iterate map” function, its goal is to generate an exhaustive list of edges:

The way to proceed is the same for RDDs or Dataframe: .union is used to concatenate the original RDD or DF with the inverted one. The reversal is achieved by mapping keys / values in a different order for RDDs : rdd.map(lambda x : (x[1], x[0])) et by selecting the columns in a different order for DFs : df.select(col("v").alias("k"), col("k").alias("v")) alias allows us to rename properly columns’ names:

rdd = iterate_map_rdd(rdd)
rdd.take(20)

[(1, 2),
 (2, 3),
 (2, 4),
 (4, 5),
 (6, 7),
 (7, 8),
 (2, 1),
 (3, 2),
 (4, 2),
 (5, 4),
 (7, 6),
 (8, 7)]

The output is quite similar with a Dataframe but with named columns:

df = iterate_map_df(df)
df.show(20)

+---+---+
|  k|  v|
+---+---+
|  1|  2|
|  2|  3|
|  2|  4|
|  4|  5|
|  6|  7|
|  7|  8|
|  2|  1|
|  3|  2|
|  4|  2|
|  5|  4|
|  7|  6|
|  8|  7|
+---+---+

CCF-Iterate job generates adjacency lists AL = (a-1, a-2, …, a-n) for each node v, and if the node id of this node v-id is larger than the min node id a-min in the adjacancy list, it first creates a pair (v-id, a-min) and then a pair for each (a-i, a-min) where a-i 2 AL, and a-i 6 = amin. If there is only one node in AL, it means we will generate the pair that we have in previous iteration.

However, if there is more than one node in AL, it means we might generate a pair that we didn’t have in the previous iteration, and one more iteration is needed.

Nota: if v-id is smaller than a-min, we do not emit any pair.
The pseudo code of CCF-Iterate provided in the original paper is quite similar to the count_nb_new_pair() implementation. We use this function in conjuction of a .groupByKey() and a .flatMap applied on RDDs:

With a dataframe, the main concept remains the same. But the way to count new pairs is a little bit different. We aggregate rows and group them by the column k for key in our case. We determine the min of the values with array_min("v"). Then we sum the count obtained with size("v"):

rdd = iterate_reduce_rdd(rdd)
rdd.take(16)

[(2, 1),
 (3, 1),
 (3, 2),
 (4, 2),
 (4, 1),
 (5, 2),
 (5, 4),
 (7, 6),
 (8, 7),
 (8, 6)]

Similar results with dataframes:

df = iterate_reduce_df(df)
df.show()

+---+---+
|  k|  v|
+---+---+
|  2|  3|
|  4|  5|
|  2|  4|
|  2|  5|
|  6|  7|
|  6|  8|
|  7|  8|
|  1|  2|
|  1|  3|
|  1|  4|
+---+---+

The compute_cc_rdd and compute_cc_df are exactly the same.

• the number of iteration is initialized to zero
• a while loop takes place: if the number of pair at the beginning of the iteration remains unchanged at the end, we break in order to stop the loop. This is done with the condition if start_pair == nb_new_pair.value:. More precisely, the accumulator nb_new_pairis used as an incremental variable. If we want to compute the real number of new pairs, we have to set the values back to zero at the beginning of each iteration and check if the value is not nul at the end.
• as explained in the CCF algorithm paper, the jobs in the loop are:
  • iterate_map
  • iterate_reduce
  • and iterate_dedup (the deduplication is simply achieved using rdd.distinct() or df.distinct())

Finally, the workflow_rdd() and workflow_df() functions are just wrappers containing all the previously seen functions:

• the dataset loading with load_rdd(path) and load_df(path)
• its preprocessing with preprocess_rdd(df_raw) and preprocess_df(df_raw)
• then the timer is started: start_time = time.time()
• after that, the computation of the connected components is launched df = compute_cc_df(df) / rdd = compute_cc_rdd(df)
• we print the number of connected components in the graph equal to df.select('k').distinct().count() (same code for RDDs
• and finally, we dispaly the duration in seconds which is equal to the delta: time.time() - start_time

Side note:

Here we don’t include in the timer the reading of the dataset. This time of reading can be decreased with more nodes in the clusters because of the redundancy of the data / distributed storage according to the Hadoop / HDFS replication factor (by default 3). But this is actually not what we’re interested in. On the contrary, the computed number of distinct components should be counted in the duration, because it is our final objective.

Scalability Analysis

We use datasets and Hadoop clusters with Spark both of increasing sizes.

Datasets

Source: Stanford Large Network Dataset Collection web site

Datasets information

Nodes represent pages and directed edges represent hyperlinks between them for

• Stanford University (stanford.edu)
• University of Notre Dame (domain nd.edu)
• Berkely.edu and Stanford.edu domains
• Web pages released in by Google as a part of Google Programming Contest.

A for loop in the main function parse all the datasets one by one, and for each dataset the CC are computed using RDDs then Dataframes:

Computation with Databricks

• we create a cluster with the following settings available for community edition : 1 Driver: 15.3 GB Memory, 2 Cores, 1 DBU (A Databricks Unit is a normalized unit of processing power on the Databricks Lakehouse Platform used for measurement and pricing purposes. The number of DBUs a workload consumes is driven by processing metrics, which may include the compute resources used and the amount of data processed)

• then we create a table on this cluster and upload our datasets

• finally, we run our script in a notebook

The python script is identical to the one used on GCP: only the path of the various datasets were

Computation using Google Cloud Dataproc

Cluster creation

• create buckets with your input data and scripts
• enable Dataproc API
• create a cluster with the following settings:
  • from the web UI with the options below:
    • Enable component gateway
    • Jupyter Notebook
    • the nodes configurations
    • a scheduled deletion of the cluster after an idle time period without submitted jobs
    • Allow API access to all Google Cloud services in the same project.
  • or in command line:

gcloud dataproc clusters create node-2 \
    --enable-component-gateway \
    --region us-central1 \
    --zone us-central1-c \
    --master-machine-type n1-standard-2 \
    --master-boot-disk-size 500 \
    --num-workers 2 \
    --worker-machine-type n1-standard-4  \
    --worker-boot-disk-size 500  \
    --image-version 2.0-debian10  \
    --optional-components JUPYTER  \
    --max-idle 3600s  \
    --scopes 'https://www.googleapis.com/auth/cloud-platform'  \ 
    --project iasd4-364813

3 ways to launch a job

• notebook (for the developpement / analysis part)

• Web UI

we can add properties to specify the excecutor’s memory & cores number (but it seems that we need to have 1 executor per node otherwise we encounter OOM especially for dataset “BerkStan”):

• the GCP equivalent spark submit command where shell variable starting with $ must be initialized and the path_main.pychanged by the URL of the script in the GCS’ bucket: gs://iasd-input-data/compute_CCF_with_RDD_and_DF.py

gcloud dataproc jobs submit pyspark path_main.py \
    --cluster=$CLUSTER_NAME \
    --region=$REGION \
    --properties="spark.submit.deployMode"="cluster",\
    "spark.dynamicAllocation.enabled"="true",\
    "spark.shuffle.service.enabled"="true",\
    "spark.executor.memory"="15g",\
    "spark.driver.memory"="16g",\
    "spark.executor.cores"="5"

Once a job is submitted, we check its status in Yarn application manager

and get into the spark’s details (stages, jobs…)

Computation using the LAMSADE cluster

We add our private ssh key and connect to the master node of the LAMSADE’s hadoop cluster. With the linux command line, the archives are directly downloaded using wget, then unzip and putin HDFS:

This cluster comes with Spark version 3.2.0 as showned in the spark-shell:

The previous script used with Google Cloud Platform is slightly modified:

• the datasets’ paths are changed with the Hadoop Distributed File System
• the pattern “###” is used before prints in order to easily filter with grep the relevent informations in the spark-submitresults:

Here is the command used to launch jobs:

/opt/cephfs/shared/spark-3.2.0-bin-hadoop2.7/bin/spark-submit \
	--master spark://vmhadoopmaster.cluster.lamsade.dauphine.fr:7077 \
	./compute_CCF_with_RDD_and_DF.py &> results.txt

spark-submit prints most of it’s output to STDERR. To redirect the entire output to one file, you can use:

spark-submit something.py > results.txt 2>&1
# or
spark-submit something.py &> results.txt

Side note:
We haven’t found the exact number of slaves nodes in the cluster configuration (core-site.xml): the hosts files and some hdfs commands are not accessibles to us with our account / low rights (for security reasons). But it seems that the whole cluster is virtualized on a single big VM with 9 slaves for spark:

obrunet@vmhadoopmaster:/opt/cephfs/shared/spark-3.2.0-bin-hadoop2.7/conf$ cat slaves
vmhadoopslave1
vmhadoopslave2
vmhadoopslave3
vmhadoopslave4
vmhadoopslave5
vmhadoopslave6
vmhadoopslave7
vmhadoopslave8
vmhadoopslave9

Conclusion

Recap of the clusters used:

Summary of the calculation times in seconds for both resilient distributed datasets and dataframes (rdd / df):

(*) lasts too long: result not recorded
(**) the web-BerkStan is finally a single connected component with too many edges with regard to the number of vertices (unfortunately this couldn’t be guessed before the end of the computation!): it causes out of memory errors occuring in executors for the GCP Dataproc and no space left on the device problems for the driver of the LAMSADE cluster.

Comparison of results

Comments about this experimental analysis

• Even if RDDs operate at a lower level - our first intuition was that computation with RDDs will take less time - this is not always true. In the table above there are few cases where using DFs lead to faster results.
• The Databricks community edition use a single VM, while the Google Cloud involve 2 worker nodes and the LAMSADE cluster 9 slaves: we can clearly see that with more nodes, the computation takes less time (whether with RDDs or DFs). The parallelism of operations such as map, filter or reduce… significantly improves performances.
• One can notice that if you run the same script twice in the same conditions i.e on the same dataset with the same cluster, you can get slightly different results: this can be a consequence of
  • shared ressources for the LAMSADE cluster
  • we could assume the way partitions are dispatched can have an impact: if the same connected component is computed on several nodes because each slave is dealing with different vertices (of the same component), it will take more time than after a shuffle. So graph with very few connected components might be affected (see the web-BerkStan dataset)

Strenghts and weaknesses of the algorithm

• It makes use of a loop, which is not in accordance with the distributed processing paradigm but to our knowledge we can’t proceed without any loop.
• It is not suited for “highly” connected graphs with a high ratio of vertices compared to edges.
• As stated previously, the computation time is influenced by the way the data / partitions are splitted among nodes of the cluster. The same run can lead to a different number of iterations, the lack of reproductibility could also be an issue.
• Nevertheless, finding connected components in graph remains a complex task, the CCF algorithm has proven to be a robust solution, further more a solution that can be parallelized.

Alternative solutions

One also might consider using the spark’s graphx librairy (not maintained anymore & only in Scala) or graph databases such as Neo4J or specific librairies like NetworkX.

Appendix

• PySpark Script run on GCP Dataproc
• Notebook with the explanations in depth
• Databricks notebook - online & local archive
• PySpark Script run on the LAMSADE cluster

References

Paper

• “CCF: Fast and scalable connected component computation in MapReduce”
• Local archive

Datasets

• Stanford Large Network Dataset Collection web site
  • Web graph of Berkeley and Stanford
  • Web graph from Google
  • Web graph of Notre Dame
  • Web graph of Stanford.edu

Documentation

• Latest version of Spark
• Databricks community edition
• Google Cloud Dataproc

• Introduction
• How FL works? (Technical overview)
• Different types of FL
• Open source frameworks
• Applications / Examples
• Challenges and considerations
• Final thoughts
• References

Introduction

Existing data is not fully exploited primarily because it sits in silos and privacy concerns restrict its access.

Even if data anonymisation could bypass some limitations, removing metadata or PII is often not enough to preserve privacy. Another reason why data sharing is not systematic is that collecting, curating, and maintaining high-quality datasets takes considerable time, effort, and expense. Consequently such data sets may have significant business value, making it less likely that they will be freely shared. Furthermore, with creativity and “thinking out of the box” one can invent new use cases from your data…

Centralising or releasing data, however, poses not only regulatory, ethical and legal challenges, related to privacy and data protection, but also technical ones. Controlling access and safely transferring it is a non-trivial, and sometimes impossible task.

“[…] Collaborative learning without centralising data or the promise of federated efforts…”

Federated learning (FL) is a learning paradigm seeking to address the problems of data governance and privacy by training algorithms collaboratively without exchanging the data itself.

Originally developed for different domains, such as mobile and edge device use cases, it recently gained attention for other applications.

How FL works? (Technical overview)

ML algorithms are hungry for data. Furthermore, the real-world performance of your ML model depends not only on the amount of data but also the relevance of the training data.

The ML process occurs locally at each participating institution and only model characteristics (e.g., parameters, gradients) are transferred. Let’s see how it works step by step:

1. Choose a model (either pre-trained or not) on the central server.
2. Distribute this initial model to the clients (devices / local servers).
3. Each client keeps “confidentially” training it on-site using its own local data.
4. Then the locally trained models are sent back to the central server via encrypted communication channels. The updates from all clients are averaged / aggregated into a single shared improved model.
5. Finally, this model is sent back to all devices & servers.

The cool part is that the training process in federated learning is iterative. This means the server and clients can send the updated parameters back and forth multiple times, enabling a continuous share of knowledge among participants.

Here is an other illustration on how it works:

For IOT or edge computing, the process breakdown is quite similar & can be summarized by the picture below:

Your phone personalizes the model locally, based on your usage ( A ). Many users’ updates are aggregated ( B ) to form a consensus change ( C ) to the shared model, after which the procedure is repeated.

An important note is that the training data still remains on the user’s device; only the training result is encrypted and sent to the cloud. This collaborative way of training and developing machine learning models is powerful and has real-world applications.

“[…] only the model updates and parameters are shared, not the training data itself….”

Recent research has shown that models trained by FL can achieve performance levels comparable to ones trained on centrally hosted data sets and superior to models that only see isolated single-institutional data.

How the model aggregation is actually done?

The global loss function is obtained via a weighted combination of K local losses computed from private data Xk, which is residing at the individual involved parties and never shared among them:

where wk > 0 denote the respective weight coefficients.

In practice, each participant typically obtains and refines a global consensus model by conducting a few rounds of optimisation locally and before sharing updates, either directly or via a parameter server. The more rounds of local training are performed, the less it is guaranteed that the overall procedure is minimising. The actual process for aggregating parameters depends on the network topology.

Nota: that aggregation strategies do not necessarily require information about the full model update

Different types of FL

FL solutions can be classified according to the way they allow data to be aggregated, according to the network topology and finally how different datasets are used.

Broadly, there are three types of gad bridge:

• Intra-company: Bridging internal data silos.
• Inter-company: Facilitating collaboration between organizations.
• Edge computing: Learning across thousands of edge devices.

FL design choices / Topologies

Different communication architectures:

• (a) Centralised: the aggregation server coordinates the training iterations and collects, aggregates and distributes the models to and from the Training Nodes (Hub & Spoke).
• (b) Decentralised: each training node is connected to one or more peers and aggregation occurs on each node in parallel.
• (c) Hierarchical: federated networks can be composed from several sub-federations, which can be built from a mix of Peer to Peer and Aggregation Server federations (d). FL compute plans—trajectory of a model across several partners.
• (e) Sequential training/cyclic transfer learning.
• (f) Aggregation server,
• (g) Peer to Peer.

Vertical & Horizontal FL

This distributed, decentralized training process comes in three flavors. In horizontal federated learning, the central model is trained on similar datasets. In vertical federated learning, the data are complementary; movie and book reviews, for example, are combined to predict someone’s music preferences. Finally, in federated transfer learning, a pre-trained foundation model designed to perform one general task, is trained on another dataset to do something else to repurpose it.

Let’s take the example of two banks from the same country. Although they have non-overlapping clientele, their data will have similar feature spaces since they have very similar business models. They might come together to collaborate in an example of horizontal federated learning.

“[…] Horizontal Fl (or sample-based FL / homogenous FL), is introduced in the scenarios that data sets share the same feature space but different in samples.”

In vertical federated learning, two companies providing different services (e.g. banking and e-commerce) but having a large intersection of clientele might find room to collaborate on the different feature spaces they own, leading to better outcomes for both.

“[…] Vertical FL (or feature-based FL) is applicable to the cases that two data sets share the same sample ID space but differ in feature space.”

Federated Transfer Learning: To my knowledge, there isn’t any concrete production implementation yet, because transfer learning is already used widely without the need of decentralization, but one might consider the advantage of more datas…

Open source frameworks

• FATE
• Substra
• OpenMined’s PyGrid & PySyft
• OpenFL
• TensorFlow Federated
• Flower: A Friendly FL Framework
• IBM Federated Learning
• NVIDIA Clara
• Enterprise-grade Federated Learning Platforms

I haven’t tested those frameworks myself, so it wouldn’t be possible to create an objective benchmark. Moreover things are moving rapidly… But it’s obviously in the interest of cloud providers to make such frameworks robust and secure in order to create an hybrid architecture able to comminucate with on-premise nodes.

Applications / Examples

• FL was first introduced by Google in 2017 to improve text prediction in mobile keyboard
• Due to Covid19, one of the biggest real-world federated collaborations, involved 20 distinct hospitals from five different continents training an AI model to forecast the oxygen requirements of patients with COVID-19.

There are many more domains of interest such as robotics, finance (credit scoring, risk prediction & prevention, get a user’s digital footprint that will help to prevent fraud, KYC without transferring data to the cloud), recommender systems, and call center analytics.

Taxonomy for applications of federated learning across different domains and sub-domains:

Challenges & considerations

The clients involved in federated learning may be unreliable as they are subject to more failures or drop out since they commonly rely on less powerful communication media (i.e. Wi-Fi) and battery-powered systems (i.e. smartphones and IoT devices)

Same old issues

Despite the advantages of FL, it does not solve all issues: particularly the same data quality problems

Expensive Communication:

Federated networks are potentially comprised of a massive number of devices (e.g., millions of smart phones), and communication in the network can be slower than local computation by many orders of magnitude. Communication in such networks can be much more expensive than that in classical data center environments. In order to fit a model to data generated by the devices in a federated network, it is therefore necessary to develop communication-efficient methods that iteratively send small messages or model updates as part of the training process, as opposed to sending the entire dataset over the network.

Data heterogeneity

Inhomogeneous data distribution poses a challenge for FL algorithms and strategies, as many are assuming IID (independent & non-identically distributed): the assumption of iid variables is central to many statistical methods & algorithms and can add complexity or cause problems to the model.

Leakage or attacks

FL can indirectly expose private data used for local training, e.g., by model inversion of the model updates, the gradients themselves or adversarial attacks

Through reverse engineering, it’s still possible to identify and obtain the data from a specific user. However, privacy techniques such as differential privacy can strengthen the privacy of federated learning but at the cost of lower model accuracy.

Traceability and accountability

Traceability of all system assets including data access history, training configurations, and hyperparameter tuning throughout the training processes is thus mandatory.

It may also be helpful to measure the amount of contribution from each participant, such as computational resources consumed, quality of the data used for local training, etc.

Systems Heterogeneity

The storage, computational, and communication capabilities of each device in federated networks may differ due to variability in hardware (CPU, memory), network connectivity (3G, 4G, 5G, wifi), and power (battery level). Additionally, the network size and systems-related constraints on each device typically result in only a small fraction of the devices being active at once. For example, only hundreds of devices may be active in a million-device network. Each device may also be unreliable, and it is not uncommon for an active device to drop out at a given iteration. These system-level characteristics make issues such as stragglers and fault tolerance significantly more prevalent than in typical data center environments. There will be a tradeoff between maintaining the performance of the device and model accuracy.

Final thoughts

FL has many benefits as it enhances user data privacy, its adherence with regulatory compliance, and its ability to break silos, but on the other had there are still many technical limitations, and there is a lot of work for competing companies to be on the same page in the way to create and use the same federated network…

Despite the fact that FL requires rigorous technical consideration to ensure that the algorithm is proceeding optimally without compromising safety or patient privacy. Nevertheless, it has the potential to overcome some limitations of approaches that require a single pool of centralised data.

A still open question at the moment is how inferior models learned through federated data are relative to ones where the data are pooled. Another open question concerns the trustworthiness of the edge devices and the impact of malicious actors on the learned model.

“Keeping data private is the major value addition of FL for each of the participating entities to achieve a common goal.”

FL enables collaborative research for, even competing, companies, it might give birth to a new data ecosystem and create data alliances.

“FL can be paired with other privacy-preserving techniques like differential privacy to add noise and homomorphic encryption to encrypt model updates and obscure what the central server sees.”

Credits & References:

As previously said, this post is an aggregation of multiples sources, it is hard to give credits precisely for each part, it is widely inspired by the Nature presentation “The future of digital health with federated learning - Nature” and by this Wikipedia webpage. I would like to warmly thanks the authors. Credits should also be given for the following references:

General blog posts:

• Introduction to Federated Learning - Medium
• Federated machine learning for fintech. - Medium
• What is federated learning? - IBM
• Understanding the types of FL
• Federated Learning — Decentralized ML
• Federated Learning: The Shift from Centralized to Distributed On-Device Model Training
• A brief introduction to Federated Learning — FL Series Part 1

Technical papers / insights:

• Using Federated Learning to Bridge Data Silos in Financial Services - Nvidia
• Top 7 Open-Source Frameworks for Federated Learning - Apheris
• A Secure Federated Transfer LearningFramework
• Federated Learning: Challenges, Methods, and Future Directions

FL applications:

• Federated Learning: A New AI Business Model - Medium
• Federated learning for predicting clinical outcomes in patients with COVID-19

Misc :)

• Online comic by Google AI

why sharing data?

quasi indentifiers • What is a quasi-idenWfier? – CombinaWon of adributes (that an adversary may know) that uniquely idenWfy a large fracWon of the populaWon. – There can be many sets of quasi-idenWfiers. If Q = {B, Z, S} is a quasi-idenWfier, then Q + {N} is also a quasi-idenWfier.

We saw examples before …

• Massachuseds governor adack • AOL privacy breach • Neglix adack • Social Network adacks

K-anonymity

Each released record should be indistinguishable from at least (k-1) others on its QI attributes • Alternatively: cardinality of any query result on released data should be at least k • k-anonymity is (the first) one of many privacy definitions in this line of work – l-diversity, t-closeness, m-invariance, delta-presence…

![image info](./img/1.jpg

coarsening ![image info](./img/2.jpg

clusering ![image info](./img/3.jpg

micro agg ![image info](./img/4.jpg

Joining the published data to an external dataset using quasiidenWfiers results in at least k records per quasi-idenWfier combinaWon. – Need to guarantee k-anonymity against the largest set of quasi-idenWfiers

Does k-Anonymity guarantee sufficient privacy ?

homo attck

![image info](./img/5.jpg

background

![image info](./img/6.jpg ![image info](./img/7.jpg ![image info](./img/8.jpg

L-Diversity: Privacy Beyond K-Anonymity

L-diversity principle: A q-block is l-diverse if contains at least l ‘well represented” values for the sensitive attribute S. A table is l-diverse if every q-block is l-diverse

L-Diversity Principle: Every group of tuples with the same Q-ID values has ≥ L disInct “well represented” sensiIve values. Ques%ons: • What kind of adversarial adacks do we guard against? • Why is this the right definiWon for privacy? – What does the parameter L signify?

![image info](./img/9.jpg

Privacy SpecificaWon for L-Diversity • The link between idenWty and adribute value is the sensiWve informaWon. “Does Bob have Cancer? Heart disease? Flu?” “Does Umeko have Cancer? Heart disease? Flu?” • Adversary knows ≤ L-2 negaWon statements. “Umeko does not have Heart Disease.” – Data Publisher may not know exact adversarial knowledge • Privacy is breached when idenWty can be linked to adribute value with high probability Pr[ “Bob has Cancer” | published table, adv. knowledge] > t

Therefore, in order for privacy, check for each individual u, and each disease s Pr[ “u has disease s” | T*, adv. knowledge about u] < t And we are done … ?? 25 Data publisher does not know the adversary’s knowledge about u • Different adversaries have varying amounts of knowledge. • Adversary may have different knowledge about different individuals. adv.

Distinct l-diversity – Each equivalence class has at least l well-represented sensitive values – Limitation: • Doesn’t prevent the probabilistic inference attacks • Ex. In one equivalent class, there are ten tuples. In the “Disease” area, one of them is “Cancer”, one is “Heart Disease” and the remaining eight are “Flu”. This satisfies 3-diversity, but the attacker can still affirm that the target person’s disease is “Flu” with the accuracy of 80%.

Entropy l-diversity – Each equivalence class not only must have enough different sensitive values, but also the different sensitive values must be distributed evenly enough. – It means the entropy of the distribution of sensitive values in each equivalence class is at least log(l) – Sometimes this maybe too restrictive. When some values are very common, the entropy of the entire table may be very low. This leads to the less conservative notion of ldiversity.

Recursive (c,l)-diversity – The most frequent value does not appear too frequently – r1 <c(rl +rl+1 +…+rm )

Limitations

l-diversity may be difficult and unnecessary to achieve.  A single sensitive attribute  Two values: HIV positive (1%) and HIV negative (99%)  Very different degrees of sensitivity  l-diversity is unnecessary to achieve  2-diversity is unnecessary for an equivalence class that contains only negative records  l-diversity is difficult to achieve  Suppose there are 10000 records in total  To have distinct 2-diversity, there can be at most 10000*1%=100 equivalence classes

L-Diversity:

Guarding against unknown adversarial knowledge. Limit adversarial knowledge – Knows ≤ (L-2) negaWon statements of the form “Umeko does not have a Heart disease.” • Consider the worst case – Consider all possible conjuncWons of ≤ (L-2) statements

T-Closeness

k-anonymity prevents identity disclosure but not attribute disclosure • To solve that problem l-diversity requires that each eq. class has at least l values for each sensitive attribute • But l-diversity has some limitations • t-closeness requires that the distribution of a sensitive attribute in any eq. class is close to the distribution of a sensitive attribute in the overall table.

Privacy is measured by the information gain of an observer. – Information Gain = Posterior Belief – Prior Belief – Q = the distribution of the sensitive attribute in the whole table – P = the distribution of the sensitive attribute in eq. class

Principle: – An equivalence class is said to have t-closeness • if the distance between the distribution of a sensitive attribute in this class and the distribution of the attribute in the whole table is no more than a threshold t. – A table is said to have t-closeness • if all equivalence classes have t-closeness.

Conclusion • t-closeness protects against attribute disclosure but not identity disclosure • t-closeness requires that the distribution of a sensitive attribute in any eq. class is close to the distribution of a sensitive attribute in the overall table.

![image info](./img/4.jpg

![image info](./img/4.jpg

![image info](./img/4.jpg

![image info](./img/4.jpg

![image info](./img/4.jpg

References:

• Other Privacy Definitions: l-diversity and t-closeness by Murat Kantarcioglu
• Measures of Anonymity/Privacy: k-Anonymity, L-Diversity, t-Closeness by Ashwin Machanavajjhala (Duke University)
• Programming Differential Privacy - Website & Github repo
• K-anonymity, the parent of all privacy definitions by Damien Desfontaines

INTRODUCTION

Objective

The goal of this project is to train several neural networks for playing the game of Go, and to find the most performant architecture. This challenge was proposed by Professor Tristan Cazenave (LAMSADE & Chair AI Advanced of PRAIRIE) during his Deep Learning course, part of the Master IASD at Paris-Dauphine - PSL.

Game of Go

Go is an abstract strategy board game for two players in which the aim is to surround more territory than the opponent. The game was invented in China more than 2,500 years ago and is believed to be the oldest board game continuously played to the present day. […] There are over […] 20 million current players, the majority of whom live in East Asia. [01]

Rules:
The playing pieces are called stones. One player uses the white stones and the other black. The players take turns placing the stones on the vacant intersections (points) on a board. Once placed on the board, stones may not be moved, but stones are removed from the board if the stone (or group of stones) is surrounded by opposing stones on all orthogonally adjacent points, in which case the stone or group is captured. The game proceeds until neither player wishes to make another move. When a game concludes, the winner is determined by counting each player’s surrounded territory along with captured stones and komi (points added to the score of the player with the white stones as compensation for playing second). Games may also be terminated by resignation.

AlphaGo versus Lee Sedol

In March 2016, Seoul, South Korea, a match took place between AlphaGo and Lee Sedol. AlphaGo was developed by Google DeepMind, it uses a combination of machine learning and tree search techniques, combined with extensive training, both from human and computer play. Lee Sedol is widely considered to be one of the greatest player of the past decade, he has won 18 world titles.

Contrary to expectations, AlphaGo played a number of highly innovative moves which contradicted centuries of received Go knowledge and finally win 4-1 [02]. Later, several improvements were made based on reinforcement learning models: AlphaGo Zero, AlphaZero…

The Dataset

The data used for training comes from the Katago program self played games. KataGo is one of the strongest open-source self-play-trained Go engine, with many improvements to accelerate learning. The dataset we’re going to use is built of 1,000,000 different games in total for the training split (depending on a random state i.e. one random board position). The games of the validation set are never used during training.

KataGo plays Go at a much better level than ELF and Leela, therefore, the networks are trained with better data and provides better results. The input data is composed of 31 19x19 planes (color to play, ladders, current state on two planes, two previous states on multiple planes).

Rules

• In order to be fair about training ressources, the number of networks’ parameters must be lower than 100 000.
• It should use Tensorflow 2.9 & Keras.
• An example of a CNN with two heads is provided in file golois.py (see next chapter).
• The trained networks should also have the same policy & value heads and be saved in h5 format.

Tournaments:

• Frequently, challengers have to upload their models, and a tournament is organized between them.
• It is a round robin tournament: each contestant meets every other participant. It then return the results / rankings.
• Each network will be used by a PUCT engine that takes 2 seconds of CPU time at each move to play in the tournament.

Objectives

Training deep neural networks and performing tree search are the two pillars of current board games programs. This is also true for the Game of Go. Using a network with two heads instead of two networks was reported to bring significant ELO improvement, the choice of the NN architecture can also lead to many ELO improvement.

Output / Metrics:

• The policy (a vector of size 361 with 1.0 for the move played, 0.0 for the other moves) head is evaluated with the accuracy of predicting the moves of the games
• The value (between 0.0 and 1.0 given by the Monte-Carlo Tree Search representing the probability for White to win) head is evaluated with the Mean Squared Error (MSE) on the predictions of the outcomes of the games.

BASELINE - USUAL CNN

Please refer to the notebook attached named v01_BASELINE_CNN_more_epochs_dataviz.ipynb, comments for most of the lines of code are provided to ease understanding.

Explanation of the code

Constants:

• planes = 31 # each plane corresponds to the properties of a specific game of go
• moves = 361 # number of possible moves in the board (= 19 x 19)
• N = 10000 # number of situations considered in a game

That’s why input_data.shape = (10000, 19, 19, 31)

Variables:

• policy: best shot / move to 1 (–> accurary)
• value: probability for the white to win (between 0 and 1)
• end: position at the end of the game
• group: put connected stones together.

Train & valiation:
At each epoch input_data, value, end, groups are changed:

• first by the getBatch function in order to get data for the training part (see the model.fit(…)) in a deterministic way (reproducible)
• then by the getValidation function to assess the model performance on a validation data set (see the model.evaluate(…)). Obviously getValidation will return data that the model hasn’t seen. A shuffle is made at the beginning, so that the timeline is not preserved. Consequently, the RNN architecture (LSTM, GRU and so on…) is not adequate.

CNN architecture:

Classically, CNN for Go have more than one head. At least, these networks use a policy head, to prescribe moves, and a value head, to evaluate the board quality in terms of future incomes. This output configuration has been popularized by the groundbreaking AlphaZero. 07

The baseline model uses 5 common convolution layers. Then each of the two heads (for the value or the policy) are composed of a convolution layer, a flatten one and - for the value - 2 denses layers:

then the compilation is done with the following parameters:

• epochs = 30
• batch = 128
• filters = 32

Results

Side note: here, unlike most common code with keras, we’ve created a loop for multiple epochs in order to call each time getBatch and getValidation. As a side effect the history is not saved.
I’ve changed / tweaked a little bit the for loop for the different epochs, so that we can evaluate the model after each fit() call (it’ll also take more time), results are kepts in two pandas dataframes for the train and validation parts:

With the following code, you can concatenate all the results of the various epochs:

and then plot the metrics:

You can see the results below:

Results:
train - loss: 4.37, policy_loss: 3.67 - value_loss: 0.69 - policy_categorical_accuracy: 0.20 - value_mse: 0.11
valid - loss: 4.44, policy_loss: 3.74 - value_loss: 0.69 - policy_categorical_accuracy: 0.19 - value_mse: 0.11

Overfitting vs underfitting

The metrics on the validation dataset follow closely the ones on the train set, thus the model isn’t overfitting. Consequently we don’t have to use DropOut, Data augmentation (such as flipping images…) or early_stopping strategies so that the model would generalize better. On the other hand, we aren’t sure that the model is not underfitting, because more complex models would probably perform better: let’s discuss the potential improvement we can make.

Potential improvements

• With classical CNN there is an issue with vanishing gradient:

When there are more layers in the network, the value of the product of derivative decreases until at some point the partial derivative of the loss function approaches a value close to zero, and the partial derivative vanishes.

With shallow networks, sigmoid function can be used as the small value of gradient does not become an issue. When it comes to deep networks, the vanishing gradient could have a significant impact on performance. The weights of the network remain unchanged as the derivative vanishes. During back propagation, a neural network learns by updating its weights and biases to reduce the loss function. In a network with vanishing gradient, the weights cannot be updated, so the network cannot learn. The performance of the network will decrease as a result. [03]

This is also demonstrated practically this paper [07]: non-linearity plays an important role in neural network. Without them, they lose their expressiveness power. It also has an important impact on the neural net training. In particular, the activation shape the derivatives of the network

Changing the activation function could be a solution, for instance the Swish function - proposed by Google - could be an alternative to the sigmoid function. It performs better than ReLU in the case of deeper neural networks, it allows data normalization and leads to quicker convergence and learning of the NN. Swish can work around and prevent the vanishing gradient program, and hence allow training for small gradient updates. Anyway it has some limitations: it is time-intensive to compute for deeper layers with large parameter dimensions.

• Using other architecture is an other way to get better results, some of them may act as a workaround for the vanishing gradient issue.
• A better optimizer or an adaptation of the learning through the epochs, in order to converge more rapidly and to try as musch as possible to avoid local minimums.
• We can also consider different weights between the two head’s metrics.

The later ideas will be used in the following chapters. For example with the same CNN, if we only change the activation function to swish and train the model during 240 epochs, the results get better:

train - loss: 3.74 - policy_loss: 3.04 - value_loss: 0.69 - policy_categorical_accuracy: 0.30 - value_mse: 0.11
valid - loss: 3.73 - policy_loss: 3.03 - value_loss: 0.69 - policy_categorical_accuracy: 0.30 - value_mse: 0.11

The policy accuracy has increased from 0.2 to 0.3.

SHUFFLENET

ShuffleNet is a CNN designed specially for mobile devices with very limited computing power. The architecture utilizes two new operations, pointwise group convolution and channel shuffle, to reduce computation cost while maintaining accuracy.

How the ShuffleNet works ? [08]

• Channel Shuffle for Group Convolutions
  Xception and ResNeXt balance an excellent trade-off between representation capability and computational cost, by introducing depthwise separable convolutions or group convolutions into the building blocks, but cannot fully take the 1x1 convolutions, called pointwise convolutions, into account. In tiny networks, expensive pointwise convolutions result in limited number of channels to meet the complexity constraint, which might significantly damage accuracy.

Let’s compare 3 different cases relative to colored picture with 3 channels (RGB):

Figure 1(a) illustrates a situation of two stacked group convolution layers, which has twofold properties, blocking the information flow between channel groups and weakening representations.

Figure 1(b) shows that the group convolution is allowed to obtain input data from different groups. Notice that the input channels are fully related to the output ones.

Figure 1(c) sets up the feature map from the previous group layer, which is then implemented by a channel shuffle operation. Channel shuffle operation is able to construct more powerful structures with multiple group convolutional layers.

The configuration in Figure 1(c) is preferable, because the channel shuffle operation still applies in the stacked layers, though the convolutions in Figure 1(b) have different number of groups. Additionally, it is differentiable so it is embedded into network structures.

• ShuffleNet Unit

Figure 2(a) is a bottleneck unit with depthwise convolution (3x3 DWConv).

Figure 2(b) is a ShuffleNet unit with pointwise group convolution and channel shuffle.

The purpose of the second pointwise group convolution is to recover the channel dimension to match the shortcut path.

Figure 2(c) is a ShuffleNet unit with stride of 2.

Because of the pointwise group convolution with channel shuffle, all components in ShuffleNet unit can be computed efficiently.

Implementation

Here is the code used for the model version 12, it starts with a Conv2D, followed by several blocks of bottleneck:

The main branch of the block consists of:

• line 13: 1×1 Group Convolution with 1/6 filters followed by Batch Normalization and ReLU
• line 17: Channel Shuffle
• line 18: 3×3 DepthWise Convolution followed by Batch Normalization
• line 21: 1×1 Group Convolution followed by Batch Normalization

The tensors of the main branch and the shortcut connection are then concatenated and a ReLU activation is applied to the output.

Results

• Total params: 28,485
• Trainable params: 24,581
• Non-trainable params: 3,904

train - loss: 6.58 - policy_loss: 5.88 - value_loss: 0.68 - policy_categorical_accuracy: 0.003 - value_mse: 0.11
valid - loss: 6.58 - policy_loss: 5.88 - value_loss: 0.68 - policy_categorical_accuracy: 0.003 - value_mse: 0.11

The results are not good, the accuracy is really low: it’s a little bit disappointing, but I should have tried multiple shufflenet architectures. It could have also been interesting to use the Adam optimizer to compare to the default choice.

RESNET

How the ResNet works ?

The use of residual networks can improve the training of a policy network. Training is faster than with usual CNN & residual networks achieve a relatively high accuracy on the test set. The principle of residual nets is to add the input of the layer to the output of each layer. With this simple modification training is faster and enables deeper networks:

The usual layer used in computer Go program such as AlphaGo is composed of a convolutional layer and of a ReLU layer as shown in figure above. The residual layer used for image classification adds the input of the layer to the output of the layer. 04

How exactly could this prevent the vanishing gradient problem (VGP)?

Initially ResNets were not introduced to specifically solve the VGP, but to improve learning in general. The authors of ResNet, in the original paper, noticed that NN without residual connections don’t learn as well as ResNets, although they are using batch normalization, which, in theory, ensures that gradients should not vanish. But ResNets may also potentially mitigate (or prevent to some extent) the VGP:

The skip connections allow information to skip layers, so, in the forward pass, information from layer l can directly be fed into layer l+t (i.e. the activations of layer l are added to the activations of layer l+t), for t≥2, and, during the forward pass, the gradients can also flow unchanged from layer l+t to layer l.

The VGP occurs when the elements of the gradient (the partial derivatives with respect to the parameters of the NN) become exponentially small, so that the update of the parameters with the gradient becomes almost insignificant (i.e. if you add a very small number 0<ϵ≪1 to another number d, d+ϵ is almost the same as d) and, consequently, the NN learns very slowly or not at all. Given that these partial derivatives are computed with the chain rule, this can easily occur, because you keep on multiplying small (finite-precision) numbers.

The deeper the NN, the more likely the VGP can occur. The addition of the information from layer l will make the activations bigger, thus, to some extent, they will prevent these activations from becoming exponentially small. A similar thing can be said for the back-propagation of the gradient. (source StackExchange)

Implementation

Here is the code of version 17: unfortunately i’ve not uploaded the h5 on the google drive in order to be part of the tournament!

The addition is made line 6, with the variable “ident” initialized with the value of x before the 2 convolutions:

Results

• Total params: 99,471
• Trainable params: 99,471
• Non-trainable params: 0

train - loss: 3.22 - policy_loss: 2.53 - value_loss: 0.69 - policy_categorical_accuracy: 0.37 - value_mse: 0.11
valid - loss: 3.24 - policy_loss: 2.55 - value_loss: 0.69 - policy_categorical_accuracy: 0.37 - value_mse: 0.11

The results are quite promissing: the accuracy is good. Here, the use of the Adam optimizer was a “game changer”, because the accuracy was “only” 0.31 with SGD.

What is Adam?

Adam optimization - first presented as a conference paper at ICLR 2015 - is an extension to Stochastic gradient decent and can be used in place of classical SGD to update network weights more efficiently. Adam uses Momentum and Adaptive Learning Rates to converge faster.

• Momentum

When explaining momentum, researchers and practitioners alike prefer to use the analogy of a ball rolling down a hill that rolls faster toward the local minima, but essentially what we must know is that the momentum algorithm, accelerates stochastic gradient descent in the relevant direction, as well as dampening oscillations.

• Adaptive Learning Rate

Adaptive learning rates can be thought of as adjustments to the learning rate in the training phase by reducing the learning rate to a pre-defined schedule. In keras, one can change the “decay” value for that purpose.

Imagine a ball, we started from some point and then the ball goes in the direction of downhill or descent. If the ball has the sufficient momentum than the ball will escape from the well or local minima in our cost function graph.

Gradient Descent with Momentum considers the past gradients to smooth out the update. It computes an exponentially weighted average of your gradients, and then use that gradient to update the weights.

MOBILENET

How the MobileNet works?

MobileNet and then MobileNetV2 are parameter efficient NN architectures for computer vision. Instead of usual convolutional layers in the block they use depthwise convolutions. They also use 1x1 filters to pass from a small number of channels in the trunk to 6 times more channels in the block.

Mobile Networks are commonly used in computer vision to classify images. They obtain high accuracy for standard computer vision datasets while keeping the number of parameters relatively lower than other neural networks architectures.

The proposed MobileNetV2 network architecture (for an other task / purpose):

[…] The principle of MobileNetV2 is to have blocks as in residual networks where the input of a block is added to its output. But instead of usual convolutional layers in the block they use depthwise convolutions. Moreover the number of channels at the input and the output of the blocks (in the trunk) is much smaller than the number of channels for the depthwise convolutions in the block. In order to efficiently pass from a small number of channels in the trunk to a greater number in the block, usual convolutions with cheap 1x1 filters are used at the entry of the block and at its output. […]

There is a trade-off between the accuracy and the speed of the networks. Some experiments were made and explained in this paper [05] with various depth and width settings of networks: when increasing the size of the networks there is a balance to keep between the depth and the width of the networks:

[…] to improve the performance of a network it is better to make both the number of blocks (i.e. the depth of the network) and the number of planes (i.e. the width of the network) grow together. […]

Implementation

Here is the code of version 21, with the depthwise convolution line 5:

we can see the the “bottleneck_block” function called several times in a loop before the two heads, which are unchanged:

several tries were made with different loss_weights:

Results

• Total params: 99,577
• Trainable params: 94,969
• Non-trainable params: 4,608

train - loss: 6.10 - policy_loss: 2.24 - value_loss: 0.61 - policy_categorical_accuracy: 0.42 - value_mse: 0.09
valid - loss: 6.14 - policy_loss: 2.23 - value_loss: 0.62 - policy_categorical_accuracy: 0.42 - value_mse: 0.08

Only the change of weight for losses managed to decrease the value mean squared error, otherwise the policy accuracy still remains the highest values i could get.

CONCLUSION

Results comparison

The table below summarises all the differents architectures tried for this project. That way, we can see the influence of a specific parameter between two version’s results. The models used for the tournament are selected in green. Unfortunately, the version 17 was not uploaded:

Strategy

• i sould have tried more versions of shufflenet but at that moment, other architectures seem to be more efficient.
• at first, i focused on increasing the accuracy, because the MSE of the value didn’t change that much.
• i’ve changed parameters one by one between versions in order to see its impact and try to modify both depth or width, or both.
• a huge gap in the results was obtained by changing the optimizer.
• the only way to decrease significantly the value’s MSE - at the end - was to change the losses’ weighs, to give more importance on the later for the optimizer because models don’t seem to learn the value.
• the metrics evolution graphs for each model version are not presented here but remains quite the sames: this is especially true for the evolutions of policy loss and of categorical accuracy.

Other solutions

• The keras & tensorflow version installed by default on the SaturnCloud (which provide freely 30 hours of GPU monthly) virtual machine doesn’t allow me to use other optimizers such as LION (release recently by Google Brain) or ADAMW.
• An other solution would be to use Transformers, to my knowledge there is not any ground breaking transformers’ architecture for computer vision yet, but some models might deserve a try.
• We could have; of course; use more deep models but that would leads to more than 100,000 parameters.
• Combining models with Reinforcement Learning is an other option but goes beyond the scope of this study.
• Learning rate schedules and decay is also to condiser: we can view the process of learning rate scheduling as:
  • Finding a set of reasonably “good” weights early in the training process with a larger LR.
  • Tuning these weights later in the process to find more optimal weights using a smaller LR.
• or Cosine Annealing [07]: [..] using a different optimization strategy, a neural net can end in a better optimum. [..] this can be achieved by using Stochastic Gradient Descent with warms Restart. In particular, the learning rate is restarted multiple times. This way, the objective landscape is explored further and the best solution of all restart is kept. Furthermore, using a peculiarly aggressive learning rate strategies like cosine annealing can lead to better convergence.
• Whereas Transfer Learning is not suitable here (because of the parameters number limitation), knowledge distillation (the process of transferring knowledge from a large model to a smaller one), on the other hand could be really interesting.

References

• 01 - Game of Go - Wikipedia
• 02 - AlaphaGo - Deepmind
• 03 - Vanishing gradient - kdnuggets
• 04 - Residual Networks for Computer Go, Tristan Cazenave. IEEE Transactions on Games, Vol. 10 (1), pp. 107-110, March 2018
• 05 - Improving Model and Search for Computer Go, Tristan Cazenave. IEEE Conference on Games 2021
• 06 - Mobile Networks for Computer Go, Tristan Cazenave. IEEE Transactions on Games, Vol. 14 (1), pp. 76-84, January 2022
• 07 - Cosine Annealing, Mixnet and Swish Activation for Computer Go, Tristan Cazenave, Julien Sentuc, Mathurin Videau. Advances in Computer Games 2021
• 08 - ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices by SyncedReview on Medium

Introduction

Clustering is grouping objects based on similarities (according to some defined criteria). It can be used in many areas: customer segmentation, computer graphics, pattern recognition, image analysis, information retrieval, bioinformatics, and data compression…

The k-means algorithm is well known for its efficiency in clustering large data sets. However, working only on numeric values prohibits it from being used to cluster real world data containing categorical values:

One hot encoding is often a bad idea:

• it doesn’t always make sense to compute the euclidean distance between points with ohe features
• if attributes have hundreds or thousands of categories: this will increase both computational and space costs of the k-means algorithm. An other drawback is that the cluster means, given by real values between 0 and 1, do not indicate the characteristics of the clusters.

One solution is to use:

• the k-modes algorithm which enables the clustering of categorical only data in a fashion similar to k-means.
• the k-prototypes algorithm, through a combination of the principles of k-means & k-modes can deal with mixed data.

!pip install umap-learn
!pip install lightgbm
!pip install kmodes
!pip install shap
!pip install kmodes
# conda install -c conda-forge kmodes


from google.colab import drive
drive.mount('/content/drive')

Theory

The k-modes algorithm follows the same k-means’ steps

The k-means algorithm

Steps:

1. Select the Number of Clusters, k
2. Select k Points at Random
3. Make k Clusters (by assigning the closest points to the cluster)
4. Compute New Centroid of Each Cluster
5. Assess the Quality of Each Cluster (by measuring the Within-Cluster Sum of Squares (WCSS) to quantify the variation within all the clusters)
6. Repeat Steps 3–5 (with the centroids we calculated previously to make 3 new clusters)

The goal is to minimize the mean of the squared Euclidean distance between the data points and centroids:

where W is an n × k partition matrix, Q = {Q1, Q2,…, Qk } is a set of k centroids, and d(·, ·) is the distance and X the n data points. The algorithm converges to a local minimum point.
The computational cost of the algorithm is O(Tkn) (T = the iteration number).

The k-modes algorithm

The limitations of k-means can be removed by making the following modifications:

1. using a simple matching dissimilarity measure for categorical objects,
2. replacing means of clusters by modes,
3. using a frequency-based method to find the modes to solve problem

Dissimilarity measure

Often referred to as simple matching “Kaufman and Rousseeuw”, it can be defined by the total mismatches of the corresponding attribute categories of the 2 elements X and Y: the smaller the number of mismatches is, the more similar the 2 elements:

Mode of a set

A mode of X = {X1, X2,…, Xn} is a vector Q = [q1, q2,…, qm] that minimises:

It implies that the mode of a data set X is not unique. For example, the mode of set {[a, b], [a, c], [c, b], [b, c]} can be either [a, b] or [a, c].

The k-modes algorithm

the cost function becomes:

Where m is the number of (categorical) features.
The proof of convergence for this algorithm was not available at the time of the paper’s release. However, its practical use has shown that it always converges. Like the k-means, the k-modes algorithm also produces locally optimal solutions that are dependent on the initial modes and the order of objects (rows) in the data set.

The k-prototypes algorithm

The dissimilarity between two mixed-type objects X and Y:

where the first term is the squared Euclidean distance measure on the numeric attributes and the second term is the simple matching dissimilarity measure on the categorical attributes. The weight γ is used to avoid favouring either type of attribute.

the cost function becomes:

The 2 terms are nonnegative, minimising the total is equivalent to minimising both of them.

Clustering performance

• Experiments have shown good results on both synthetic and real world datasets.
• It was designed in order to be able to scale but it was more than 20 years ago ! The k-modes and k-prototypes implementations both offer support for multiprocessing via the joblib library, similar to e.g. scikit-learn’s implementation of k-means, using the n_jobs. But it doesn’t seem to be parallelized on multiple nodes: there is an unofficial implementation on PySpark…
• An important observation was that the k-modes algorithm was much faster than the k-prototypes algorithm. The key reason is that the k-modes algorithm needs many less iterations to converge than the k-prototypes algorithm because of its discrete nature.

Conclusion

• Pros:
  • k-modes & k-prototypes can deal with categorical features
  • they are scalable & efficient
  • k-modes allows missing values and can deal with outliers whereas K-prototypes do not allow numeric attributes to have missing values & is still sensitive to numeric outliers.
• Cons:
  • the weight γ adds an additional problem: the average standard deviation of numeric attributes is suggested by the author but the user’s knowledge is important in specifying γ: if one thinks should be favoured on numeric attributes (small γ) or categorical ones (large γ).
  • the dissimilarity measure makes a separation between categorical & numerical features
• Unchanged:
  • we still face the common problem: how many clusters are in the data?
  • how to initialize the centroids? prefer the init='Cao' by default (based on density / DBSCAN) over ‘Huang’ or ‘random’ which are less efficient. All algorithms are sensitive to the order of observations: it is worth to run it several times, shuffling data in between, averaging resulting clusters and running final evaluations with those averaged clusters centers as starting points
  • standardization of numerical features is still required (except in you want to weight features or in special cases: geospacial datas, keeping the same units…)
  • Validation of clustering results in case of lack of a priori knowledge to the data: use visualisation techniques
  • No ‘silhouette’ implemented / found for the consistency interpretation with dissimilarity metrics.

In practice

import pandas as pd
import numpy as np

from kmodes.kprototypes import KPrototypes

import warnings
warnings.filterwarnings('ignore', category = FutureWarning)
pd.set_option('display.float_format', lambda x: '%.1f' % x)


df = pd.read_csv('/content/drive/MyDrive/kproto/10000 Sales Records.csv')
df.drop(['Country', 'Order Date', 'Order ID', 'Ship Date'], axis=1, inplace=True)
print(df.shape)
df.head()

(10000, 10)

df.select_dtypes('object').nunique()

Region             7
Item Type         12
Sales Channel      2
Order Priority     4
dtype: int64

df.isna().sum()

Region            0
Item Type         0
Sales Channel     0
Order Priority    0
Units Sold        0
Unit Price        0
Unit Cost         0
Total Revenue     0
Total Cost        0
Total Profit      0
dtype: int64

Get the position of categorical columns

categorical_col_positions = [df.columns.get_loc(col) for col in list(df.select_dtypes('object').columns)]
print('Categorical columns           : {}'.format(list(df.select_dtypes('object').columns)))
print('Categorical columns position  : {}'.format(categorical_col_positions))

Categorical columns           : ['Region', 'Item Type', 'Sales Channel', 'Order Priority']
Categorical columns position  : [0, 1, 2, 3]

Convert dataframe to a numpy matrix (required for k-prototypes)

df_matrix = df.to_numpy()
df_matrix

array([['Sub-Saharan Africa', 'Office Supplies', 'Online', ...,
        2920025.64, 2353920.64, 566105.0],
       ['Europe', 'Beverages', 'Online', ..., 51008.75, 34174.25,
        16834.5],
       ['Middle East and North Africa', 'Vegetables', 'Offline', ...,
        1003700.9, 592408.95, 411291.95],
       ...,
       ['Sub-Saharan Africa', 'Vegetables', 'Offline', ..., 388847.44,
        229507.32, 159340.12],
       ['Sub-Saharan Africa', 'Meat', 'Online', ..., 3672974.34,
        3174991.14, 497983.2],
       ['Asia', 'Snacks', 'Offline', ..., 55081.38, 35175.84, 19905.54]],
      dtype=object)

# Choose optimal K using Elbow method
costs = []
for nb_cluster in range(1, 6):
        kprototype = KPrototypes(n_jobs=-1, n_clusters=nb_cluster, random_state=0)
        kprototype.fit_predict(df_matrix, categorical=categorical_col_positions)
        costs.append(kprototype.cost_)
        print(f'Nb of cluster: {nb_cluster}, cost: {kprototype.cost_}')

Nb of cluster: 1, cost: 3.6016173492135216e+16
Nb of cluster: 2, cost: 9627986092172950.0
Nb of cluster: 3, cost: 4960707512782195.0
Nb of cluster: 4, cost: 2927457054391281.0
Nb of cluster: 5, cost: 1975342819318915.2

Sometimes (e.g for a value of k=6) you gert a ValueError: Clustering algorithm could not initialize. Consider assigning the initial clusters manually.
Surprisingly this a feature, not a bug: kmodes is telling you that what you’re doing likely does not make sense given the data you’re presenting it. And because every data set is different, it’s up to you to figure out why :)

import plotly.express as px

df_cost = pd.DataFrame(list(zip(range(1, 6), costs)), columns=['Nb_clusters', 'Costs'])
px.line(df_cost, x="Nb_clusters", y="Costs", 
        title='Elbow method - Cost depending on k',
        width=700, height=500
       ).show()

Re-fit the model with the optimal k value

optimal_k = 3
kprototype = KPrototypes(n_jobs=-1, n_clusters=optimal_k, random_state=0)
clusters = kprototype.fit_predict(df_matrix, categorical = categorical_col_positions)
clusters

array([2, 1, 1, ..., 1, 0, 1], dtype=uint16)

model object

kprototype

KPrototypes(gamma=249301.104054418, n_clusters=3, n_jobs=-1, random_state=0)

Cluster centroid

kprototype.cluster_centroids_

array([['7904.365546218487', '593.5265126050414', '457.78549579832264',
        '4622760.54755461', '3559121.1234873924', '1063639.424067227',
        'Europe', 'Household', 'Offline', 'L'],
       ['4046.6694875411376', '163.25910672308513', '105.79686882933991',
        '467709.4517003633', '281142.9546497409', '186566.49705061832',
        'Sub-Saharan Africa', 'Personal Care', 'Online', 'C'],
       ['6093.2754219843555', '384.2645039110791', '275.0975010292268',
        '1995888.1232729559', '1380539.5273034181', '615348.5959695352',
        'Europe', 'Cosmetics', 'Online', 'H']], dtype='<U32')

Check the iteration of the clusters created

kprototype.n_iter_

14

Assign the cluster label to each record / row

# Add the cluster to the dataframe
df['Cluster Labels'] = kprototype.labels_
df['Segment'] = df['Cluster Labels'].map({0:'First', 1:'Second', 2:'Third'})

# Order the cluster
df['Segment'] = df['Segment'].astype('category')
df['Segment'] = df['Segment'].cat.reorder_categories(['First','Second','Third'])

Cluster interpretation

df.rename(columns = {'Cluster Labels':'Total'}, inplace = True)
df.groupby('Segment').agg(
    {
        'Total':'count',
        'Region': lambda x: x.value_counts().index[0],
        'Item Type': lambda x: x.value_counts().index[0],
        'Sales Channel': lambda x: x.value_counts().index[0],
        'Order Priority': lambda x: x.value_counts().index[0],
        'Units Sold': 'mean',
        'Unit Price': 'mean',
        'Total Revenue': 'mean',
        'Total Cost': 'mean',
        'Total Profit': 'mean'
    }
).reset_index()

df_initial = df.copy()
df_initial.drop(columns=['Segment', 'Total'], inplace=True)

Among several clustering methods, which one is the more efficient?

Visualization - UMAP Embedding

One of the comparison methods will be visual, so we need a way to visulize the quality of clustering. For instance by using the Uniform Manifold Approximation and Projection for Dimension Reduction (UMAP) - a dimensionality reductin technique (like PCA or t-SNE) - to embed the data into 2 dimensions. This will allow to visually see the groups of customers, and how well did the clustering algo do the job. There are 3 steps to get the proper embeddings:

1. Yeo-Johnson transform the numerical columns & One-Hot-Encode the categorical data
2. Embed these two column types separately
3. Combine the two by conditioning the numerical embeddings on the categorical embeddings as suggested here

import umap
from sklearn.preprocessing import PowerTransformer


#Preprocessing numerical
numerical = df_initial.select_dtypes(exclude='object')

for c in numerical.columns:
    pt = PowerTransformer()
    numerical.loc[:, c] = pt.fit_transform(np.array(numerical[c]).reshape(-1, 1))
    
##preprocessing categorical
categorical = df_initial.select_dtypes(include='object')
categorical = pd.get_dummies(categorical)

#Percentage of columns which are categorical is used as weight parameter in embeddings later
categorical_weight = len(df_initial.select_dtypes(include='object').columns) / df_initial.shape[1]

#Embedding numerical & categorical
fit1 = umap.UMAP(metric='l2').fit(numerical)
fit2 = umap.UMAP(metric='dice').fit(categorical)

import matplotlib.pyplot as plt


#Augmenting the numerical embedding with categorical
intersection = umap.umap_.general_simplicial_set_intersection(fit1.graph_, fit2.graph_, weight=categorical_weight)
intersection = umap.umap_.reset_local_connectivity(intersection)
embedding = umap.umap_.simplicial_set_embedding(fit1._raw_data, intersection, fit1.n_components, 
                                                fit1._initial_alpha, fit1._a, fit1._b, 
                                                fit1.repulsion_strength, fit1.negative_sample_rate, 
                                                200, 'random', np.random, fit1.metric, 
                                                fit1._metric_kwds, densmap=False, densmap_kwds={}, output_dens=False)

plt.figure(figsize=(20, 10))
plt.scatter(x=embedding[0][:, 0], y=embedding[0][:, 1], s=2, cmap='Spectral', alpha=1.0)
plt.show()

fig, ax = plt.subplots()
fig.set_size_inches((20, 10))
scatter = ax.scatter(embedding[0][:, 0], embedding[0][:, 1], s=2, c=clusters, cmap='tab20b', alpha=1.0)

# produce a legend with the unique colors from the scatter
legend1 = ax.legend(*scatter.legend_elements(num=optimal_k),
                    loc="lower left", title="Classes")
ax.add_artist(legend1)

Classification evaluation

Another comparison that can be done ib by treating the clusters as labels and building a classification model on top. If the clusters are of high quality, the classification model will be able to predict them with high accuracy. At the same time, the models should use a variety of features to ensure that the clusters are not too simplistic. Overall, I’ll check the quality by:

1. Distinctivness of clusters by cross-validated F1 score
2. The informativness of clusters by SHAP feature importances

#Setting the objects to category 
lgbm_data = df.copy()
for c in lgbm_data.select_dtypes(include='object'):
    lgbm_data[c] = lgbm_data[c].astype('category')

from lightgbm import LGBMClassifier
import shap
from sklearn.model_selection import cross_val_score


clf_kp = LGBMClassifier(colsample_by_tree=0.8)
cv_scores_kp = cross_val_score(clf_kp, lgbm_data, clusters, scoring='f1_weighted')
print(f'CV F1 score for K-Prototypes clusters is {np.mean(cv_scores_kp)}')

CV F1 score for K-Prototypes clusters is 1.0

clf_kp.fit(lgbm_data, clusters)

LGBMClassifier(colsample_by_tree=0.8)

explainer_kp = shap.TreeExplainer(clf_kp)
shap_values_kp = explainer_kp.shap_values(lgbm_data)
shap.summary_plot(shap_values_kp, lgbm_data, plot_type="bar", plot_size=(15, 10))

Classifiers for clustering methods that have F1 score close to 1 have produced clusters that are easily distinguishable. Different clustering methods (and classifiers) would probably use more or less features, and some of the categorical features become important. The more features used, the more informative is the clustering algorithm.

Alternative solutions to consider

• k-Mediods
  • similar to k-means also partitional (breaking the dataset up into groups) and attempt to minimize the distance between points & clusters’ centers
  • in contrast to the k-means algo, k-mediods chooses actual data points as centers (mediods or exemplars) for a greater interpretability
  • furthermore, k-mediods can be used with arbitrary dissimilarity measures such as Gower, where as k-means generally requires Euclidean distance for efficient solutions
  • minimizes a sum of pairwise dissimilarities instead of a sum of squared Euclidean distances
  • more robust to noise and outliers than k-means.
• Dimensionality Reduction
  • to transform our data into a lower dimensional space while retaining as much info as possible or to homogenize our mixed dataset
  • tow different techniques:
    • Factorial analysis of Mixed Data - FAMD (a king of PCA on OHE categorical variables & standardize the numerical ones)
    • UMAP as seen above (prediction upon manifold learning & ideas from topological data analysis).

References:

• Papers:
  • k-modes [HUANG97] [HUANG98]
  • k-modes with initialization based on density [CAO09]
  • k-prototypes [HUANG97]
• Implementations in python (interface similar to scikit-learn): Nico de Vos’ github repo
• Examples of use cases:
  • Customer Clustering by Anton Ruberts (personnal web site
  • Sales records by Audhi Aprilliant (towards datascience
  • Customer Segmentation by Shivam Soliya (medium)
• Other ressources:
  • A Guide for Clustering Mixed Data by Eoghan Keany (medium)
  • Exploring Clustering Algorithms: Explanation and Use Cases by Aravind CR (neptune AI blog)
  • 10 Tips for Choosing the Optimal Number of Clusters by Matt.O

I’ve also found a great article covering other clustering models by Advancinganalytics.co.uk

You can find an Anki deck with the following snippets / plots each in a dedicated flashcard in order to memorize all this stuff here, in my Github repository

Introduction

What is plotly express?

• a high-level data visualization package
• it allows you to create interactive plots with very little code.
• built on top of Plotly Graph Objects (go provides a lower-level interface for developing custom viz).

This cheat sheet covers all you need to know to get started with plotly in Python.

Basics

import plotly express

import plotly.express as px

interactive controls

Functions:

• Basics: scatter, line, area, bar, funnel, timeline
• Part-of-Whole: pie, sunburst, treemap, icicle, funnel_area
• 1D Distributions: histogram, box, violin, strip, ecdf
• 2D Distributions: density_heatmap, density_contour
• Matrix or Image Input: imshow
• 3-Dimensional: scatter_3d, line_3d
• Multidimensional: scatter_matrix, parallel_coordinates, parallel_categories
• Tile Maps: scatter_mapbox, line_mapbox, choropleth_mapbox, density_mapbox
• Outline Maps: scatter_geo, line_geo, choropleth
• Polar Charts: scatter_polar, line_polar, bar_polar
• Ternary Charts: scatter_ternary, line_ternary

Code pattern

px.plotting_fn(
    dataframe,                  # pd.DataFrame
    x=["column-for-x-axis"],    # str or a list of str
    y=["columns-for-y-axis"],   # str or a list of str
    title="Overall plot title", # str
    xaxis_title="X-axis title", # str
    yaxis_title="Y-axis title", # str
    width=width_in_pixels,      # int
    height=height_in_pixels     # int
) 

Scatter plot

color can be discrete/categorical

df = px.data.iris()

px.scatter(
    df, 
    x="sepal_width", 
    y="sepal_length", 
    color="species",
    size='petal_length', 
    hover_data=['petal_width'],
    width=500,
    height=350
).show()

color can also be continuous

px.scatter(
    px.data.iris(), 
    x="sepal_width", 
    y="sepal_length", 
    color='petal_length',
    width=500,
    height=350
).show()

a scatter plot with symbols that map to a column

px.scatter(
    px.data.iris(), 
    x="sepal_width", 
    y="sepal_length", 
    color="species",
    symbol="species",
    width=500,
    height=350
).show()

Line Plot

df = px.data.gapminder().query("country=='Canada'")

px.line(
    df, 
    x="year", 
    y="lifeExp", 
    title='Life expectancy in Canada',
    width=500,
    height=350
).show()

Line Plot with column encoding color

df = px.data.gapminder() \
    .query("continent=='Oceania'")

px.line(
    df, 
    x="year", 
    y="lifeExp", 
    title='Life expectancy in Canada',
    color='country',
    width=500,
    height=350
).show()

Line chart with markers

df = px.data.gapminder().query("continent == 'Oceania'")

px.line(
    df,
    x='year',
    y='lifeExp',
    color='country',
    markers=True,
    symbol="country", # optional
    width=500,
    height=350
).show()

Line plot on Date axes

px.line(
    px.data.stocks(),
    x='date',
    y="GOOG",
    width=500,
    height=350
).show()

Connected Scatterplots

df = px.data.gapminder() \
    .query("country in ['Canada', 'Botswana']")

fig = px.line(
    df,
    x="lifeExp",
    y="gdpPercap",
    color="country",
    text="year",
    width=500,
    height=350
)

fig.update_traces(textposition="bottom right")
fig.show()

Bar chart / plot

by default vertical

df = px.data.gapminder().query("country == 'Canada'")

px.bar(
    df,
    x='year',
    y='pop',
    width=500,
    height=350
).show()

Bar chart with Long Format Data

long_df = px.data.medals_long()
display(long_df)

px.bar(
    long_df,
    x="nation",
    y="count",
    color="medal",
    title="Long-Form Input",
    width=500,
    height=350
).show()

Bar chart with Wide Format Data

wide_df = px.data.medals_wide()
display(wide_df)

px.bar(
    wide_df,
    x="nation",
    y=["gold", "silver", "bronze"],
    title="Wide-Form Input",
    width=500,
    height=350
).show()

Swap the x and y arguments to draw horizontal bars.

wide_df = px.data.medals_wide()
display(wide_df)

px.bar(
    wide_df,
    y="nation",
    x=["gold", "silver", "bronze"],
    width=500,
    height=350
).show()

Histogram

px.histogram(
    px.data.tips(),
    x="total_bill",
    width=500,
    height=350
).show()

Histogram that use a column with categorical data

px.histogram(
    px.data.tips(),
    x="day",
    width=500,
    height=350
).show()

Histogram & choosing the number of bins

px.histogram(
    px.data.tips(),
    x="total_bill",
    nbins=20,
    width=500,
    height=350
).show()

Histogram on Date Data

fig = px.histogram(
    px.data.stocks(),
    x="date",
    width=500,
    height=350
)
fig.update_layout(bargap=0.2)
fig.show()

Histogram on Categorical Data

px.histogram(
    px.data.tips(),
    x="day", 
    category_orders=dict(day=["Thur", "Fri", "Sat", "Sun"]),
    width=500,
    height=350
).show()

Several histogram for the different values of one column

px.histogram(
    px.data.tips(),
    x="total_bill",
    color="sex", 
    width=500,
    height=350
).show()

Colored Bar

px.bar(
    px.data.gapminder().query("country == 'Canada'"),
    x='year',
    y='pop',
    hover_data=['lifeExp', 'gdpPercap'], 
    color='lifeExp',
    labels={'pop':'population of Canada'}, 
    width=500,
    height=350
).show()

Grouped Bar / Histogram

px.histogram(
    px.data.tips(),
    x="sex",
    y="total_bill",
    color='smoker',
    barmode='group', 
    width=500,
    height=350
).show()

Grouped Bar with Avg

px.histogram(
    px.data.tips(),
    x="sex",
    y="total_bill",
    color='smoker',
    barmode='group',
    histfunc='avg',
    width=500,
    height=350
).show()

Bar Chart with Text

px.bar(
    px.data.medals_long(),
    x="medal",
    y="count",
    color="nation",
    text_auto=True,
    width=500,
    height=350
).show()

Heatmap

df = px.data.medals_wide(indexed=True)
display(df)

px.imshow(
    df,    
    width=500,
    height=350
).show()

Displaying Text on Heatmap

z = [[.1, .3, .5, .7, .9],
     [1, .8, .6, .4, .2],
     [.2, 0, .5, .7, .9],
     [.9, .8, .4, .2, 0],
     [.3, .4, .5, .7, 1]]

px.imshow(
    z,
    text_auto=True,    
    width=500,
    height=350
).show()