What flummoxed me in the first post was the fact was that the Euclidean distances between articles which are related to each other was greater than the distances between ones that were not related to each other. So, I visited the original article Alexandru Nedelcu(Twitter|Blog) on data mining and started reading it in more detail. If you look at Euclidean distance closely, it really is a measure of dissimilarity i.e. the greater the distance the more dissimilar the two things are. Also, since I did not give Ruby Array operations much thought; I implemented what I thought was right.

The curious thing about Ruby Arrays is a method called Zip that converts the arguments passed to it into an array and merges the caller with elements from each argument. What this means is that if you have two arrays ‘arrOne’ and ‘arrTwo’ which are defined as follows:

arrOne = [4, 5, 6]

arrTwo =[7, 8, 9]

Then, an operation like [1,2,3].zip(a, b) would result in a output similar to this:

[[1, 4, 7], [2, 5, 8], [3, 6, 9]]

First thing to do was to mimic the behavior in PowerShell. For our purposes that would suffice. One other reason for doing this is that it makes the code very small (you will see why in a minute).

Function Get-RbyZip($a,$b){
 [int]$i=0
 $lenA = $a.length
 $lenB = $b.length
 $aryRes = @()
 try{
  for($i=0;$i -lt $lenA; $i++)
  {
   if($i -ge $lenB){
    $aryRes += , @($a[$i],$null)
   }else{
    $aryRes += , @($a[$i],$b[$i])
   }
  }
  return $aryRes
 }
 catch{
  Write-Host "Error occured in [Get-RbyZip]" -Fore Red
 }
}

We need to tweak our Get-Point function as well since that is the heartline of our code and needed some fixing of stupidity (me, I was to blame).

Function Get-Point($tagSet,$tagSpace)
{
 try{
  $pntInSpace = $tagSpace|%{if($tagSet -contains $_){1}else{0}}
  return $pntInSpace
 }catch{
  Write-Host "Error occured in [Get-Point]" -Fore Red
 }
}

So, now our functions for getting the distances come out and they look small than the last time you saw them because of that one call to Get-RbyZip and since have a matrix to work with the calculations look a lot easier to follow and understand

Function Get-EuclideanDistance($pntA, $pntB)
{
 try{
  $zipC = Get-RbyZip $pntA $pntB
  for($i=0; $i -lt $zipC.length; $i++){
   $sumSq += [Math]::Pow(($zipC[$i][0] - $zipC[$i][1]),2)
  }
  
  $euclidDist = [Math]::Sqrt($sumSq)
  return $euclidDist
 }
 catch{
  Write-Host "Error occured in [Get-EuclideanDistance]" -Fore Red
 }
}

Function Get-CosineSimilarity($pntA, $pntB)
{
 try{
  $zipC = Get-RbyZip $pntA $pntB
  for($i=0;$i -lt $zipC.length; $i++){
   $sumSqA += [Math]::Pow($zipC[$i][0],2)
   $sumSqB += [Math]::Pow($zipC[$i][1],2)
   $dotPrd_AB += (($zipC[$i][0])*($zipC[$i][1]))
  }
  
  $magA = [Math]::Sqrt($sumSqA)
  $magB = [Math]::Sqrt($sumSqB)
  
  if(($magA -gt 0) -and ($magB -gt 0)){
   $cosSimilarity = $dotPrd_AB/($magA*$magB)
  }else{
   $cosSimilarity = 0
  }
  
  return $cosSimilarity
 }catch{
  Write-Host "Error occured in [Get-CosineSimilarity]" -Fore Red
 }
}

Now here is the biggest change and probably the one that fixed most of the issues was the way were generating reference points for our comparisons. Instead considering the whole dataset we limited ourselves to only the data from one subset belonging to a particular item and tried to fix its point in space relative to the different set that was not connected to this one in anyway. Identifying this problem was eureka moment of my day. So, once this was fixed the results started looking much better.

Function Measure-Similarity($hshArticle, $byTheseTags){
 
 $articleTags = $hshArticle.Keys|%{$hshArticle[$_].Tags}
 $tagSpace = ($byTheseTags + $articleTags)|Sort|Select -Unique
 
 $thisPoint = Get-Point $byTheseTags $tagSpace
 
 $articleToPoint = @{}
 
 foreach($key in $hshArticle.Keys){
  $keyTags = $hshArticle[$key].Tags
  $articleToPoint += @{$key = (Get-Point $keyTags $tagSpace)}
 }
 
 $euclidDist = @{}
 $cosSimilarity = @{}
 foreach($article in ($articleToPoint.Keys|Sort)){
  $euclidDist += @{$article = (Get-EuclideanDistance $thisPoint $articleToPoint[$article])}
  $cosSimilarity += @{$article = (Get-CosineSimilarity $thisPoint $articleToPoint[$article])}
 }
 
 return $euclidDist, $cosSimilarity
}

If you are looking to run some quick tests using the new code, here is some data that I was using.

$hshArticle = @{}
$hshArticle["Working with Perfmon CSV logs in Powershell - Part 3"] = @{Tags = @("Fun-Stuff")}
$hshArticle["Working with Perfmon CSV logs in Powershell - Part 2"] = @{Tags = @("PowerShell","csv","csv file","csv format","DBD","perfmon","perl")}
$hshArticle["Working with Perfmon CSV logs in Powershell - Part 1"] = @{Tags = @("PowerShell","csv","csv file","measure","perfmon","Powershell","summarize")}
$hshArticle["Reading a tweeters timeline using PowerShell-Take2"] = @{Tags = @("PowerShell","Powershell","Twitter API","XML","proof of concept","twitter","formatting")}
$hshArticle["Reading a tweeters timeline using PowerShell"]= @{Tags = @("PowerShell","Powershell","Twitter API","XML","proof of concept","twitter","initial release")}

$matchTags = "PowerShell", "csv","perfmon"

$ed, $cs = Measure-Similarity $hshArticle $matchTags

Write-Host "For Euclidean Distance, lesser the more similar two things are" -Fore Green
Write-Host "For Cosine Similarity, the closer to 1 the better, as Cosine for zero is 1" -Fore Green
foreach($article in ($ed.Keys|Sort)){
 Write-Host " Article:" $article -Fore Green
 Write-Host "  Euclidean Distance:" $ed[$article] -Fore Yellow
 Write-Host "  Cosine Similarity: " $cs[$article] -Fore Yellow
}

And here is how the output looks with this test data:

Fixed it hopefully

That is all for today folks!

[Coming Soon: Learning from Turing|How to prove you really are human? using fun Powershell Quizzes]

So, armed with the article by Alex and some Wikipedia searches I tried to do some data mining on my blog; more out of interest than anything else. I tried to find out if it was easy to find posts similar to one another. If I was successful in doing this then, extending this to find similar posts on various blogs I follow would be the next logical step.

As mentioned in the first paragraph, grouping things by similarity/dissimilarity or distance from one another is our main aim. The most common of these distance calculating methods is the Euclidean distance; and is defined as the geometric distance in the multidimensional space. It is computed as:

There is a slight problem with the Euclidean distance; which is that we need to represent the data as points in a multi-dimensional space. We will have to first convert our data into points in space. One easy way to do this as pointed out by Alex, was to represent a match in the collection by 1 and a miss by 0. Ok, seems easy so let’s do it:

Function Get-Point($tagValue,$tagsToMatch)
{
    $pntInSpace = $tagsToMatch|%{if($_ -contains $tagValue){1}else{0}}
    $pntInSpace
}

Now that we have a way of converting our data into points the next thing to do would be to calculate the Euclidean distance since we already know the formula from above

Function Get-EuclideanDistance($a,$b)
{ 
    try
    {
     if($a.length -eq $b.length){
            $lenArr = $a.Length
           
      $sumSqB = ($b|%{[Math]::Pow($_,2)}|Measure-Object -Sum|Select Sum).Sum
      $sumSqA = ($a|%{[Math]::Pow($_,2)}|Measure-Object -Sum|Select Sum).Sum
            $sumAB = 0
           
            for($i=0;$i -lt $lenArr;$i++){
                $sumAB += ($a[$i]*$b[$i])
            }
            $sumTwoAB = 2*$sumAB
   
            $euclideanDist = [Math]::sqrt(($sumSqA+$sumSqB-$sumTwoAB))
            return $euclideanDist
     }else{
      throw $("The points have different number of dimensions")
     }
    }
    catch
    {
        Write-Host $_ -Fore Red|FL *
    }
 
}

A slight variation of the Euclidean distance called the Squared Euclidean distance exists which puts more weight on points farther away from one another by removing the square root from the picture:

Function Get-SquaredEuclideanDistance($a,$b)
{
    try
    {
  if($a.length -eq $b.length)
  {
   $lenArr = $a.Length
            $disAtoB = 0
            for($i=0;$i -lt $lenArr;$i++)
   {
    $disAtoB += [Math]::pow(($b[$i]-$a[$i]),2)
   }
            $sqeuclideanDist = [Math]::sqrt($disAtoB)
            return $sqeuclideanDist      
  }else{             
   throw $("The points have different number of dimensions")
  }   
 }   
 catch
    {        Write-Host $_ -Fore Red|FL *    }
}

The above two methods will give dramatically varying results if the scale of measurement is changed. This is disastrous if you are grouping objects by fixed weights. So, this maybe a problem and if so, is there a better way to do this? Yes, there is a better way to do the distance calculations and it is introduced in the next paragraph.

The third method called the Cosine similarity, measures the similarity between two vectors by measuring the cosine of the angle between them. This method is used for text mining of documents. The diagrammatic representation and formulation are given below. 

Now, that we know what Cosine similarity is and the formula to calculate it; let us look at the code

Function Get-CosineSimilarity($a, $b)
{
    try
    {
        if($a.length -eq $b.length)
        {
          $lenArr = $a.Length
               
          $sumSqB = ($b|%{[Math]::Pow($_,2)}|Measure-Object -Sum|Select Sum).Sum
          $sumSqA = ($a|%{[Math]::Pow($_,2)}|Measure-Object -Sum|Select Sum).Sum
               
          if(($sumSqB -le 0) -or ($sumSqA -le 0))
          {
               $cosineSimilarity = 0
          }elseif(($sumSqA -gt 0) -and ($sumSqB -gt 0))
          {
               $sumAB = 0
               for($i=0;$i -lt $lenArr;$i++){
                   $sumAB += ($a[$i]*$b[$i])
               }
                                     
         $cosineSimilarity = $sumAB/([Math]::sqrt($sumSqA)*[Math]::sqrt($sumSqB))
          }else{
               $cosineSimilarity = 0
          }
                return $cosineSimilarity
       }else{
           throw $("The points  have different number of dimensions")
       }
    }
    catch
    {
        Write-Host $_ -Fore Red|FL *
    }
}

Now that we have written the functions that we need, our next point of interest is getting the relevant data to work on. Which in our case would be the rss feed. Interacting with the feed is easy since it is in XML format and we can quickly build a hash out of the feed:

$objWebClient = New-Object System.Net.WebClient
$hshArticle = @{}
$feedUrl = 'https://sqlchow.wordpress.com/feed'

$feedContent = [xml]$objWebClient.DownloadString($feedUrl)

foreach($item in $feedContent.rss.channel.item)
{
 $articleTitle = $item.title
 $articleCategory = $item.category|%{$_}
 $hshArticle["$articleTitle"] = @{Tags= $articleCategory}
}

#quickly check, if we have all the post titles
$hshArticle.Keys|%{$_}

#quickly check, if we have all the tags from each post
$hshArticle.Keys|%{$hshArticle[$_].Tags}

Finally, putting it all together

$matchTags = "PowerShell", "csv","perfmon"

Foreach($key in $hshArticle.Keys)
{
  
    Write-Host $key -Fore Green
    $pntA = $hshArticle[$key].Tags| %{Get-Point $_ $matchTags}
    $dimInPntA = $pntA.Length
    Write-Host $pntA -ForegroundColor Yellow
    $pntB = 1..$dimInPntA| %{1}
    #$pntB = $matchTags|%{Get-Point $_ $hshArticle[$key].Tags}
    Write-Host $pntB -ForegroundColor Yellow
    $ed = Get-EuclideanDistance $pntA $pntB
    $cs = Get-CosineSimilarity $pntA $pntB
    Write-Host "Euclidean distance: $ed"  -Fore Cyan
    Write-Host "Cosine Similarity: $cs"  -Fore Cyan

}

And the output is as below:

Although, the output looks in line with the principals showed above, I believe the output is wrong.  Still, trying to figure out why though. Also, it is 4:20 AM and maybe I will get to this topic tomorrow.

If you find anything that can help me, kindly let me know.