In this article I introduce Python.Included which solves this problem elegantly and makes Python-interop fun and easy for .NET developers. As a proof-of-concept I present Numpy.NET which is a .NET Standard library that provides a strong typed API to Python’s NumPy package and does not require a local Python installation on Windows.

What is the problem?

Everyone has a different Python installation. Some still use Python 2.7, some use Python 3.5 or 3.6, and some are already on 3.7. When you use pythonnet it has to be compiled with different settings for every minor Python version and that version has to be installed for the code to run. So if you are working in a team, everyone is forced to have the same Python setup. With our SciSharp team, for instance, this is already not the case. If you want to deploy your .NET application on a machine you have to deploy Python first. From the perspective of a .NET developer it kind of sucks.

And yet, if you are working on Machine Learning and Artificial Intelligence, despite the efforts of Microsoft and SciSharp, at the moment you simply can not completely avoid Python. If you check out the list of projects using pythonnet it is evident that many companies in the AI field currently interface .NET with Python.

Python.Included to the rescue

But what if you could simply reference a Nuget package and everything is automatically set up correctly without additional manual tinkering? That was the vision that led me to create Python.Included which packages python-3.7.3-embed-amd64.zip in its assembly effectively allowing to reference Python via NuGet.

To prove that it works and to quickly provide all numpy-functionality that is still missing in NumSharp, I created Numpy.NET which is built upon Python.Included.

Proof-of-concept: Numpy.NET

Numpy.NET provides strong-typed wrapper functions for numpy, which means you don’t need to use the dynamic keyword at all, but this is a rabbit hole to delve into in another article. Instead, we focus on how Numpy.NET uses Python.Included to auto-deploy Python on demand and the NumPy package in order to call into it.

This is the setup code that Numpy will execute behind the scenes. You don’t have to do any of this. As soon as you are using one of its functions, i.e.

var a = np.array(new [,] {{1, 2}, {3, 4}});,

Numpy.dll sets up the embedded Python distribution which will unpack from the assembly in your local home directory (only if not yet installed):

var installer = new Python.Included.Installer();
installer.SetupPython(force:false).Wait();

Next (if not yet done in a previous run) it unpacks the numpy pip wheel which is packed into Numpy.dll as embedded resource and installs it into the embedded Python installation.

installer.InstallWheel(typeof(NumPy).Assembly, "numpy-1.16.3-cp37-cp37m-win_amd64.whl").Wait();

Finally, the pythonnet runtime is initialized and numpy is imported for subsequent use.

PythonEngine.Initialize();
Py.Import("numpy");

All this is happening behind the scenes and the user of Numpy.dll does not have to worry about a local Python installation at all. In fact, even if you have installed any version of Python it won’t matter.

Performance considerations

Pythonnet is known for being slow, so you might ask yourself if it is really a good idea to interface Python libraries with .NET using pythonnet. As always, it depends.

My measurements show that the overhead of calling into numpy with .NET compared to calling it directly from Python is about a factor 4. To be clear, this does not mean that Numpy.NET is four times slower than numpy in Python, it means that there is an overhead of calling through pythonnet. Of course, since Numpy.NET is calling numpy the execution-time of a NumPy function itself is exactly the same.

Whether or not that overhead is a problem entirely depends on the use case. If you are going back and forth between CLR and Python from within a nested loop you might have a problem. But mostly Python libraries are designed to be efficient and avoid looping over data. Numpy allows you to run an operation on millions of array elements with only one call. PyTorch and Tensorflow allow you to execute operations entirely on the GPU. So when used correctly, the interop-overhead will be negligible compared to the execution time of the operations when dealing with large amounts of data.

Roadmap

I know that there are a number of numpy ports for .NET, for instance via IronPython. But the IronPython project is still only supporting Python 2.7 and progressing very slowly. Libraries that depend on Python 3 are not available through IronPython and will not be in the near future.

My focus will be to make more Machine Learning and AI libraries available for .NET through pythonnet. The SciSharp team is also discussing ideas to make a faster version of pythonnet which avoids the use of the inherently slow DynamicObject.

Please try out Numpy.NET and let me know how it worked out for you. I’ll be grateful for any comments or suggestions and I hope that my work will help the .NET Machine Learning community to grow and prosper.

Python is the language for Machine Learning in part because of its great libraries like NumPy and TensorFlow. However, C# developers are terribly in need of powerful Open Source libraries for Data Science too. NumSharp, being a best-effort C# port of NumPy, developed by the SciSharp STACK organization, has recently taken a huge step forward by fully implementing slicing which allows creation of arbitrary sub-sets of N-dimensional arrays as highly efficient views over the original data. This makes it a useful tool for Machine Learning in C# in conjunction with TensorFlow.NET.

What is the big deal?

If you haven’t worked with NumPy you might not know how awesome slicing is. Python arrays allow to return a slice of an array by indexing a range of elements like this: a[start:stop:step]. But only with NumPy’s sophisticated array implementation slicing becomes a really powerful data manipulation technique without which Machine Learning or Data Science can not be imagined any more.

Luckily for those who can not or do not want to switch to Python for Machine Learning – and I am guilty of that charge too – NumSharp brings that power into the .NET world. As one of the developers of NumSharp, I present to you a few important use cases for slicing with exemplary code snippets in C#. Note that in C# is not possible to index in the same way as in Python due to differences in the language syntax. We decided to keep Python syntax for slice definitions however, so we use strings to index slices in C#. Check out this example and see how close NumSharp gets to NumPy.

>>> import numpy as np
>>> a=np.arange(16).reshape(4,4)
>>> a
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15]])
>>> a[:,1:3]
array([[ 1,  2],
       [ 5,  6],
       [ 9, 10],
       [13, 14]])

When written in C# with NumSharp the code is almost the same. Note the slight difference where the slice is indexed using a string as parameter for the indexer.

using NumSharp;
// class definition ommitted for brevity
var a=np.arange(16).reshape(4,4);
var slice=a[":,1:3"];

As you can see, the NumSharp team put a lot of effort in to keep the code as similar to Python as possible. This is very important because this way, existing Python code that relies on NumPy can now easily be ported to C#.

Use case: Working with multiple Views of the same Data

Being able to pass only local portions of the underlying data (i.e. small patches of a big image) in and out of functions without copying is essential for runtime-performance, especially with big data sets. A slice is indexed with local coordinates, so your algorithms don’t need to know about the global structure of your data, effectively simplifying your life and ensuring highest possible performance due to avoiding unnecessary copying.

Use case: Sparse Views and Recursive Slicing

By specifying a step in addition to start and stop of the slice range sparse views of the array can be created. This is something that not even C# 8.0 with its new array slicing syntax can do (to my knowledge). When working with interleaved data this feature becomes incredibly important. You can keep the complexity of your algorithms as low as possible by designing them to work on contiguous data and feeding them a sparse slice which simulates a contiguous data source.

Slices can be further sliced which is a very important feature if you work with high-dimensional data. This also helps reduce algorithmic complexity as you reduce the dimensions of your data by recursive slicing.

Use case: Efficiently handling High-Dimensional Data

If you need to treat a data array as a volume and work with parts of it without having to do mind-boggling coordinate transformation calculations, then .reshape() is your friend.
All arrays created by .reshape() or slicing operations are only views of the original data. When you iterate over, read or write elements of a view you access the original data array. NumSharp transparently does the appropriate index transformations for you, so you can index into the slice using relative coordinates.

Use case: Reversing the order of elements at no extra cost

Slicing with a negative step is effectively reversing the slice’s order. What’s nice about it is, that it requires no copying or enumeration of the data to complete this operation much like IEnumerable.Reverse(). The difference is, that the view (which is the result of the operation a["::-1"]) presents the data in reversed order and if you can index into that reversed sequence without ever having to enumerate it at all.

Use case: Reducing complexity by reducing dimensions

When working with high-dimensional data, algorithms on that data can get highly complicated too. When working on the .ToString() method of NumSharp’s NDArray which can print out arbitrarily high-dimensional volumes I noticed how simple and beautiful that algorithm gets by systematically and recursively slicing the ND-volume in (N-1)D-volumes and so forth.

This divide et impera approach is made possible by slicing with NumSharp’s indexing notation over the range notation which returns lower-dimensional sub-volumes.

Range Notation vs. Index Notation

The range notation [“start:stop:step”] allows you to access a sub range of the given volume with the same dimensionality. So even slicing out only one column of a 2D matrix still gives you a 2D matrix with just one column.

var matrix = np.arange(16).reshape(4,4);
Console.WriteLine(matrix);

results in

array([[0, 1, 2, 3],
       [4, 5, 6, 7],
       [8, 9, 10, 11],
       [12, 13, 14, 15]])

slicing the 3rd column with range notation

var column = matrix[":,2:3"];
Console.WriteLine(column);

results in a 2D matrix

array([[2],
       [6],
       [10],
       [14]]

The index notation gives you a (N-1)-dimensional slice at the specified position of the N-dimensional parent volume. So carving out a column from a 2D matrix with index notation gives you a 1D vector:

var column1 = matrix[":,2"];
Console.WriteLine(column1);

results in

array([2, 6, 10, 14])

If you haven’t spotted the difference at a glance, here the two slicing definitions from above side by side: range [":,2:3"] vs index [":,2"] which makes a big difference. A full reference of the new slicing notation will be available at the NumSharp wiki.

Side Note: ArraySlice<T>

While implementing slicing of N-dimensional views, I came to the conclusion that it might be interesting for a whole lot of other domains in .NET, so I factored it out into its own standalone library called SliceAndDice. It features ArraySlice<T> which is a light weight wrapper around any indexed C# data structure (like T[] or IList<T>) and allows you to make use of the same reshaping, slicing and view mechanics without all the other heavy numerical computation stuff. It is a nice and clean implementation of slicing awesomeness in just a few hundred lines of code!

Summary

NumSharp has just recently been empowered with the same slicing and view mechanics that arguably make NumPy one of the most important libraries of Python’s Machine Learning ecosystem. SciSharp STACK, being an Open Source organization consisting of only a handful of skilled developers, tries very hard to bring that same power to the .NET world. This recent improvement of NumSharp is an important stepping stone towards this goal.