PyTorchPipe (PTP) fosters the development of computational pipelines and comparison of diverse neural network-based models.

PTP frames training and testing procedures as pipelines consisting of many components communicating through data streams. Each such a stream can consist of several components, including one problem instance (providing batches of data), (zero-or-more) trainable models and (any number of) additional components providing required transformations and computations.

As a result, the training & testing procedures are no longer pinned to a specific problem or model, and built-in mechanisms for compatibility checking (handshaking), configuration management & statistics collection facilitate running diverse experiments.

In its core, to accelerate the computations on their own, PTP relies on PyTorch and extensively uses its mechanisms for distribution of computations on CPUs/GPUs.

PTP relies on PyTorch, so you need to install it first. Refer to the official installation guide for its installation. It is easily installable via conda_, or you can compile it from source to optimize it for your machine.

PTP is not (yet) available as a pip package, or on conda. However, we provide the setup.py script and recommend to use it for installation. First please clone the project repository::

git clone [email protected]:IBM/pytorchpipe.git
cd pytorchpipe/

Then, install the dependencies by running::

python setup.py install

This command will install all dependencies via pip_.

NOTE

If you plan to develop and introduce changes, please call the following command instead::

  python setup.py develop

This will enable you to change the code of the existing components/workers and still be able to run them by calling the associated ptp-* commands. More in that subject can be found in the following blog post on dev_mode.

A project of the Machine Intelligence team, IBM Research, Almaden.

• Tomasz Kornuta ([email protected])