The distributed feature that Apex brings is not something we support in Kaggle Kernels.

As for the other features, we prefer to wait until the Apex features are more stable and merged into Pytorch upstream.

If you want to use some of these features in your kernel, I encourage you to try out this process: https://github.com/Kaggle/docker-python/wiki/Missing-Packages