The culprit is the fact that compiler runs without --initialize-from-dill and --incremental options, therefore can't reuse previously created dill file.

On a sample app passing both of those options to the compiler results in dropping build time from ~10s to ~2s.