I tested encryption of complete VMs using phpVirtualBox 5.1-0 and Virtual Box 5.1.22 and while the encryption seems to work as expected, decryption doesn't. If I uncheck the activation checkbox in the encryption tab, the inputs for cipher and new passwords are deactivated. After pressing OK, another dialog opens and asks for a password, which most likely is the formerly set encryption password. I input the password, press OK and a decryption progress bar is shown. Very few seconds afterwards the following error is shown. Decryption fails this way, the VMs and all attached disks stay encrypted.

The password and password identifier must be empty if the output should be unencrypted

I think the problem is that you seem to provide the password ID on decryption, which seems to be wrong according the code and docs. It's only a guess of course, but the old password is definitely needed and you dialogue asking for that password is showing the password ID as well. So I guess that is the problem.

In some circumstances it might be required to decrypt previously encrypted images.
This can be done in the GUI for a complete VM or using VBoxManage with the following command:

VBoxManage encryptmedium "uuid|filename" --oldpassword "file|-"

https://www.virtualbox.org/manual/ch09.html#diskencryption-encryption

As you can see, no password ID. Additionally, the code where the error is thrown:

if (task.mstrCipher.isNotEmpty())
{
[...]
}
else if (task.mstrNewPasswordId.isNotEmpty() || task.mstrNewPassword.isNotEmpty())
    throw setError(VBOX_E_INVALID_OBJECT_STATE,
                            tr("The password and password identifier must be empty if the output should be unencrypted"));

https://www.virtualbox.org/browser/vbox/trunk/src/VBox/Main/src-server/MediumImpl.cpp#L10056

error details.txt

I've created an issue on Sourceforge first, until I found this here.