NAK. IMHO CTxDestination belongs in script, as it encodes a particular subset of scripts. key.h is a wrapper for ECDSA logic, and works at a lower level.

Agree with @sipa here.

key.h isn't concerned with sending and destinations. There is no functionality using destinations at all in there. It's very low-level.

In script.h however, there are various utility functions that take a destination. Apart from the wallet it has the most methods involving these:

void CScript::SetDestination(const CTxDestination& address);
isminetype IsMine(const CKeyStore& keystore, const CTxDestination& dest);
bool ExtractDestination(const CScript& scriptPubKey, CTxDestination& addressRet);
bool ExtractDestinations(const CScript& scriptPubKey, txnouttype& typeRet, std::vector<CTxDestination>& addressRet, int& nRequiredRet);

Unless these utilities are somehow split off from the 'core' script interpreter, it makes no sense to move CTxDestination.