**UPDATE: it was recently brought to my attention by Roman Rubsamen who does an excellent job of curating the large body of research in optimization and mathematics and of Portfolio Optimizer that the same general methodology was created and extensively tested by Higham in 2016 (the shrinkage targets in IPS and correlation thresholds are different). Higham’s paper demonstrates the usefulness of the general approach on large matrices and the speed of convergence for different variations. While I was not aware of this paper when I created IPS I am very pleased that there is strong validation for this approach in the literature. Roman’s excellent Portfolio Optimizer is a free Web API that provides access to advanced portfolio optimization algorithms and is really worth checking out.

In the previous post the framework for Generalized Downside Implied Correlations was introduced. You can use this correlation matrix derived from joint risk metrics to replace or augment/blend with traditional correlations for use in analysis or optimization. The challenge is that the resulting matrix may not be positive semi-definite (PSD) or well-conditioned. The solution is to find a reasonably fast and simple solution that can gradually adjust or shrink the correlation matrix while still ensuring PSD.

Shrinkage methods are important because they produce a well-conditioned, invertible matrix, which is crucial for applications like optimization. Traditional shrinkage methods such as Ledoit-Wolf seek to improve covariance matrix estimates. Linear versions primarily reduce eigenvalue variance by shrinking the sample covariance matrix towards a target (like the identity matrix). Nonlinear versions use a spectral transformation, often involving the Hilbert transform, to adjust the magnitude of each eigenvalue individually. This data-driven approach seeks to minimize both variance and bias in the eigenvalue estimates, leading to more accurate covariance matrices. Imagine the sample covariance matrix as a noisy picture. Linear Ledoit-Wolf tries to ‘smooth’ this picture by blending it with a simpler, clearer picture (like a blurry version of the original). Nonlinear Ledoit-Wolf, on the other hand, uses a more sophisticated technique. It analyzes the noise in different parts of the picture and adjusts the smoothing level accordingly. This targeted approach allows for a clearer and more accurate representation of the underlying reality. While the theory generally supports PSD results, numerical issues in the implementation can sometimes lead to very small negative eigenvalues. While both methods reduce the chance of negative eigenvalues, they fail to guarantee PSD.

The Iterative PSD Shrinkage (IPS) combines both gradual shrinkage with explicit PSD enforcement using a post-shrinkage check such as the Cholesky decomposition.

The primary goal of this method is to adjust the correlation matrix as little as possible by shrinking to the target matrix to preserve the original relationships while still ensuring PSD. The framework is quite simple, flexible and computationally efficient. As a result it is well-suited to specialized correlation structures such as reverse-implied downside risk correlations. Iterative PSD shrinkage is fast but generally not the fastest method compared to either shrinkage methods that use analytical solutions or other methods for ensuring a matrix is Positive Semi-Definite (PSD). However, it does not require the distribution assumptions that analytical methods assume and converges quickly while offering more control and precision.

Iterative PSD Shrinkage/IPS Steps

1. Start with Your Data:
   • Gather the sample covariance or correlation matrix (S) from your data
2. Define a Target Matrix:
   • Create a target matrix (T) to shrink toward. Or automatically determine the target within the algorithm based on average correlation in the universe.
   • Examples:
   • Constant correlation matrix: A simple, average correlation structure. This is ideal when the average correlation of all assets in the universe is homogeneous or >.5
   • Identity matrix: Assume no correlation between assets so correlations =0. This is ideal when the average correlation of all assets in the universe is heterogeneous or <.5

3. Set Initial Shrinkage Level λ Lambda:

• Calculate an initial shrinkage parameter using a simple heuristic for shrinkage intensity that captures variance or noise/instability in the matrix scaled by the strength of correlations:

4. Combine Matrices:

• Shrink SSS toward T using a weighted blend:

5. Check for Positive Semi-Definiteness (PSD):

• Test if the matrix is PSD using Cholesky decomposition. If the decomposition fails due to having negative eigenvalues the matrix isn’t PSD.

6. Adjust λ Lambda:

• Increase Lambda by a small amount using alpha and repeat the blending process. This shrinks the S closer to the T.

7. Repeat Until PSD:

• Continue adjusting Lambda iteratively until the adjusted matrix is PSD.

8. Finalize the Matrix:

• Once PSD is achieved, the resulting matrix is your shrinkage-enhanced matrix. This is the final matrix used for optimization.

You can build more complex methods within IPS such LW shrinkage or even using projections onto the nearest correlation matrix that is PSD which is used in Higham’s Algorithm. You can also change the shrinkage intensity heuristic to use the difference between observed and historical correlation matrices. The general idea of IPS is to constantly test and adjust iteratively to ensure PSD. Using cross validation or walk-forward testing can help determine the best choice of alpha and also the best methods/targets to use within the IPS framework. However this simple approach provides the crucial advantage of being easy to understand and implement.