Small setup differences with respect to https://github.com/MotionbyLearning/example_scripts/blob/main/solution_large_design_matrix/dask_array_sparse_mat.ipynb :

• when setting up $A$, I made sure that the start point and the end point of each arc are not the same point (am I correct in this assumption @rogerkuou?)
• when setting up $y$, I have set the probability of 0, 1 and -1 to 90%, 5% and 5%, respectively - are these reasonable @rogerkuou ?

scipy.sparse.linalg.lsqr, as np.linalg.lstsq only accepts $A$ and $x$, so more advanced optimizations (e.g. including weights) should be implemented outside these functions. It would be great to have some data/test examples where these are needed to check what can be done to implement them.

@rogerkuou I have updated the notebook, lowering the tolerance parameters in the optimization, so that the sparse approach gives identical results (e.g. results that are np.allclose) to the "dense" counterpart.

Concerning the direct comparison of A @ x to y, using the fitted x: I am not sure whether this actually makes sense. The problem that we are looking at looks more like a classification problem, with the A matrix also representing categorical values, right? So some transformations would need to be applied to A @ x in order to recover a matrix that look like the integer array y...

@fnattino Thanks a lot for the nice solution! As we already discussed, the x estimator does not make sense for now since we are randomly simutaing y in this example.

I created #6 , inheriting all your work. In that PR I made another solution notebook in ith a more sensibly simultade arcs. Now we can conclude that this solution does give mathematically sensible solution for both weight and non-weighted y, and is computationally efficient.

I will merge #6 and close this one