Cluster1 sources· last seen 5h ago· first seen 5h ago

[R] PCA on ~40k × 40k matrix in representation learning — sklearn SVD crashes even with 128GB RAM. Any practical solutions?

Hi all, I'm doing ML research in representation learning and ran into a computational issue while computing PCA. My pipeline produces a feature representation where the covariance matrix A^TA is roughly 40k × 40k. I need the full eigendecomposition / PCA basis, not just the top-k components. Cur

Lead: r/MachineLearningBigness: 22pca40kmatrixrepresentationsklearn
📡 Coverage
10
1 news source
🟠 Hacker News
0
🔴 Reddit
50
18 upvotes across 1 sub
📈 Google Trends
0
Full methodology: How scoring works

Receipts (all sources)

Hi all, I'm doing ML research in representation learning and ran into a computational issue while computing PCA. My pipeline produces a feature representation where the covariance matrix A^TA is roughly 40k × 40k. I need the full eigendecomposition / PCA basis, not just the top-k components. Cur