Hey!

We introduce a set of advanced theoretically grounded quantization algorithms that enable massive compression for large language models and vector search engines.

From TurboQuant: Redefining AI efficiency with extreme compression by Google Research.

The practical bit is the combination: PolarQuant handles most of the compression, then QJL spends a single residual bit to correct bias. Google claims lossless-or-near-lossless KV-cache compression on long-context benchmarks, at least 6x memory reduction, and up to 8x attention-logit speedup on H100s.

It seems that graphics cards and memory prices can finally drop.

Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer’s contribution.

From Attention Residuals by Kimi Team.

Impressive.

近期几个大型机器学习模型似乎证明了，现有的科研项目已经呈现出了无法承担的成本；未来的模式可能更接近，大型资本通过海量资源先形成产品研发和应用，科研机构再顺着挖掘基础理论

新时代的贝尔实验室？

也有可能是国家集中力量，联合各家科研机构和企业攻关

最新的ChatGPT模型，运行时大概要10张A100，800g显存，显卡费用就要几十万

好像药品已经很久之前就进入这个状态了？

简化科研市场模型：产品的研发周期逐渐变长，成本与风险逐渐增加，投资回报周期变长。小型企业开始失去研发成本，