That's pretty innovative. But a brand new tokenizer and training method need a lot of effort to do the retraining. Whether it will be widely implemented depends on real frontier performance it can reach.
I hope Apple Silicon will one day be competitive with GPUs. It dominates much of the CPU market, and it has a GPU, but it's still not even remotely comparable to Nvidia for inference (factor of >5 in some cases, especially in prefill) or even for gaming ("edge computing" in Nvidia terms).
I feel like their GPU architecture is going to need a significant redesign. There are limits to this tile-based mobile architecture.
I keep getting clickbait youtube videos about how "NCM" obsoletes "LLMs" and "Nvidia memory chips" (yes, it's that bad).
This makes me wonder what these people think an "LLM" even is. The name itself implies that it is large and about language. It does not imply that an "LLM" is inherently autoregressive or that it has any particular architecture even though most LLMs are transformers. Anything can be a large language model as long as it is large and models language.
NCM is basically a hierarchical diffusion transformer where the further layers take one big token and then generate two smaller tokens. So it is more like an evaluation strategy. You could apply this to any model architecture.
I personally think the name NCM itself is terrible because it contains the word "next" which inherently implies an ordering and somehow we are supposed to think that "LLM" which contains no mention of ordering, is constrained to ordered token production.
I feel like their GPU architecture is going to need a significant redesign. There are limits to this tile-based mobile architecture.
This makes me wonder what these people think an "LLM" even is. The name itself implies that it is large and about language. It does not imply that an "LLM" is inherently autoregressive or that it has any particular architecture even though most LLMs are transformers. Anything can be a large language model as long as it is large and models language.
NCM is basically a hierarchical diffusion transformer where the further layers take one big token and then generate two smaller tokens. So it is more like an evaluation strategy. You could apply this to any model architecture.
I personally think the name NCM itself is terrible because it contains the word "next" which inherently implies an ordering and somehow we are supposed to think that "LLM" which contains no mention of ordering, is constrained to ordered token production.