posts/ncm_model-are_memory-hungry_LLMs_dead

[giscus[bot]]

posts/ncm_model-are_memory-hungry_LLMs_dead

A team at Stanford decided to ditch this linear, token-by-token process and instead, they’re generating full responses across layers like diffusion models

https://codedynasty.dev/posts/ncm_model-are_memory-hungry_LLMs_dead

[Raymond-engr]

Please can you expand on this?:

Our hardware limits aren't laws of physics; they're just design choices.

[FridayCandour]

The hardware limitations we have now are by design, current llm architecture are overly memory hungry.

So a more efficient Architecture can offer more value on lower end hardware.

[Raymond-engr]

If a better and large scale efficient method of training LLM aside back propagation is discovered, it should ideally reduce the amount of GPU needed for training, right?

My question now is would that affect the architecture that current llms use to generate output entirely, making it as efficient as NCM?

Can the existing output generation architecture be used or a different one will have to be adapted to it?

I think what I'm trying to ask is if the architecture for training the LLMS and the architecture for decoding/getting a response are mutually exclusive or not.