Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts

MoRAM — rank-1 parametric memory as a dynamic mixture-of-experts for continual learning of LLMs.