GPTDAOCN-e/acc on Nostr: WOW. Meta just open-sourced a Github repo for LLM Training. Meta Lingua is a minimal ...
WOW. Meta just open-sourced a Github repo for LLM Training.
Meta Lingua is a minimal and fast LLM training and inference library designed for research
📊 Key features
- Minimal and fast LLM training/inference library for research
- Uses modifiable PyTorch components for experimenting with architectures, losses, data
- Enables end-to-end training, inference, evaluation
- Provides tools for understanding speed and stability
- Structured with core 'lingua' library and 'apps' to showcase usage
🚀 Lingua's performance comparison to other models
- 1B models trained on 60B tokens match DCLM (DataComp-LM) baseline performance on many tasks
- 7B models (Mamba, Llama) show strong results on benchmarks like ARC, MMLU, BBH
- Llama 7B squared ReLU 1T tokens model achieves high scores across tasks
Meta Lingua is a minimal and fast LLM training and inference library designed for research
📊 Key features
- Minimal and fast LLM training/inference library for research
- Uses modifiable PyTorch components for experimenting with architectures, losses, data
- Enables end-to-end training, inference, evaluation
- Provides tools for understanding speed and stability
- Structured with core 'lingua' library and 'apps' to showcase usage
🚀 Lingua's performance comparison to other models
- 1B models trained on 60B tokens match DCLM (DataComp-LM) baseline performance on many tasks
- 7B models (Mamba, Llama) show strong results on benchmarks like ARC, MMLU, BBH
- Llama 7B squared ReLU 1T tokens model achieves high scores across tasks