Training Made Easy
pico-train makes the process of training models simple and efficient.
With pico-train, you can train language models of various sizes with minimal configuration. The framework handles the complexities of distributed training, gradient accumulation, and checkpoint management, allowing researchers to focus on experimenting with model architectures and training paradigms.
Small-Scale Focus
Train and study models from 1M to 1B parameters, making experimentation with training paradigms practical and accessible.
Advanced Checkpointing
Access model activations, gradients, and other rich information throughout training for mechanistic interpretability research.
Easy Retraining
Simple, modular codebase designed for researchers to modify and retrain the entire model suite with custom training paradigms.
PyTorch Lightning
Built on PyTorch Lightning for efficient, scalable training with minimal boilerplate code.
Minimal Dependencies
Lightweight framework with only essential dependencies, making it easy to install and modify.
Research Ready
Designed with researchers in mind, providing tools and flexibility needed for academic exploration.