Building small language models remains more of an art than a science. We believe it shouldn't be.
Pico provides a lightweight, modular framework for systematic, hypothesis-driven research. Built around two core libraries: pico-train for model training and pico-analyze for in-depth analysis, Pico creates a sandbox for researchers to develop and test new ideas.
pico-train makes the process of training models simple and efficient.
With pico-train, you can train language models of various sizes with minimal configuration. The framework handles the complexities of distributed training, gradient accumulation, and checkpoint management, allowing researchers to focus on experimenting with model architectures and training paradigms.
Train and study models from 1M to 1B parameters, making experimentation with training paradigms practical and accessible.
Access model activations, gradients, and other rich information throughout training for mechanistic interpretability research.
Simple, modular codebase designed for researchers to modify and retrain the entire model suite with custom training paradigms.
Built on PyTorch Lightning for efficient, scalable training with minimal boilerplate code.
Lightweight framework with only essential dependencies, making it easy to install and modify.
Designed with researchers in mind, providing tools and flexibility needed for academic exploration.
pico-analyze provides comprehensive tooling to capture and analyze training metrics, enabling researchers to understand how models learn.
Out of the box, pico-analyze provides a suite of tools to capture and analyze training metrics, including:
Using checkpoints from pico-train, the analysis framework pico-analyze lets you extract critical insights about model behavior throughout training. These insights can help identify optimization issues and guide architectural improvements.
We use pico-train to train a family of pico-decoder models on the carefully pre-processed Dolma dataset. All models are trained identically with scale as the only difference, accompanied by rich training checkpoints for interpretability research.
*All models are trained on 400B tokens of our carefully pre-processed Dolma dataset. We've tokenized, chunked into 2049-token chunks, and shuffled the data, making it ready for immediate use with no additional preprocessing needed!
load_dataset('pico-lm/pretokenized-dolma', streaming=True)These papers use the Pico framework to study learning dynamics in language models.
Built with ❤ by the Pico team
Code and Artifacts are licensed under Apache License 2.0