A transformer language model pretrained from scratch in PyTorch, then instruction-tuned to chat. It is the third in an eight-month line of from-scratch language models, after an autocomplete model and a first chat model, each one a little less wrong than the last.
The goal was never to compete with frontier models. It was to understand the whole stack by building it: tokenization, the transformer itself, a pretraining loop, and then supervised fine-tuning to turn a next-token predictor into something you can actually talk to. You can chat with the result in the live demo, hosted as a Hugging Face Space.