Glossary · AI
What is
Pretraining?
The initial training of a foundation model on massive amounts of unlabeled text.
By Anish· Founder · Vedwix
·Definition
Pretraining teaches a model the patterns of language by predicting the next token across trillions of tokens of internet, book, and code data. The result is a "base model" with broad knowledge but no instruction-following behavior. Pretraining is enormously expensive (millions to hundreds of millions of dollars) and usually only frontier labs do it from scratch.
Example
Llama 3 was pretrained on 15 trillion tokens from public web data, books, code, and more.
How Vedwix uses Pretraining in client work
We never pretrain. We always start from a strong base and fine-tune.
Building with Pretraining?
We ship this.
If you're building with Pretraining in production, we can help — from architecture review to full implementation.
Brief us