Self-Supervised Learning (SSL) is a machine learning paradigm where a model learns from the input data itself without requiring explicit human-annotated labels.
In Large Language Models
In the context of Large Language Models, self-supervised learning is the core mechanism used during the Pre-training stage. The model is trained to predict the next token in a sequence based on the context of preceding tokens. The “ground truth” or label is automatically derived from the text itself—specifically, the actual next word in the sequence serves as the target.
This approach allows LLMs to leverage vast amounts of raw text data for training, as obtaining manual labels for such a scale would be infeasible.
