What is a "GPT" in AI?
GPT is an abbreviation for Generative Pretrained Transformer. A transformer is a neural network architecture for large language models (LLM's); it's generative because its purpose is to generate text; and pretrained refers to a technique employed in the training of the model, in which it's first trained unsupervised on a large dataset, and then goes through a supervised training for specific tasks.
GPT is in the name of some AI-powered software products, such as ChatGPT. An AI model based on this architecture is also called a GPT model. Note that while all GPT models are LLM's, not all LLM's are GPT's.
Etymology
The term "transformer" was introduced to the literature by the paper Attention Is All You Need (2017), by Google Brain.
The dominant sequence transduction models are based on complex recurrent or
Attention Is All You Need [https://doi.org/10.48550/arXiv.1706.03762]
convolutional neural networks that include an encoder and a decoder. The best
performing models also connect the encoder and decoder through an attention
mechanism. We propose a new simple network architecture, the Transformer,
based solely on attention mechanisms, dispensing with recurrence and convolutions
entirely. Experiments on two machine translation tasks show these models to
be superior in quality while being more parallelizable and requiring significantly
less time to train.
OpenAI released an LLM called "GPT-1" in 2018. This is a GPT model. They also published a paper titled Improving Language Understanding by Generative Pre-Training that explains their technique.
In this paper, we explore a semi-supervised approach for language understanding tasks using a
Improving Language Understanding by Generative Pre-Training [https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf] (accessed 2025-02-21)
combination of unsupervised pre-training and supervised fine-tuning.
In essence, they found that training was traditionally done in a supervised manner, which was laborious since the dataset had to be labelled in order to be usable. Unsupervised training means they can simply give the training program all the data without any labelling, cross their fingers, and hope the AI starts figuring out some patterns by itself.
This isn't adequate as the result will fail to achieve specific goals the researchers would have in mind for the AI, but it does allow them to use vast amounts of data that would be impossible to use if they had to manually label or process the data beforehand somehow—for example, they can just use all the text from the Internet without having to read any of the text themselves.
After the unsupervised training ends, they have a model that isn't very good for specific goals, but does have a good general understanding of all patterns in the dataset. The researchers then begin a supervised training with this pre-trained model in order to achieve more specific goals, such as answering questions or classifying texts.
In other words, the "pre-training" gives them a model with a configuration that is a good starting pointing for the main training, allowing the main training to take less time and achieve better results than it would if it started with an untrained model.
We introduced a framework for achieving strong natural language understanding with a single
Improving Language Understanding by Generative Pre-Training [https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf] (accessed 2025-02-21)
task-agnostic model through generative pre-training and discriminative fine-tuning. By pre-training
on a diverse corpus with long stretches of contiguous text our model acquires significant world
knowledge and ability to process long-range dependencies which are then successfully transferred to
solving discriminative tasks such as question answering, semantic similarity assessment, entailment
determination, and text classification, improving the state of the art on 9 of the 12 datasets we
study.
Surf the Web
- NanoGPT [https://github.com/karpathy/nanoGPT/] (accessed 2025-02-21), the simplest Python script for fine-tuning a GPT model.