But have you wondered how it actually works? How does it understand what you’re asking, and how does it come up with a response that’s so relevant and helpful? In this blog, we’ll take a peek under the hood of ChatGPT and explore the technical details of how it works in simplest terms.

What is ChatGPT?

Think of ChatGPT like a super smart parrot that has read millions of books, websites, conversations, articles, and more. It doesn't understand the real world like humans do, but it's insanely good at mimicking human language.

ChatGPT is the super smart parrot - except instead of copying voices, it's trained to predict the next possible word in a sentence, over and over again.

What is GPT?

GPT stands for Generative Pre-trained Transformer. It's a LLM trained on massive amounts of text, and it powers Gen-AI products like ChatGPT.

• Generative: it can generate new text.

• Pre-trained: before it ever chats with you, it's trained on massive text from the internet.

• Transformer: it's a special kind of architecture - a kind of blueprint - it's used in modern AI to handle large chunks of text and figure out what's important.

GPT: It predicts the next token based on patterns learned during training, so it only knows things up to its knowledge cutoff. Ask it "Tell me today's weather" and it can't know, because it has no real-time data

ChatGPT: is the product built on top of GPT: a chat interface, extra training to follow instructions, and optional tools like web search. Ask it the same question and it can look up the weather and answer, e.g. "Today's weather in Uttar Pradesh is sunny."

How Transformer works?

1. Input: A user types a message or question into a chat interface.

Example: "How are you?".

2. Tokenization: Now the input is divided into numbers of tokens.

Example: ["How", "are", "you", "?"] - 4 tokens.

#make your own tokenizer
tokenizer={'A':1, 'a':111, 'B':2, 'b':222, 'C':3, 'c':333, 'D':4, 'd':444, 'E':5, 'e':555, 'F':6, 'f':666, 'G':7, 'g':777, 'H':8, 'h':888, 'I':9, 'i':999, 'J':10, 'j':1010, 'K':11, 'k':1111, 'L':12, 'l':1212, 'M':13, 'm':1313, 'N':14, 'n':1414, 'O':15, 'o':1515, 'P':16, 'p':1616, 'Q':17, 'q':1717, 'R':18, 'r':1818, 'S':19, 's':1919, 'T':20, 't':2020, 'U':21, 'u':2121, 'V':22, 'v':2222, 'W':23, 'w':2323, 'X':24, 'x':2424, 'Y':25, 'y':2525, 'Z':26, 'z':2626, ' ':0, ',':27, '.':28, '?':29, ':':30, ';':31, 'क':32, 'ख':33, 'ग':34, 'घ':35, 'ङ':36, 'च':37, 'छ':38, 'ज':39, 'झ':40, 'ञ':41, 'ट':42, 'ठ':43, 'ड':44, 'ढ':45, 'ण':46, 'त':47, 'थ':48, 'द':49, 'ध':50, 'न':51, 'प':52, 'फ':53, 'ब':54, 'भ':55, 'म':56, 'क़':57, 'ख़':58, 'ग़':59, 'ज़':60, 'ड़':61, 'ढ़':62, 'फ़':63, 'य':64, 'र':65, 'ल':66, 'ळ':67, 'व':68, 'ह':69, 'श':70, 'ष':71, 'स':72, 'ऱ':73, 'ऴ':74, 'अ':75, 'आ':76, 'इ':77, 'ई':78, 'उ':79, 'ऊ':80, 'ऋ':81, 'ॠ':82, 'ऌ':83, 'ॡ':84, 'ए':85, 'ऐ':86, 'ओ':87, 'औ':88, '०':89, '१':90, '२':91, '३':92, '४':93, '५':94, '६':95, '७':96, '८':97, '९':98, '0':2986387, '1':987654, '2':98765, '3':98754, '4':972352765, '5':347384638364, '6':87648764864, '7':777777, '8':8888888, '9':9999999 }
class encoder:
    def encode(str):
        encoded=[]
        for char in str:
            encoded.append(tokenizer[char])
        return encoded
class decoder:
    def decode(arr):
        decoded=[]
        for num in arr:
            decoded.append(next(k for k,v in tokenizer.items() if v==num))
        return "".join(decoded)

#converting string to tokens
tokens=encoder.encode("hey how are you?")
print(tokens)

#converting tokens back to string
s=decoder.decode(tokens)
print(s)

You can try : Tokenization

3. Vector Embeddings: As the machines can't understand human language directly, so we convert the sequence of tokens into numerical vectors or numbers simply which captures context or semantic meaning of each token. These embeddings allow model to process textual data mathematically, which helps it to understand the details and relationship of language. Similar meanings will have similar embeddings.

Example: (A) - "How" = [-0.004, -0.011, ..., 0.014], "are" =[0.013, 0.023, ..., -0.024], "you"=[0.201, 0.232, ..., 0.121], "?"=[0.121, 0.012, ..., -0.011].

(B) - the word "Good" will be represented as the set of numbers that capture its positive sentiments and common use as an adjective. That means it will be placed close to words which also have similar-meaning like "great" or "pleasant", allowing the model to understand that these words are related.

4. Positional Encoding: This is also included to help the model understand the position of a token within a sequence, ensuring the order and relative positions of tokens are understood and considered during processing.

Example: "The cat sat on the mat" and "The mat sat on the cat." are entirely different. - position matters!

5. Encoder: The encoder helps process and prepare the input data. The encoder contains two mechanisms: the self-attention and feed-forward mechanisms.

The self-attention mechanism relates every word in the input sequence to every other word, allowing the process to focus on the most important words. It's like giving each word a score that represents how much attention it should pay to every other word in the sentence.

The feed-forward mechanism is like your fine-tuner. It takes the scores from the self-attention process and further refines the understanding of each word. This helps optimize the learning process.

6. Decoder: After the encoder has done its part, the decoder takes the stage. It uses its own previous outputs — the output embeddings from the previous time step of the decoder — and the processed input from the encoder.

This dual input strategy ensures that the decoder takes into account both the original data and what it has produced thus far. The goal is to create a contextually appropriate final output sequence.

7. Output: At this stage we have got a new sequence of tokens representing the final output.

It's just like the final roar of victory from Optimus Prime after a hard-fought battle!

Some Terms:

• Softmax function: The softmax function takes a set of numbers and converts them into a set of probabilities, where the probabilities add up to 1. The model uses it to choose the next token.

  

• Multi-head attention: The self-attention mechanism is implemented using multi-head attention, which allows the model to attend to multiple parts of the input sequence simultaneously.

• Temperature: Temperature in AI models controls how random or creative their text generation is. Low temperatures (≈0–0.3) make outputs precise and factual, while higher ones (≈0.9–1.5) add creativity and unpredictability.

• Knowledge cutoff: It refers to the point in time after which an AI model has not been trained on new data.

• Vocab size: It refers to the total number of unique tokens (words, subwords, or characters) that an AI model can recognize.

In a nutshell: ChatGPT is a product built on GPT, and GPT is built on the Transformer. Text is split into tokens, turned into numbers, run through attention to figure out what matters, and the model predicts the next token over and over until the answer is complete. Terms like softmax, temperature, and vocab size are the knobs and building blocks behind that process.