What are tokens and context length?

AI models don't read words like humans do; they process information in tokens. A token can be a single character, a part of a word (like "ing"), or a whole word. In Questsmith, you can generally estimate that 1 token is about 4 characters (or roughly 0.75 words).

1. How Tokens Work

Think of tokens as the "Lego bricks" of language. When you send a prompt, the AI breaks it down into these bricks, analyzes the patterns, and predicts which brick should come next. This is why you might occasionally see the AI struggle with very complex or made-up words—it's trying to find the best token match.

2. What is Context Length?

Context Length is the AI's "short-term memory." It represents the maximum number of tokens the model can "see" at one time when generating a response. Once your story exceeds this limit, the AI begins to "forget" the earliest parts of the adventure to make room for new actions.

Pro Tip: To keep your story coherent even when you hit your context limit, use Story Cards and Memory. These "pinned" details are always sent to the AI, ensuring it remembers your quest goals and character names even if the actual text of Chapter 1 has scrolled out of its context window.