Conservative, tokenizer-free token estimate: about 4 ASCII characters per token, and one token per
non-ASCII code point (CJK, emoji and accented text tokenize far worse than English). Deterministic and
monotonic, so it is safe for budgeting; it over-estimates rather than under-estimates.
Conservative, tokenizer-free token estimate: about 4 ASCII characters per token, and one token per non-ASCII code point (CJK, emoji and accented text tokenize far worse than English). Deterministic and monotonic, so it is safe for budgeting; it over-estimates rather than under-estimates.