Is the model "thinking of the last word for the next line already" or is "the word that came last in the second line was already showing high p from considering only the first line"? Shouldn't we care about such a distinction?
If “aim” is already highly probable after the first line, the example is much less interesting.
A more cool question is whether the model is carrying a latent representation of the destination that isn’t yet reflected in the immediate token probabilities?
A more cool question is whether the model is carrying a latent representation of the destination that isn’t yet reflected in the immediate token probabilities?
I don’t know. Anthropic is investigating: https://www.anthropic.com/research/tracing-thoughts-language...