How Many Tokens Did You Spend on This Task?

“How many tokens did you spend on this task?”

This is the question I ask when someone starts telling me that a task is complex, unsolvable, or that they don’t understand how to approach it.

Of course, you can’t get an exact answer to this question. Unless you’re using my LLM accounting and observability system, but that’s still in development.

But this question forces you to verify that the person at least tried to understand the unfamiliar subject with an LLM’s help.

If someone says something like “I don’t use AI,” then continuing the conversation with them is probably pointless. They’re unlikely to solve tasks that their competitor can solve using modern technology.

If someone can roughly estimate the volume of computation the LLM performed to solve the task, that’s telling. For example, they might say how many times they hit Claude’s rate limit. Or how many sessions they had and roughly how large those sessions were.

I believe task complexity can be measured by the volume of tokens required to solve it. There should be some minimum volume that needs to be spent to grasp the conditions of any task, even one that seems simple at first glance. Assuming efficient usage, of course.

It would be good to start tracking this somehow.