Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)
66 points
0 comments
Haha. somehow i just love the naming. it just makes sense :D
This looks super clean. I'm curious about the --judge command. How does it evaluate if the cheaper model's response is a "tie" or acceptable? Is it using a specific LLM-as-a-judge prompt template?
Useful but how it compares with other providers model
I think this is great, and the next frontier is to analyze how well calls can be handled by a local model. Realistically, to do that usefully requires response time as a new dimension of judging: Can a local model provide an acceptably accurate response in an acceptable amount of time?
What we need is an AI gateway/router that will actually first analyze the input tokens and then decide what model to use. If it's so simple that a dirt cheap qwen 3.5 flash or whatever will be fine, then it chooses that. If it deems we need GPT 5.6, then it uses that, etc. does anything like this already exist?
Evals and OpenInference (OpenTelemetry) might be useful.Costed opcodes (like the shelved eWASM opcodes cost chart) would be useful for this model routing problem as well.Is this the cost to converge problem, the minimize cost to converge upon sufficiently low error problem, or the minimize cost and error problem?EA methods: mutation, crossover, selectionGradient descent as a mutation, crossover, and selection pattern; back up when the error/cost stops decreasing for too long and try a different branch.A simple experiment: vary only a nonce in the prompt and compare output value. The nonce is a parameter. The model is a hyperparameter.
A cheaper model can look fine per request, but one weak answer may create another call or a human review step. That seems like the hardest cost to capture.
this would be more interesting as a local LLM anlysis; throw out all the costs, and figure out primary-subagent model architecture, and maximize token generation and prefill.I don't see how anyone can operationalize this information.