BigWeed [none/use name]

  • 1 Post
  • 122 Comments
Joined 7 months ago
cake
Cake day: July 12th, 2025

help-circle









  • Hey, good job, looks cool. A few observations: It makes sense to aggressively use token caching. It makes sense to improve context accuracy by treating docs as external to the input prompt. Smaller context = more calls, faster CoT, higher llm concurrency. We see a lot of this now, where coding agents aggressively cache documents, and use of tools to do more complicated tasks, e.g. code gen as tool use. I also see more moving away from RAG and more towards using good ole gnu tools like grep as exploration hints. RAG approaches are definitely is more prone to contextual bandit problems. But IMO, balancing exploration with exploitation is fundamentally a reinforcement learning problem rather than a linear flow from recursive llm calls, but it acts a reasonable surrogate model, and our RL algorithms consistently fall short. For example, you could help govern exploration behavior using deep-q or ppo to guide search, e.g. have your llm produce a top-5 exploration s-expression horizon, feed the exploration into the rl algorithm with cosine similarity on vectors + metadata (file size, LOC, etc) as features, then rank the exploration space. This would further reduce token count and increase parallelization in exploration (since top-n results are computed as batch in the llm with only marginal overhead).

    Personally, I just wish I had a good search tool that didn’t expose all of the LLM tomfoolery. For example, today I needed to find where a particular sql table was defined in the code since it used some crazy ORM and an initial grep didn’t yield results so I jammed into openai codex and it found it. This was useful to me where github search was useless. I need more context-aware search, and it would be better if I could combine it with all of the other docs and slack convos floating around. Actual coding tasks require a greater level of human interaction where opaque results are less useful.

    But again, good job, looks cool.








  • “If it’s raining outside then we set the price of umbrellas to its maximum, $20.” One analyst said. “As long as there’s an ATM in the store, customers won’t leave when they run out of money.” The analyst goes on to suggest this periodical cash influx is the only way the store is staying afloat, citing poor management and excessive building budget.