[bot] RubyLLM::Agent agentic run not instrumented

[braintrust-bot]

Summary

RubyLLM::Agent (added in ruby_llm v1.12.0, February 2025) provides a class-based agentic execution surface that is not instrumented by this SDK. The current ruby_llm integration only patches RubyLLM::Chat — so individual Chat#complete calls made inside an agent will produce isolated spans, but there is no parent "agent run" span to tie a multi-turn, multi-tool orchestration together.

What is missing

RubyLLM::Agent is a declarative wrapper around a RubyLLM::Chat instance:

class WorkAssistant < RubyLLM::Agent
  model "gpt-4o"
  instructions "You are a helpful assistant."
  tools SearchDocs, LookupAccount
  temperature 0.2
end

agent = WorkAssistant.new
response = agent.ask("What's in the docs about billing?")

agent.ask() delegates to the underlying Chat instance and may drive multiple LLM calls + tool executions in sequence. Without a parent span, the trace shows a flat sequence of disconnected ruby_llm.chat spans with no root that conveys the agent context (initial prompt, final answer, total tokens, total cost).

What instrumentation should capture

Parent span (e.g., ruby_llm.agent):

• Input: the ask() prompt and agent configuration (model, instructions, tools, temperature)
• Output: final response after all turns complete
• Metrics: aggregated token usage across all turns

Existing pattern: The ruby_llm integration already creates nested ruby_llm.tool.{name} child spans under each ruby_llm.chat span (see lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb). A ruby_llm.agent parent span follows the same nesting convention and would make multi-turn agent traces legible in Braintrust.

Braintrust docs status

not_found — The Braintrust ruby_llm integration docs (https://www.braintrust.dev/docs/integrations/sdk-integrations/ruby-llm) cover chat completions, tool calls, token usage, and streaming, but do not mention RubyLLM::Agent or agents. Braintrust does define "function" (tool) as a first-class span type (https://www.braintrust.dev/docs/guides/traces), and an agent span is the natural root for those.

Upstream sources

• RubyLLM Agent docs: https://rubyllm.com/agents
• RubyLLM v1.12.0 release (agents introduced): https://github.com/crmne/ruby_llm/releases/tag/v1.12.0
• RubyGems: https://rubygems.org/gems/ruby_llm (v1.12.0+, well above the 1.8.0 minimum this SDK targets)

Local repo files inspected

• lib/braintrust/contrib/ruby_llm/patcher.rb — defines only ChatPatcher; no AgentPatcher
• lib/braintrust/contrib/ruby_llm/integration.rb — patchers returns [ChatPatcher] only; loaded? checks defined?(::RubyLLM::Chat) but RubyLLM::Agent also exists in v1.12.0+
• lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb — wraps Chat#complete and Chat#execute_tool; no agent wrapping