[bot] ruby_llm Chat span metadata misses accessible generation parameters (tool_prefs, params, schema) and omits tools from streaming spans

[braintrust-bot]

Summary

The ruby_llm Chat integration captures only 3–4 fields in span metadata (provider, stream, model, and tools in non-streaming). The RubyLLM::Chat class exposes several accessible attr_reader attributes that contain key generation configuration — tool_prefs, params, and schema — none of which are captured. Additionally, tools is explicitly omitted from streaming spans even though it is an accessible attr_reader.

This is a significant detail gap compared with the OpenAI and Anthropic integrations, which each capture 15–20 configuration fields per span.

What is missing

1. tool_prefs not captured (accessible via attr_reader)

RubyLLM::Chat exposes attr_reader :tool_prefs, which is a hash with two keys:

• tool_prefs[:choice] — tool choice mode (:auto, :required, :none), set via chat.with_tools(..., choice: :required). Introduced in ruby_llm v1.13.0.
• tool_prefs[:calls] — parallel tool call limit, set via chat.with_tools(..., calls: 1). Also introduced in v1.13.0.

These correspond directly to tool_choice and parallel_tool_calls that the OpenAI and Anthropic integrations capture.

2. params not captured (accessible via attr_reader)

RubyLLM::Chat exposes attr_reader :params, a hash of provider-specific generation parameters set via chat.with_params(max_tokens: 1000, top_p: 0.9, ...). This is the primary way users configure generation parameters like max_tokens, top_p, top_k, seed, and similar values.

These parameters directly affect generation quality and cost, and are the exact fields that the OpenAI (max_tokens, top_p, seed) and Anthropic (max_tokens, temperature, top_p, top_k) integrations capture.

3. schema not captured (accessible via attr_reader)

RubyLLM::Chat exposes attr_reader :schema, which holds the structured output format schema when users call chat.with_format(MyModel). Capturing this identifies which structured output format was requested — comparable to response_format captured by the OpenAI integration.

4. tools omitted from streaming spans

The extract_metadata method explicitly skips tools for streaming calls:

# lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb (lines ~162–164)
if !stream && respond_to?(:tools) && tools&.any?
  metadata["tools"] = extract_tools_metadata
end

tools is an attr_reader on RubyLLM::Chat and is available during streaming calls. When users stream a tool-using chat, the span metadata contains no information about which tools were available — making it impossible to correlate tool execution spans with their governing chat configuration.

Comparison with other integrations

Field	OpenAI	Anthropic	ruby_llm
model	✓	✓	✓
temperature	✓	✓	✗ (in @temperature, no attr_reader)
max_tokens	✓	✓	✗ (in params)
top_p / top_k	✓	✓	✗ (in params)
seed	✓	–	✗ (in params)
tools	✓	✓	✓ non-streaming only
tool_choice	✓	✓	✗ (in tool_prefs[:choice])
parallel_tool_calls	✓	–	✗ (in tool_prefs[:calls])
response_format / schema	✓	–	✗ (in schema)

Braintrust docs status

not_found — The Braintrust ruby_llm integration docs at https://www.braintrust.dev/docs/integrations/sdk-integrations/ruby-llm describe basic chat tracing but do not document which metadata fields are captured. No mention of tool_choice, generation parameters, or structured output format in the Ruby LLM integration docs.

Upstream sources

• RubyLLM::Chat source with attr_readers: https://github.com/crmne/ruby_llm/blob/main/lib/ruby_llm/chat.rb
• ruby_llm v1.13.0 release (tool_prefs/tool_choice added): https://github.com/crmne/ruby_llm/releases/tag/v1.13.0
• with_tools(choice:, calls:) documentation: https://rubyllm.com/guides/tools

Local repo files inspected

• lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb — extract_metadata (lines ~151–167) captures only provider, stream, model, and tools (non-streaming); no tool_prefs, params, or schema
• lib/braintrust/contrib/openai/instrumentation/chat.rb — METADATA_FIELDS captures 20 fields including model, temperature, top_p, max_tokens, response_format, seed, tools, tool_choice, parallel_tool_calls
• lib/braintrust/contrib/anthropic/instrumentation/messages.rb — METADATA_FIELDS captures model, max_tokens, temperature, top_p, top_k, stop_sequences, tools, tool_choice, thinking, service_tier