A r.uby.dev project.
Welcome to the canonical llm.rb repository.
llm.rb is an advanced runtime for building capable AI applications on CRuby. By default it has zero runtime dependencies although certain functionality – such as ActiveRecord support – require optional dependencies that are opt-in.
Features
The runtime supports OpenAI, OpenAI-compatible endpoints, Anthropic, Google Gemini, Mistral, DeepSeek, DeepInfra, xAI, Z.ai, AWS Bedrock, Ollama, and llama.cpp. It has first-class support for streaming, tool calls, MCP and A2A, embeddings, vector stores, OCR, context compaction, and the RAG pattern.
There are multiple HTTP backends to choose from, tools can be run concurrently or in parallel via threads, async tasks, fibers, ractors, and fork, and it is also possible to make a tool call while the model is still streaming.
The runtime builds on top of three core concepts: providers, contexts, and agents, so once you learn the fundamentals, everything else falls into place naturally. And once you learn llm.rb, you will also be able to use mruby-llm and wasm-llm because the API is pretty much identical.
For detailed explanations, configuration, and advanced patterns, see the deepdive.md.
Install
gem install llm.rb
Quick start
LLM::Agent
The LLM::Agent class is the default high-level interface,
and it is recommended for most use-cases. It manages tool execution
automatically, guards against infinite loops, manages conversation
state, and much more.
require "llm"
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, stream: $stdout)
agent.talk "Hello world"
LLM::Context
The LLM::Context class is at the heart of the runtime
and it is what LLM::Agent uses under the hood.
It requires that the tool call loop be managed manually -
sometimes that can be useful, but usually for advanced use-cases.
If you're new to llm.rb, try LLM::Agent first.
require "llm"
llm = LLM.deepseek(key: ENV["KEY"])
ctx = LLM::Context.new(llm, stream: $stdout)
ctx.talk "Hello world"
LLM::Tool
Subclasses of LLM::Tool are plain Ruby classes with
an optional set of typed parameters. The model can choose to
call them on your behalf, and they're one of the most powerful features
for extending the feature set or abilities of a model.
class ReadFile < LLM::Tool
name "read-file"
description "Read a file"
parameter :path, String, "The filename or path"
required %i[path]
def call(path:)
{contents: File.read(path)}
end
end
LLM::Stream
Streams can be simple IO objects or subclasses of
LLM::Stream with structured callbacks for content,
reasoning, tool calls, tool returns, and compaction.
class MyStream < LLM::Stream
def on_content(content)
print content
end
def on_reasoning_content(content)
warn content
end
end
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, stream: MyStream.new)
agent.talk "Explain Ruby fibers."
LLM::Schema
LLM::Schema subclasses produce typed, structured
output from any model call. Pass a schema to LLM::Context#talk,
LLM::Agent#talk, or LLM::Provider#complete to receive validated
JSON instead of free text. Schemas work alongside tools and streams.
LLM::Schema can define objects, arrays, enums, nested schemas,
and more. It is also used internally by LLM::Tool for parameter
definitions, so you already benefit from it when you declare tool
parameters.
class Weather < LLM::Schema
property :city, String, "The city name"
property :temperature, Float, "Current temperature"
property :conditions, String, "Weather conditions"
required %i[city temperature conditions]
end
llm = LLM.openai(key: ENV["KEY"])
agent = LLM::Agent.new(llm, schema: Weather)
res = agent.talk "Weather in Paris?"
res.content! # => {city: "Paris", temperature: 15.0, conditions: "Cloudy"}
LLM::REPL
The LLM::Agent#repl
method drops you into a curses-based TUI for talking to an
agent interactively. The path: option saves and restores
runtime state across sessions. The tools: option attaches
extra tools for the duration of the session. It is like
binding.pry but for agents. For the full reference see the
REPL section in the
deepdive.
require "llm"
require "llm/tools"
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, name: "my-agent")
agent.repl(path: "agent.json", tools: LLM::Tool.subclasses)
LLM::MCP
The Model Context Protocol (MCP) has first-class support
in llm.rb. The stdio and http transports work out of the
box. MCP tools are translated into subclasses of
LLM::Tool that can be used with LLM::Context
or LLM::Agent.
require "llm"
llm = LLM.deepseek(key: ENV["KEY"])
mcp = LLM::MCP.stdio(argv: ["ruby", "server.rb"])
agent = LLM::Agent.new(llm, stream: $stdout, tools: mcp.tools)
agent.talk "Run the tool"
LLM::A2A
The Agent 2 Agent (A2A) protocol has first-class support
in llm.rb. The http and jsonrpc transports work out of the
box. A2A skills are translated into subclasses of
LLM::Tool that can be used with LLM::Context
or LLM::Agent.
require "llm"
llm = LLM.deepseek(key: ENV["KEY"])
a2a = LLM::A2A.rest(url: "https://remote-agent.example.com")
agent = LLM::Agent.new(llm, stream: $stdout, tools: a2a.skills)
agent.talk "Run the skill"
RAG
Most providers offer an embedding model that can be used for semantic search, or similarity search. An embedding model can generate embeddings that can then be stored in a database that is optimized for storing and querying vectors, such as SQLite's sqlite-vec or PostgreSQL's pg-vector.
llm.rb also includes support for OpenAI's vector store API. It provides a vector database as a HTTP service but we won't cover that here.
require "llm"
llm = LLM.openai(key: ENV["KEY"])
body = "llm.rb is Ruby's capable AI runtime."
embedding = llm.embed([body]).embeddings.first
Document.create!(
title: "llm.rb",
body:,
embedding:,
)
Concurrency
The runtime supports six different concurrency strategies that have different attributes. The choice between all of them often depends on the requirements of your application.
IO-bound tools are a good fit for the :async, :thread,
and :fiber strategies while true parallelism can be achieved
with the :fork and :ractor strategies. The
:sequential strategy runs tools one at a time and is the default.
The :fork strategy also provides a separate process that offers
isolation from its parent.
You can learn more about the llm.rb concurrency model in the deepdive.md.
require "llm"
llm = LLM.deepseek(key: ENV["KEY"])
tools = [FetchNews, FetchStocks, FetchFeeds]
agent = LLM::Agent.new(llm, tools:, concurrency: :fork)
agent.talk "Run the tools in parallel"
ORM
Because both LLM::Context, and LLM::Agent
can be serialized to JSON and stored in a simple string, both ActiveRecord
and Sequel support can be implemented within a single column on a single row.
The runtime includes first-class support for both ActiveRecord and Sequel, and
for both Rack-based applications and Rails-based applications. On databases
where it is supported, such as PostgreSQL, the column can be optimized by using
the jsonb type.
require "active_record"
require "llm"
require "llm/active_record"
class Agent < ApplicationRecord
acts_as_agent
set name: "my-agent",
instructions: "solve the user's query",
model: "deepseek-v4-pro",
tools: [Research, FinalizeResearch, ActOnResearch]
private
# By convention, this method defines the provider for a model.
# If necessary, it can be renamed with: provider: :your_method.
def set_provider
LLM.deepseek(key: ENV["KEY"])
end
# By convention, this method returns the context options given
# to LLM::Context or LLM::Agent.
def set_context
{}
end
end
agent = Agent.create!
agent.talk "perform research"
FAQ
Cloud
The following cloud-based providers are available to choose from. In no particular order:
🇺🇸 OpenAI 🇺🇸 DeepInfra 🇺🇸 xAI 🇺🇸 Google (Gemini) 🇺🇸 AWS bedrock 🇺🇸 Anthropic 🇨🇳 DeepSeek 🇨🇳 zAI 🇪🇺 Mistral
Weights
The following providers provide access to open-weight models. In no particular order:
🇺🇸 DeepInfra 🇺🇸 AWS bedrock 🇨🇳 DeepSeek 🇨🇳 zAI 🇪🇺 Mistral
Local
The following providers can be run locally on your own hardware. In no particular order:
- Ollama
- Llamacpp
Resources
If you like what you read so far, check out the deepdive.md to learn more. Unfortunately it wasn't possible to cover every feature without the README becoming a small book. The r.uby.dev homepage also includes more learning material and resources.
License
This software is released under the terms of the MIT license. See LICENSE for details.
No comments yet
Be the first to share your take.