Programmatic Tools

What this page covers

• Why we offer two capabilities: programmatic Codemode and reusable Skills
• How the toggles behave in the UI/backend
• When to use each mode, and when to combine them
• Configuration levers (allow_direct_tool_calls, reranker hook) and their tradeoffs

Goals

• Codemode: let the agent write Python to compose tools directly, reducing LLM tool-call overhead and enabling control flow, parallelism, and error handling.
• Skills: package reusable workflows (instructions, resources, scripts) so an agent can load, read resources, and run scripts without writing new code each time.
• Coexistence: you can enable Skills, Codemode, or both. Codemode provides its own tool discovery/execution; Skills surface curated capabilities.

Toggles and behavior

• Enable Codemode: adds CodemodeToolset (list/search tools, execute Python code). When Codemode is on, selected MCP servers are used to build Codemode’s registry (tools are exposed via Codemode meta-tools rather than direct MCP tools).
• Enable Skills: adds AgentSkillsToolset (skills discovery, load, resource read, script run). Uses the configured skills/ directory.
• Both on: Skills are automatically wired into Codemode via wire_skills_into_codemode(). Skill operations become available as generated bindings (from generated.skills import ...) inside execute_code, alongside MCP tool bindings. The AgentSkillsToolset is still attached for direct tool access, but the primary interaction path is through Codemode's code execution.

Codemode toolset capabilities

• list_tool_names (fast names-only, supports server, keywords, limit)
• search_tools (returns tool definitions; can be reranked, includes deferred tools by default)
• get_tool_details (full schema, output shape, and examples for one tool)
• list_servers (connected MCP servers)
• execute_code (Python in sandbox; import bindings from generated.mcp.<server>)
• call_tool (single tool call) — optional, see below

Direct tool calls: allow_direct_tool_calls

• When false (default in agent-runtimes codemode setup):
  • call_tool is hidden; all execution goes through execute_code.
  • Instructions emphasize code-first composition, mirroring the “write code to use tools” pattern.
• When true (opt-in):
  • call_tool is exposed as a convenience for single calls.
  • Useful for simple queries where code would be overkill.
• Choose false for stricter discipline and clearer audit of composed workflows; choose true for convenience at the cost of less enforced structure.

Search quality: reranker hook

• search_tools accepts an optional async reranker: tool_reranker(tools, query, server) -> list[ToolDefinition].
• Use it to reorder results (e.g., LLM-based relevance, business priority). Failures fall back to registry order.
• This keeps discovery flexible without tying Codemode to any specific model provider.

Execution model

• Codemode runs Python in an isolated sandbox (code-sandboxes). Agents import tool bindings from generated.mcp.<server_name>.
• Use async/await, loops, conditionals, and asyncio.gather for parallelism.
• Errors are caught and surfaced in the execute_code result payload, alongside stdout/stderr.

Skills model

• Skills live in a skills/ directory with SKILL.md, resources, and scripts.
• Tooling provided by AgentSkillsToolset: list_skills, load_skill, read_skill_resource, run_skill_script.
• Skills can be authored as files or programmatic callables; they complement Codemode when you want curated, reusable behaviors.

UI & backend flow

• UI checkboxes toggle Skills/Codemode. If Codemode is enabled, MCP server selection is still available and is used to scope Codemode discovery.
• Backend route /api/v1/agents builds toolsets accordingly:
  • Skills enabled → add AgentSkillsToolset.
  • Codemode enabled → add CodemodeToolset with allow_direct_tool_calls=False by default.
  • Both enabled → add both toolsets, then wire skills into Codemode via wire_skills_into_codemode().

Skills-in-Codemode wiring

When both Skills and Codemode are enabled, agent-runtimes automatically wires skills into Codemode so that skill operations are available as generated bindings inside execute_code. This is handled by wire_skills_into_codemode() in agent_factory.py.

What the wiring does

1. Generates skill bindings — Calls generate_skill_bindings() to create generated/mcp/skills/ with wrapper functions for list_skills, load_skill, read_skill_resource, and run_skill.

2. Sets a skill tool caller — Registers a callback on the executor that routes skills__* tool calls to the real AgentSkillsToolset.

3. Stores skills metadata — Calls set_skills_metadata() on the executor so that remote sandboxes (Datalayer Runtime, Jupyter kernel) can regenerate skill bindings inline.

4. Registers a skills proxy caller — For remote sandboxes, registers a proxy caller in mcp_proxy.py so that HTTP-routed skills__* calls reach the AgentSkillsToolset.

Post-init callback pattern

Since CodemodeToolset lazily initializes its executor, the wiring is deferred using a post-init callback:

# In rebuild_codemode (app.py / routes/agents.py)
if skills_enabled:
    codemode_toolset.add_post_init_callback(
        lambda ts: wire_skills_into_codemode(ts, skills_toolset)
    )

The callback fires once when the executor is first created (e.g., on the first execute_code call), ensuring the executor exists before binding generation and caller registration happen.

Using skills in execute_code

Once wired, agents can call skill operations from within execute_code:

execute_code(code='''
from generated.skills import list_skills, load_skill, run_skill
from generated.mcp.filesystem import read_file

# Discover available skills
skills = await list_skills({})
print(f"Skills: {[s['name'] for s in skills]}")

# Load a skill's instructions
instructions = await load_skill({"skill_name": "pdf-extractor"})
print(instructions)

# Run a skill script
result = await run_skill({
    "skill_name": "pdf-extractor",
    "script_name": "extract.py",
    "args": {"path": "/data/report.pdf"}
})
print(result)
''')

Remote sandbox support

When the code sandbox is remote (e.g., a Datalayer Runtime or Jupyter kernel), the skill bindings are generated inline inside the sandbox by _generate_tools_in_sandbox(). Tool calls from the remote sandbox make HTTP requests to agent-runtimes, where the skills proxy in mcp_proxy.py intercepts server_name == "skills" and routes the call to the registered AgentSkillsToolset.

Skills-first, Codemode glue (example)

Use Skills for curated, reusable steps and Codemode for orchestration:

# 1) Load skill instructions
load_skill("pdf-extractor")

# 2) Compose multi-step flow in code
execute_code(code='''
from generated.mcp.filesystem import read_file
from generated.skills import run_skill

doc = await read_file({"path": "/data/report.pdf"})
result = await run_skill({
    "skill_name": "pdf-extractor",
    "script_name": "extract.py",
    "args": {"path": "/data/report.pdf"}
})
print(result)
''')

Choosing a pattern

• Use Skills when you want repeatable, reviewed workflows and clearer governance.
• Use Codemode when you need ad-hoc composition, control flow, and efficiency.
• Use both when you want curated capabilities (Skills) plus the ability to glue them together with code (Codemode). When both are enabled, skills are automatically wired into Codemode as generated bindings (from generated.skills import ...), giving agents a unified import pattern for all tools.

Configuration summary

• allow_direct_tool_calls: hide or expose call_tool (defaults to false in agent-runtimes wiring).
• tool_reranker: optional async hook to reorder search_tools results.
• keywords/limit on list_tool_names: faster, filtered discovery.
• include_deferred on search_tools/list_tool_names: control discovery of tools marked defer_loading.
• max_tool_calls: optional per-run safety cap on tool invocations inside execute_code.

allow_direct_tool_calls

• What it does: Controls whether the call_tool shortcut is exposed. When off, all tool usage flows through execute_code, which keeps execution auditable and code-first.
• Default: false in agent-runtimes wiring (CodemodeToolset is instantiated with direct calls disabled).
• When to set true: Simple, single-step calls where the overhead of writing code is unnecessary, or for rapid prototyping.
• When to keep false: Shared/production agents, where you want consistent logging, fewer accidental direct calls, and clearer control over how tools are composed.

tool_reranker

• What it does: Optional async hook tool_reranker(tools, query, server) -> list[ToolDefinition] that reorders search_tools results.
• Why use it: Apply model-based relevance, business rules, or safety filters to discovery without changing the registry itself.
• Behavior on error: If the hook raises or fails, Codemode falls back to registry order; search still returns results.
• Usage tip: Log applied ordering when the hook is enabled to keep discovery transparent.

Safety and clarity tips

• Prefer allow_direct_tool_calls=False in shared/production agents to keep a single, auditable execution path via execute_code.
• When enabling reranking, log or trace the applied order for transparency.
• Keep Skills concise and documented; use Codemode for bespoke multi-step tasks.

Recommended defaults (cookbook)

Production

• enable_skills: true
• enable_codemode: true
• allow_direct_tool_calls: false
• enable_tool_reranker: true (if you have a safe reranker configured)
• max_tool_calls: set a conservative limit (e.g., 50–200) to prevent runaway loops

Prototyping

• enable_skills: optional
• enable_codemode: true
• allow_direct_tool_calls: true
• enable_tool_reranker: optional
• max_tool_calls: unset or higher limit for exploration