Connect LangChain to OpenLLM Buddy

Connect LangChain (Python and JS) to OpenLLM Buddy with ChatOpenAI and a custom base URL. Includes tool calling and streaming.

SDK & framework · Integration guide

What is LangChain?

LangChain is a framework for building LLM applications: chains, retrieval, tool-calling agents, and LangGraph workflows. Visit LangChain.

Why connect OpenLLM Buddy?

ChatOpenAI accepts a custom base_url, so every LangChain and LangGraph feature works against your OpenLLM Buddy deployment.

Before you start, create an API key in the console and deploy a model. If anything fails, run the curl test in Step 5 first: no tool can fix a broken endpoint.

1

Install

Install the OpenAI integration package.

Terminal
bash
# Python
pip install langchain-openai

# JavaScript / TypeScript
npm install @langchain/openai
2

Python

Create a ChatOpenAI model that points at your endpoint.

app.py
python
import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gemma4:26b",
    base_url="https://openllmbuddy-proxy.botbuddytech.workers.dev/v1",
    api_key=os.environ["OPENLLMBUDDY_API_KEY"],
    temperature=0.7,
)

print(llm.invoke("Summarise RAG in one sentence.").content)
3

JavaScript / TypeScript

The JS package nests the URL under configuration.

app.mjs
javascript
import { ChatOpenAI } from "@langchain/openai";

const llm = new ChatOpenAI({
  model: "gemma4:26b",
  apiKey: process.env.OPENLLMBUDDY_API_KEY,
  configuration: { baseURL: "https://openllmbuddy-proxy.botbuddytech.workers.dev/v1" },
});

const res = await llm.invoke("Summarise RAG in one sentence.");
console.log(res.content);
4

Tools and agents

Bind tools the normal way. This needs a model with function-calling support.

tools.py
python
from langchain_core.tools import tool

@tool
def add(a: int, b: int) -> int:
    """Add two integers."""
    return a + b

agent_llm = llm.bind_tools([add])
print(agent_llm.invoke("What is 21 + 21?").tool_calls)

Pair it with LangGraph's prebuilt agents for multi-step workflows. If tool calls come back empty, the model may not support function calling.

5

Test the endpoint with curl

Confirm the key, URL, and model id work before debugging LangChain. A short JSON reply means the endpoint is healthy.

Terminal
bash
curl -sS https://openllmbuddy-proxy.botbuddytech.workers.dev/v1/chat/completions   -H "Authorization: Bearer YOUR_API_KEY"   -H "Content-Type: application/json"   -d '{"model": "gemma4:26b", "messages": [{"role": "user", "content": "Reply with OK"}], "max_tokens": 16}'

Troubleshooting

  • Still calling OpenAI — OPENAI_API_BASE in your environment can override the code. Print llm.openai_api_base to confirm.
  • Structured output fails — use method="json_mode" or prompt-based parsing when the model has no native JSON schema support.
  • 401 or 403 — the API key is wrong, revoked, or missing the Bearer prefix. Create a fresh key in the console.
  • 404 on /chat/completions — the base URL must end in /v1, with no trailing /chat/completions and no extra path.
  • Model not found — the model id must match your deployment exactly. Run the curl test and copy the id you used.
  • Timeouts on the first request — a GPU that was idle may still be loading the model. Retry after a minute.
  • Connection refused from Docker or a remote host — use the public HTTPS base URL, never localhost.

Connection details

Use these values wherever LangChain asks for an OpenAI-compatible provider.

Reference
text
Base URL:  https://openllmbuddy-proxy.botbuddytech.workers.dev/v1
Chat URL:  https://openllmbuddy-proxy.botbuddytech.workers.dev/v1/chat/completions
API key:   your key from the console
Model id:  gemma4:26b