I recently needed a fast way to prototype warehouse robot behaviors without writing a motion planner for every task. In this guide we will build a natural language robot controller that turns plain English into structured motion commands using an LLM, then executes them in a simple grid-world simulator. You can run the whole thing on your laptop and move it to ROS 2 or real hardware later.
What you'll need
Python 3.10 or newer, the OpenAI SDK (pip install openai), and an Oxlo.ai API key from https://portal.oxlo.ai. I use Oxlo.ai here because robotics agents tend to carry long system prompts and multi-step reasoning traces, and the flat per-request pricing keeps costs predictable even when the context grows. See https://oxlo.ai/pricing for details.
Step 1: Set up the Oxlo.ai client and world model
We start with a minimal 2D grid world and a robot state class. The Oxlo.ai client is a drop-in replacement for the OpenAI client, so we point the base URL at https://api.oxlo.ai/v1.
import json
from openai import OpenAI
# Oxlo.ai client: flat per-request pricing works well for long system prompts
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
# Simple warehouse grid: object name -> (x, y)
WORLD = {
"battery": (2, 4),
"gearbox": (5, 2),
"charger": (7, 1),
}
class Robot:
def __init__(self):
self.x = 0
self.y = 0
self.gripper_open = True
self.held = None
def status(self):
grip = "open" if self.gripper_open else f"holding {self.held}"
return f"Robot at ({self.x}, {self.y}), gripper {grip}"
Step 2: Write the system prompt
The system prompt is the robot's brain. It defines the world state, available actions, and constraints. I keep it strict so the model outputs only valid JSON arrays.
SYSTEM_PROMPT = """You are the motion planner for a warehouse pick-and-place robot operating on a discrete 2D grid.
World objects and their coordinates:
- battery at (2, 4)
- gearbox at (5, 2)
- charger at (7, 1)
The robot starts at (0, 0) with an open gripper.
Available actions (respond as a JSON array):
- {"action": "move_to", "target": [x, y]}
- {"action": "grip"}
- {"action": "release"}
Rules:
1. Only grip an object when the robot is at the object's exact coordinates.
2. Only release when the robot is at the target delivery coordinates.
3. Do not hallucinate objects or coordinates not listed above.
4. Output ONLY the JSON array. No markdown, no explanation.
Example instruction: "Pick up the battery and place it at the charger."
Example output:
[{"action": "move_to", "target": [2, 4]}, {"action": "grip"}, {"action": "move_to", "target": [7, 1]}, {"action": "release"}]
"""
Step 3: Parse natural language into motion plans
This function sends the operator's instruction to Oxlo.ai and parses the returned JSON plan. I use llama-3.3-70b because it follows structured instructions reliably and handles tool-like reasoning without extra overhead.
def plan_task(instruction: str):
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": instruction},
],
temperature=0.1,
)
content = response.choices[0].message.content.strip()
# Remove markdown fences if present
content = content.removeprefix("
```json").removeprefix("```
").removesuffix("
```
").strip()
return json.loads(content)
Step 4: Build the motion executor
The executor validates each step against the current robot state and updates it. In a real deployment you would stream these commands to a motor controller or a ROS 2 action server.
def execute_plan(plan, robot, world):
for step in plan:
action = step["action"]
if action == "move_to":
tx, ty = step["target"]
robot.x, robot.y = tx, ty
print(f" moved to ({tx}, {ty})")
elif action == "grip":
obj = None
for name, (ox, oy) in world.items():
if (ox, oy) == (robot.x, robot.y):
obj = name
break
if obj is None:
raise RuntimeError(f"Nothing to grip at ({robot.x}, {robot.y})")
robot.held = obj
robot.gripper_open = False
print(f" gripped {obj}")
elif action == "release":
if robot.held is None:
raise RuntimeError("Gripper is empty")
print(f" released {robot.held} at ({robot.x}, {robot.y})")
robot.held = None
robot.gripper_open = True
else:
raise RuntimeError(f"Unknown action: {action}")
print("Plan complete.")
Step 5: Wire the interactive loop
The main loop ties everything together. It reads an instruction, generates a plan through Oxlo.ai, executes it, and prints the robot's final state.
if __name__ == "__main__":
robot = Robot()
print("Warehouse bot ready. Type a command or 'quit'.\n")
while True:
try:
cmd = input("> ").strip()
if cmd.lower() in {"quit", "exit"}:
break
print("Planning...")
plan = plan_task(cmd)
print(f"Generated plan: {json.dumps(plan)}")
print("Executing...")
execute_plan(plan, robot, WORLD)
print(robot.status())
except Exception as e:
print(f"Error: {e}")
Run it
Save the script as robot_controller.py, export your key, and run it:
export OXLO_API_KEY="sk-..."
python robot_controller.py
Here is a sample session:
Warehouse bot ready. Type a command or 'quit'.
> Pick up the battery and deliver it to the charger
Planning...
Generated plan: [{"action": "move_to", "target": [2, 4]}, {"action": "grip"}, {"action": "move_to", "target": [7, 1]}, {"action": "release"}]
Executing...
moved to (2, 4)
gripped battery
moved to (7, 1)
released battery at (7, 1)
Plan complete.
Robot at (7, 1), gripper open
Next steps
Swap the grid-world executor for a ROS 2 action client to drive a real manipulator. You can also add vision by passing camera frames to Oxlo.ai's vision models such as kimi-k2.6 so the LLM can choose targets dynamically instead of reading hardcoded coordinates.
