Building an AI Assistant: flutter nodejs ai browser automation

This article was originally published on BuildZn.

Figured out how to tame web automation with an LLM. Everyone talks about AI agents, but getting them to reliably interact with dynamic web pages through a full-stack setup is a whole different beast. Spent weeks wrestling with flaky selectors and race conditions. Here's the blueprint that finally clicked for my personal AI assistant build using Flutter, Node.js, and browser automation. This setup slashed my daily busywork by a solid 60%.

Why Full-Stack AI Browser Automation?

My initial goal was simple: stop wasting time on repetitive online tasks. Think filling out expense reports, aggregating data from specific sites, or managing content on platforms without proper APIs. I needed a personal AI assistant build that could understand high-level commands, translate them into browser actions, and then report back. This isn't just about scripting; it's about an LLM making decisions based on current page state and a broader goal.

I looked at a few options. Pure Python? Nah, I'm a Flutter guy, wanted a native UI. JavaScript-only? Possible, but I prefer Node.js for backend heavy lifting and orchestration. So, the stack solidified:

• Flutter: For the cross-platform UI. Desktop support for Windows/macOS was key for a desktop assistant. This gives us flutter desktop automation capabilities on the client side.
• Node.js: The brain. This is where the AI agent logic lives, handles API calls to LLMs, and orchestrates Playwright. Essentially, our nodejs playwright agent server.
• Playwright: The hands. Robust, fast, and handles modern web elements way better than Puppeteer for my needs.

The core challenge? Bridging the LLM's high-level reasoning with the nitty-gritty of browser interactions. Getting an AI to decide "click this specific button" or "fill this form field" when the page layout changes, or elements appear dynamically, that's where the real work is.

The Architecture: Orchestrating LLM Decisions with Browser Actions

Here's the setup, simplified:

1. Flutter UI: User sends a command (e.g., "Summarize unread emails from Project X in Gmail").
2. Node.js Backend (API): Receives the command.
3. Initial LLM Call (Planner): The backend sends the command to an LLM (e.g., Claude 3.5 Sonnet, or OpenAI's GPT-4o). This "Planner" LLM identifies the initial high-level steps. For Gmail, it might be "1. Navigate to Gmail. 2. Log in. 3. Find unread emails. 4. Filter for Project X. 5. Extract summaries."
4. Action Executor Loop:
   • Node.js initializes Playwright.
   • For each step from the Planner, Node.js tells Playwright to perform an action (e.g., await page.goto('https://gmail.com')).
   • Observation/Reflection (LLM Call - Actuator): After each action, Node.js grabs the current page content (or specific elements). This observation, along with the overall goal and previous steps, is sent back to the LLM. The "Actuator" LLM's job is to decide the next precise browser action (e.g., click on [aria-label="Email address"], fill with myemail@gmail.com, press 'Enter').
   • This loop continues until the overall goal is met or an error occurs.
5. Result Reporting: Once the task is done, the extracted data or status is sent back to the Flutter UI.

Key Components & Their Roles:

• Flutter (Client):

  • Sends user intents via HTTP requests to Node.js.
  • Displays real-time status updates and final results.
  • Provides a simple UI for configuration and task management. It's truly a cross platform ai assistant client.

• Node.js (Backend/Agent Orchestrator):

  • Express API: Handles requests from Flutter.
  • LLM Integration: Uses @anthropic-ai/sdk or openai libraries. I used Claude 3.5 Sonnet for its cost-effectiveness and context window.
  • Playwright: @playwright/test for browser control.
  • Task Management: Simple state machine to track ongoing browser sessions and agent steps.

Building the Brain: Node.js, LLMs, and Playwright

Let's get into the code.

1. Flutter UI (Simplified)

On the Flutter side, it's pretty standard HTTP stuff.

// lib/services/ai_service.dart
import 'dart:convert';
import 'package:http/http.dart' as http;

class AIService {
  final String baseUrl = 'http://localhost:3000/api/agent'; // Your Node.js backend

  Future<String> runBrowserTask(String taskDescription) async {
    try {
      final response = await http.post(
        Uri.parse('$baseUrl/start'),
        headers: {'Content-Type': 'application/json'},
        body: jsonEncode({'task': taskDescription}),
      );

      if (response.statusCode == 200) {
        final data = jsonDecode(response.body);
        return data['result'] ?? 'Task completed.';
      } else {
        return 'Error: ${response.statusCode} - ${response.body}';
      }
    } catch (e) {
      return 'Network error: $e';
    }
  }
}

// In your Flutter widget:
// import 'package:your_app/services/ai_service.dart';
// final aiService = AIService();
// String result = await aiService.runBrowserTask("Go to Google, search 'FarahGPT', click first link, tell me the title.");
// print(result);

This just kicks off the task. The real magic happens on Node.js.

2. Node.js Backend & Agent Loop

First, set up a basic Express server.

// server.js
const express = require('express');
const { chromium } = require('playwright');
const { Anthropic } = require('@anthropic-ai/sdk'); // Or OpenAI

const app = express();
const port = 3000;

app.use(express.json());

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY, // Make sure you set this!
});

// A simple in-memory store for ongoing sessions (for demo)
const activeSessions = {};

app.post('/api/agent/start', async (req, res) => {
  const { task } = req.body;
  if (!task) {
    return res.status(400).json({ error: 'Task description is required.' });
  }

  const sessionId = Date.now().toString(); // Unique session ID
  activeSessions[sessionId] = { task, browser: null, page: null, history: [] };

  try {
    const result = await runAgentTask(sessionId, task);
    delete activeSessions[sessionId]; // Clean up
    res.json({ sessionId, result });
  } catch (error) {
    console.error(`Agent task failed for session ${sessionId}:`, error);
    delete activeSessions[sessionId];
    res.status(500).json({ error: error.message, sessionId });
  }
});

app.listen(port, () => {
  console.log(`Node.js AI agent backend listening on http://localhost:${port}`);
});

// ... rest of the agent logic below

Now, the runAgentTask function, which orchestrates the LLM and Playwright:

// agent.js (or integrated into server.js)
const { chromium } = require('playwright');
const { Anthropic } = require('@anthropic-ai/sdk');
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }); // Re-init for clarity

async function runAgentTask(sessionId, initialTask) {
  const session = activeSessions[sessionId];
  if (!session) throw new Error('Session not found.');

  const browser = await chromium.launch({ headless: true }); // Change to false for debugging
  const page = await browser.newPage();
  session.browser = browser;
  session.page = page;

  let currentObservation = `Goal: ${initialTask}. Current URL: ${page.url()}.`;
  let history = [];
  let steps = 0;
  const MAX_STEPS = 10; // Prevent infinite loops

  // Initial planning phase (optional, can be merged into the main loop)
  const initialPlan = await anthropic.messages.create({
    model: "claude-3-5-sonnet-20240620",
    max_tokens: 500,
    messages: [
      {"role": "user", "content": `You are an expert browser automation agent. Your goal is: "${initialTask}". Based on this, what are the high-level steps to achieve this? List them clearly, e.g., "1. Navigate to X. 2. Login. 3. Do Y.". Then, what is the very first specific browser action I should take? Be precise.`}
    ]
  });
  console.log('Initial Plan:', initialPlan.content[0].text);
  history.push({ role: 'assistant', content: initialPlan.content[0].text });


  while (steps < MAX_STEPS) {
    steps++;

    // Add current page state to observation
    let pageContent = '';
    try {
      // Get the outer HTML of the body or a relevant part
      pageContent = await page.evaluate(() => document.body.outerHTML);
    } catch (e) {
      console.warn('Could not get page content:', e.message);
      pageContent = `Error getting page content: ${e.message}`;
    }

    currentObservation = `Current URL: ${page.url()}\nPage content snapshot (truncated): ${pageContent.substring(0, 1000)}\n\nGoal: ${initialTask}\n\nWhat is the NEXT precise browser action to take? Respond ONLY with a JSON object. Format: {"action": "goto"|"click"|"fill"|"type"|"evaluate"|"extract"|"finish", "target": "selector_or_url", "value": "text_to_fill_or_js_code", "reason": "why this action"}. If the goal is achieved, use "finish" action with the extracted result in "value".`;

    const messages = [
      ...history, // Previous interactions for context
      {"role": "user", "content": currentObservation}
    ];

    console.log(`--- Step ${steps} ---`);
    console.log('Sending to LLM, current URL:', page.url());

    const llmResponse = await anthropic.messages.create({
      model: "claude-3-5-sonnet-20240620",
      max_tokens: 500,
      messages: messages
    });

    const responseText = llmResponse.content[0].text;
    console.log('LLM Raw Response:', responseText);

    history.push({"role": "assistant", "content": responseText});

    let action;
    try {
      action = JSON.parse(responseText.replace(/```
{% endraw %}
json\n|\n
{% raw %}
```/g, '')); // Clean markdown code blocks
    } catch (e) {
      console.error('Failed to parse LLM action JSON:', e);
      // Fallback or request LLM to retry
      history.push({"role": "user", "content": "Error: Your last response was not valid JSON. Please provide a valid JSON action."});
      continue;
    }

    console('Parsed Action:', action);

    if (action.action === 'finish') {
      console.log('Agent finished task.');
      await browser.close();
      return action.value;
    }

    try {
      switch (action.action) {
        case 'goto':
          await page.goto(action.target, { waitUntil: 'domcontentloaded' });
          break;
        case 'click':
          await page.click(action.target, { timeout: 5000 }); // Add timeout for flaky elements
          break;
        case 'fill':
          await page.fill(action.target, action.value, { timeout: 5000 });
          // Hard Rule: Playwright 1.40.0+ on type="number" inputs can be weird.
          // If you fill a number input with JS handlers and it doesn't trigger change,
          // sometimes a manual dispatchEvent or a tab press helps.
          // Example: If filling `<input type="number" id="quantity">`, and the JS on blur
          // doesn't trigger, you might need:
          // await page.dispatchEvent(action.target, 'input');
          // I found this specifically with some custom Angular/React number inputs.
          // For simpler cases, page.fill is usually fine.
          break;
        case 'type': // More human-like typing
          await page.type(action.target, action.value, { delay: 100 });
          break;
        case 'evaluate': // Run custom JS on the page
          const evalResult = await page.evaluate(action.value);
          currentObservation = `Evaluated JS: ${evalResult}`;
          break;
        case 'extract': // Extract specific data
          const extractedData = await page.$eval(action.target, el => el.textContent);
          currentObservation = `Extracted data: ${extractedData}`;
          // If this is the final extraction before finishing, LLM should propose 'finish' next.
          break;
        default:
          throw new Error(`Unknown action: ${action.action}`);
      }
      await page.waitForTimeout(1000); // Give page a moment to settle
      // Update observation with new page state for next LLM call
      currentObservation = `Action "${action.action}" on "${action.target}" completed.`;
    } catch (e) {
      console.error(`Browser action failed for ${action.action}:`, e);
      history.push({"role": "user", "content": `Error: Failed to perform action "${action.action}" on "${action.target}". Reason: ${e.message}. Current URL: ${page.url()}. Please suggest an alternative action or try again, or use "finish" if impossible.`});
    }
  }

  await browser.close();
  return "Task reached max steps without completion.";
}

This runAgentTask function is the core of the flutter nodejs ai browser automation. It iteratively calls the LLM, parses the LLM's suggested action, executes it with Playwright, and then feeds the new page state back to the LLM for the next decision. It's a classic LLM agent "plan, act, observe, reflect" loop.

What I Got Wrong First

Honestly, getting the prompt right for the Actuator LLM was the biggest pain. I started with open-ended prompts like "What should I do next?". Big mistake. The LLM would generate paragraphs, or sometimes just chat back, rather than a structured action.

The Fix: Strict JSON output for actions.
By explicitly telling Claude "Respond ONLY with a JSON object. Format: {...}", it drastically improved reliability. If it messed up the JSON, my Node.js code would catch it, and I'd feed that parsing error back into the LLM's context as a new user message: "Error: Your last response was not valid JSON. Please provide a valid JSON action." This self-correction mechanism is crucial.

Another massive headache was handling dynamic web elements. Sometimes page.click('button#submit') would fail because the button wasn't ready, or an overlay covered it.

The Fix:

1. page.waitForSelector() before interaction, often with state: 'visible'.
2. Increased timeouts on Playwright actions (e.g., click({ timeout: 5000 })).
3. LLM reflection on failure: When Playwright throws an error, I catch it and feed the error message and the current page state (HTML snippet, URL) back to the LLM. The LLM can then decide to try a different selector, wait longer, or navigate elsewhere. This is what truly differentiates a scripted bot from an agent.

One specific issue I hit with Playwright 1.40.0: when filling certain <input type="number"> fields, especially if they had custom JavaScript validation or formatting on blur, page.fill() wouldn't always trigger the necessary change events. The form would look filled, but the internal application state wouldn't update.
The Fix: After page.fill(selector, value), I sometimes had to explicitly call await page.dispatchEvent(selector, 'input') or even await page.press(selector, 'Tab') to force the blur event. This isn't documented as a common necessity for page.fill, but it saved my sanity on specific SPA frameworks.

Quantifying the Time Saved

Before this flutter nodejs ai browser automation setup, I spent roughly 3-4 hours a week on repetitive tasks like:

• Collecting specific product data from competitor websites.
• Filling out complex internal forms for project updates.
• Aggregating analytics data from various dashboards that lack direct API access.

Now, I just type a command into my Flutter app: "Scrape product details for X from site Y and put it in a Notion table." or "Fill out my weekly report for Project Z with these bullet points."

The agent handles it. It navigates, logs in (using secure environment variables), extracts, fills, and even uploads if needed. This literally cut down 2.5-3 hours of that busywork a week. That's a 60-75% reduction on those tasks. The time saved allows me to focus on actual development and strategic thinking, not mind-numbing clicks.

FAQs

Q: Can I run Playwright headlessly on a remote server?

A: Absolutely. Playwright is designed for both headless and headful execution. For production deployments, you'll almost always run it headlessly on a Linux server (e.g., a Vercel serverless function or a dedicated EC2 instance). Just make sure your server environment has the necessary browser dependencies installed.

Q: How do you handle authentication (logins) securely?

A: Never hardcode credentials. For a personal assistant, store sensitive info (usernames, passwords) in environment variables or a secure vault (like Google Secret Manager, AWS Secrets Manager, or a simple .env file for dev, encrypted for prod). Your Node.js backend can retrieve these securely and pass them to Playwright for login forms.

Q: What's the biggest bottleneck for this type of agent?

A: The LLM's latency and its ability to consistently produce correct, actionable JSON. Even with Claude 3.5 Sonnet, which is fast, a complex task with many steps means multiple round trips to the LLM, adding up. If the LLM produces invalid JSON, it adds another round trip for correction. This is where good prompt engineering and robust error handling are critical.

This setup is a game-changer for personal productivity. Honestly, I don't get why more developers aren't building these specialized cross platform ai assistant tools for their own specific workflows. The combination of Flutter for a native feel, Node.js for backend agility, and Playwright for robust web interaction, all powered by a smart LLM, is incredibly powerful. Stop clicking, start automating.