HomeSense: turning speech into physical actions, no app

Your "smart home" is a remote control with extra steps

Most smart-home setups don't remove work — they relocate it. Instead of walking to a switch, you pull out a phone, open an app, wait for it to connect, and tap a toggle. HomeSense started from the frustration of that trade and asked a blunter question: what if you could just say "turn on light two" and have it happen, with no app and no wake-word hardware in the room?

It's a voice-first home automation assistant that turns natural speech into physical actions. I showcased it at GDG DevJams 2024 (Gravitas, VIT Vellore). This is how it works under the hood.

One rule that shaped the whole design

Here's the principle the entire architecture is built around:

Anything with a physical consequence resolves locally.

Your lights should not depend on a cloud round-trip. If the Wi-Fi is flaky or a remote API is having a bad day, a light switch that stops working is unacceptable in a way that a slow music search isn't. So HomeSense only ever puts cloud services in the path for things that are inherently remote — a music catalog, an open-ended question — and keeps everything with a physical effect on the local machine and the wire to the hardware.

How a command actually flows

The system runs a continuous listening loop. Take "turn on light two":

Capture — recognizer.listen() grabs audio with a 3-second timeout and a 5-second phrase cap.
Transcribe — Google Web Speech (en-IN) turns it into text.
Tokenize — NLTK's word_tokenize, then a stop-word filter, produces ['turn', 'light', 'two'].
Classify — light + on + two present → intent light_2 : ON.
Encode — an opcode lookup maps that to the byte 3.
Transmit — serial.write(b"3") at 9600 baud.
Actuate — the Arduino firmware reads the byte and does digitalWrite(pin2, HIGH).

The response comes back as synthesized speech through pyttsx3, which runs offline.

The one-line detail the whole thing depends on

Step 3 hides the best bug story in the project. Standard English stop-word lists — the ones every NLP tutorial tells you to strip — throw away on, off, and all. Those are noise words in most text. But in a home-automation command, they are the three most load-bearing words you can say. "Turn on" and "turn off" collapse to the same tokens the moment you apply a default stop-word filter, and suddenly every light command is ambiguous.

The fix is one line: remove on, off, and all from the stop-word set before filtering. It's trivial to write and impossible to guess from the outside, and the entire device path silently depends on it. This is the kind of detail that never shows up in an architecture diagram but is the difference between a demo that works and one that doesn't.

Routing: one voice, four backends

The intent classifier is a router. Once it knows what you meant, it dispatches to one of four handlers:

Device intents → a serial bridge (pyserial) sends a single-byte opcode to the Arduino. The host-to-firmware protocol is a flat opcode map — trivial to debug with a serial monitor, trivial to extend with a new device.
Music intents → the Spotify Web API via spotipy, using OAuth 2.0.
Task intents → MySQL, with a real date-indexed schema. Tasks persist; they don't die with the process the way an in-memory list would.
Anything else → Google Gemini. If it isn't a known intent, instead of failing, the assistant answers.

That last fallback matters for how the thing feels. A rules-based assistant that says "I didn't understand that" on every unrecognized phrase feels broken. One that quietly hands the unknown case to an LLM feels like it's actually listening.

The wake loop, and keeping secrets out of the tree

The assistant idles on a low-cost listener and only spins up on wake up, then goes back to sleep on sleep — so it isn't hammering the microphone at full tilt the whole time it's running. And every credential loads from a git-ignored creds.py; the repository ships a template, never a real key.

What building it taught me

Draw the trust boundary first. Deciding "physical actions are local, remote things can be remote" up front made every later routing decision obvious.
A dumb protocol is a feature. A single-byte opcode over serial is boring, and that's exactly why it's debuggable at 2 a.m. with nothing but a serial monitor.
The defaults will betray you. Stripping on and off as stop-words is the "correct" NLP move and the wrong product move. Know your domain better than the library does.

The full command reference, serial protocol, wiring, and setup are in the repository (MIT licensed).