MLH Hack At Home 2020: Hearth

This is a submission for the MLH x DEV Writing Challenge

What I Built

Hearth is a voice-controlled smart home agent built for the moment we were all in: stuck at home, working from the same room we slept in, and slowly losing track of what day it was.

The theme for Hack at Home was "help folks stay happy at home." In May 2020, the problem in my apartment wasn't a lack of smart devices. It was that home had turned into office, gym, classroom, and bedroom at once, with nothing separating them. So Hearth does three things:

1. Voice control that knows your devices. "Turn off the desk lamp" works because Hearth syncs the real device names from Home Assistant into its language model, instead of relying on generic "light" commands.
2. Modes for a life that happens in one room. "Work mode" brightens the desk light, sets the room to 71°F, and turns on an "on a call" LED outside the door. "Done for the day" shuts the work lights off, switches to warm lighting, and reminds you to step outside.
3. Pattern suggestions from sensor history. A small Python job looks at a week of Home Assistant state history and suggests automations you keep doing by hand, like "You turn on the kitchen light around 7:40 AM every weekday. Want me to do that for you?"

How it works

• Wake word: Picovoice Porcupine runs on a Raspberry Pi 4 with a USB mic. It's fully offline, and nothing is sent to the cloud until the wake word fires.
• Speech-to-text: Google Cloud Speech-to-Text (streaming) transcribes the command after the wake word.
• Intent understanding: Dialogflow ES maps the transcript to intents like device.set, mode.activate, and history.ask. Device names come from a custom entity type that we sync from Home Assistant's entity registry through the Dialogflow API. When a light is renamed in HA, Hearth picks up the new name within a minute.
• Fulfillment: A Python Flask webhook on Google Cloud Run turns intents into Home Assistant REST API calls.
• Safety: Locks and the garage never actuate from voice alone. The webhook sends a Firebase Cloud Messaging push to the Android app, and nothing happens until someone taps Approve.
• Pattern mining: A nightly job pulls history from Home Assistant's recorder, buckets it into 15-minute windows with pandas, and finds repeated manual actions. It also runs scikit-learn's Isolation Forest to flag oddities, like the garage being open at 2 AM.
• Android app: Kotlin, XML layouts, ViewModel + LiveData, and coroutines for the network calls. It has three screens: devices, pending approvals, and suggested automations.
• Domain: The dashboard lives at [hearth-home].tech, registered through Domain.com during the weekend.

What I learned

• Generic intents break on real houses. Our first Dialogflow agent used the built-in "device" concepts and had no idea what "the Ikea lamp" was. Syncing Home Assistant's real friendly names into a custom entity type, with synonyms, was the single biggest accuracy improvement of the weekend.
• Plan for your dependencies disappearing. We originally wanted Snips for fully on-device voice, until we found out Sonos had shut down the Snips console after acquiring it. Porcupine for the wake word plus cloud STT was the realistic fallback.
• Safety has to live in the code. "Always confirm locks" as a design rule is only real if the webhook physically can't call the lock service without an approval token. That took 30 minutes to build and we never worried about it again.
• Latency is the whole experience. Wake word → cloud STT → Dialogflow → webhook → Home Assistant came in around 2.5 seconds at first. Switching to streaming recognition and keeping the Cloud Run instance warm got it to about 1.8 seconds.

Demo

Links: I have lost access to that github account and repositories and don't know where the stuff exists. Oops. Sorry!!

This is a text walkthrough of what I demoed. The setup: a Raspberry Pi 4 running Porcupine and Home Assistant 0.109 (a mix of real smart plugs and bulbs plus HA's demo integrations), the webhook on Cloud Run, and the Android app on a Xiaomi Redmi Note 3.

Disclaimer: This part has been generated with AI assistance to make it look proper for Markdown

Architecture

 🎤 USB mic ──► Porcupine (wake word, on the Pi)
                        │
                        ▼
          Google Cloud Speech-to-Text (streaming)
                        │
                        ▼
                  Dialogflow ES ◄──── custom entity type synced
                        │             from Home Assistant registry
                        ▼
         Flask webhook on Cloud Run ──────► Firebase Cloud Messaging
                        │                          │
                        ▼                          ▼
           Home Assistant REST API        📱 Android app (approvals,
           (lights, plugs, thermostat,       suggestions, devices)
            lock, garage)
                        │
                        ▼
           Nightly pattern job (pandas + scikit-learn)

Step 1: Work mode

🗣️ "Hey Hearth, start work mode."

[dialogflow] intent=mode.activate  mode=work  confidence=0.94
[webhook]    → light.turn_on  light.desk_lamp  brightness_pct=100
             → climate.set_temperature  climate.bedroom  71
             → switch.turn_on  switch.door_on_call_led
[webhook]    3/3 succeeded
[response]   "Work mode is on. Good luck out there."
[latency]    1.8s wake-to-response

The LED outside the door turns red. That's for the roommate who kept walking in during Zoom lectures.

Step 2: Device names that actually work

🗣️ "Turn off the Ikea lamp."

[dialogflow] intent=device.set  device=@ha_device:"Ikea lamp" → light.living_lamp_2
[webhook]    → light.turn_off  light.living_lamp_2
[response]   "Done."

"Ikea lamp" is a synonym we added in Home Assistant. It showed up in Dialogflow on the next sync, and no retraining was needed.

Step 3: The safety gate

🗣️ "Unlock the front door."

[dialogflow] intent=device.set  device=lock.front_door  action=unlock
[webhook]    lock.* requires approval → FCM push sent, request held
[response]   "I've sent a confirmation to your phone."

The phone shows: "Unlock Front Door? Voice request, 8:42 PM" with Approve / Deny. The lock only opens after Approve.

Step 4: Asking what happened

🗣️ "When was the garage last opened?"

[dialogflow] intent=history.ask  device=cover.garage_door
[webhook]    → GET /api/history/period  cover.garage_door  (last 24h)
[response]   "The garage opened at 6:12 PM and closed at 6:15 PM."

Step 5: Suggested automations

The Suggestions screen in the Android app shows what the nightly job found in a week of history:

Tapping a suggestion shows the Home Assistant automation it would create:

alias: "Hearth: Weekday kitchen light"
trigger:
  - platform: time
    at: "07:40:00"
condition:
  - condition: time
    weekday: [mon, tue, wed, thu, fri]
action:
  - service: light.turn_on
    entity_id: light.kitchen

Accept sends it to Home Assistant, and Reject teaches the job to stop suggesting that pattern.

What didn't make it

Voice recognition struggled with the fan running in the room, so some of our testing used typed input in the Dialogflow console. The pattern miner also needs a real week of history. For the demo, we replayed a week of logged data from my apartment.

Partner Technologies

Google Cloud

Most of Hearth runs on Google Cloud:

• Speech-to-Text: We used streaming recognition with phrase_hints populated from our device names, which noticeably helped with odd names like "the Ikea lamp." The biggest gotcha was a silent stream timing out and ending recognition. The fix was to only open a stream after Porcupine fires.
• Dialogflow ES: We built intents for devices, modes, and history questions. Custom entity types synced from Home Assistant through the Dialogflow EntityTypes API were the key trick. The console's training-phrase interface made it easy to iterate on phrasing at 3 AM.
• Cloud Run: Hosts the Flask fulfillment webhook. Deploying a container in a couple of minutes suited a hackathon well. Cold starts added about a second at first, so we set up a cheap scheduled ping to keep it warm during the demo.
• Firebase Cloud Messaging: Carries the approval pushes to the Android app. We used data messages rather than notification messages, so the app controls exactly how the approval card appears.

Domain.com

We used default domains for the project dashboard provided by the Cloud Run service.

Hackathon Experience

I attended Hack at Home, the first MLH Summer League hackathon, held online the weekend of May 2, 2020. Like everyone else that spring, I hacked from home.

I was following up the events page and checking the updates constantly while making sure project meets the guidelines.

I teamed up with my College Friends to build this project, who had experience in API architecture and any Synthesis, Data tasks and Hardware, while I took care of System Design and Android side.

We enjoyed building the project and learnt a lot going through the documentations and performance while doing the same. This provided a way to experience the moment during the pandemic outside and work on upskilling our skills, during the time and be distracted for a while.

What I'll remember most: It was my first MLH hackathon experience and although virtual, I had a wonderful experience and although my memory may have faded on the specifics, I still remember it as a valuable experience.

Thanks to MLH and everyone was part of organizing it.