the doorbell speaketh
Today I went back to the intelligent doorbell shenanigans I had been messing about with a few months ago. Back when I started playing with it, I just really wanted to see if there was a way to pipe audio out of the doorbell speaker that didn’t come from one of the preprogrammed responses Reolink gives in their app.
The “problem” I was “addressing” was the fact that, even having a smart doorbell with a camera, sometimes I am just plain busy at work, or perhaps just generally faffing about so much that I don’t notice the notification of my phone - and end up missing a delivery. So I thought, “wouldn’t it be cool if I could have a doorbell AI agent!” Something that could:
- Recognise and greet known people at the door
- Hold a multi-turn conversation with strangers to find out why they’re there?
- Reach out to me with an interrupting notification for anything that needed me personally
The honest point that I stopped last time was just getting a python script that passed LLM generated audio out of the doorbell’s built in speaker. That’s it. I called that victory.
So today I thought let’s get it properly wired up. Now some people might say, giving an agent access to speak through your doorbell is perhaps not the safest thing to do, especially given how many smart devices are in your house - what if someone manages to persuade the agent to do something like unlock the door, or turn off the alarm? But screw it.
Just kidding - I was concerned about this too, and I don’t think my wife would have much appreciated an agent deciding when our doors would or would not be locked. So I decided to scope it very tightly, the way the first iteration has been designed, the agent is actually structurally & technically incapable of doing anything other than talking.
A few architectural points to cover. I have frigate running in a docker container monitoring the feed from my doorbell camera, running person/object detection and facial recognition locally. Frigate has a library of trained faces (just myself, my wife, a handful of family members). I also have the Frigate integration for Home Assistant which exposes entities like ‘last recognized face’ and things like ‘person’ or ‘dog’. There are some other entities in Home Assistant unrelated to Frigate that I’m also using in these doorbell shenanigans, like the front door lock status, the front door sensor status - but these are structural and are in no way exposed to the agent as something that can be manipulated, or it even knows about.
In my head it would work like this:
- monitor home assistant entities for person & last recognized face
- named greeting for recognized faces / generic greeting for anyone unrecognized
- listen for response using doorbell microphone
- transcribe response with whisper
- pass response to LLM (qwen3.6:27b - running locally)
- LLM generates response
- pipe LLM audio out of doorbell speaker
- loop for n turns
There are a couple of concerns here though - what if the person/face detection fails? Well in that case I hope they use the doorbell button, and the agent will always respond to that. What if it continually fires when we’re just leaving the flat? I can use the door lock state and open sensor to filter out when we’re leaving.
I’m sure there are some other cases that will come up to do with the conversation timing, silence detection, etc. But after much pain and suffering, there is a very simple local AI agent monitoring my door. Not a lot happens right now - he has no tools, can’t do anything but say “Hi, how can I help you?” and then just have a pleasant conversation. But now that the guts are there, we can start to build! There’s an automation set up that has interrupt priority on my phone, but right now it’s not connected to the agent. Next step is scoping some very deliberate tools for the agent, and I think the notifications might be the first one.
Either that or I’ll move on to something completely different and forget about this for another four months…