|
AI Collab Score: 7 / 3
How I used Ubuntu, OpenCV, Ollama, Gemma 3 Vision, and Python to build a local vision pipeline with memory, event filtering, and stability logic.
Most AI vision demos stop at image description.
A webcam captures an image, a vision model analyzes it, and the system prints something like: “A person is sitting at a desk.” That is useful as a proof of concept, but it does not behave like an agent. A real vision agent needs more than perception. It needs state. It needs memory. It needs event logic. It needs to know what changed, whether the change matters, and whether the change is stable enough to trust. That became the goal of this lab. I built a local AI Vision Agent that starts with a live webcam feed and evolves into a small event-driven vision system. It runs locally on Ubuntu using Python, OpenCV, Ollama, and Gemma 3 Vision. By the current version, the agent can confirm events like:
The important part is not just that the model can describe what it sees. The important part is the system built around the model that turns raw vision output into confirmed events. Lab Environment
The environment for this build was intentionally local. I wanted the full pipeline running in my own lab instead of depending on a cloud-hosted vision API.
The stack included Ubuntu as the operating system, Python for the application logic, OpenCV for camera access and overlay rendering, Ollama as the local model runtime, Gemma 3 Vision for image understanding, GitHub for version control, and a local webcam as the video source. Ubuntu Python OpenCV Ollama Gemma 3 Vision GitHub Local webcam System Pipeline
At a high level, the system works as a local vision pipeline.
The webcam provides a live video stream. OpenCV reads frames continuously. At a configured interval, the application saves a frame snapshot and sends it to the local vision model through Ollama. Gemma 3 Vision returns a structured description of the scene. From there, the application compares the current scene against previous state, filters meaningful changes, confirms stable events, and renders the result back onto the live video feed. This architecture matters because video display and AI inference operate at different speeds. The camera can run continuously, but local model inference takes several seconds per analyzed frame. Separating live video from periodic AI analysis keeps the application usable. Webcam ↓ OpenCV frame capture ↓ Periodic frame snapshot ↓ Ollama API call ↓ Gemma 3 Vision response ↓ Scene memory ↓ Event filtering ↓ Stability confirmation ↓ Overlay + logs Project Structure
As the project matured, I split the logic into separate modules instead of keeping everything in one script. That made the application easier to reason about and easier to extend.
The camera module handles webcam access. The vision module handles calls to Ollama and Gemma 3 Vision. The memory layer compares previous and current scene state. The event engine filters raw changes into meaningful events. The stability engine confirms events across multiple analysis cycles. The app module coordinates the runtime loop and renders the overlay. vision-agent/ ├── app.py # main application loop and overlay ├── camera.py # webcam initialization and frame reads ├── vision.py # Ollama / Gemma 3 Vision inference call ├── memory.py # scene-to-scene comparison ├── events.py # meaningful event filtering ├── stability.py # event confirmation logic ├── config.py # model, camera, timing, and path settings ├── logger.py # structured event logging ├── frames/ # captured frame snapshots └── logs/ # event logs v0.1 — Live Camera and Local Vision Inference
The first version established the foundation: connect to the webcam, capture frames, and send periodic snapshots to the local vision model.
OpenCV handled the camera connection. The application continuously read video frames from the webcam, but it did not send every frame to the model. Instead, it saved a frame snapshot at a configured interval and submitted that image for inference. This gave me the first working pipeline: camera → frame → model → text response At this stage, the system could see and describe, but it had no memory. Every analyzed frame was treated as a standalone image. Camera Initialization
Python
camera = cv2.VideoCapture(CAMERA_INDEX) Read Frame
Python
ret, frame = camera.read() Save Snapshot
Python
cv2.imwrite(str(image_path), frame) v0.2 — Overlay and Runtime Metrics
Once the camera and model were working, the next step was observability.
The early output printed to the terminal, which was fine for debugging but not useful as a visual experience. So I added an OpenCV overlay directly on the video window. The overlay displayed the model name, live FPS, inference time, average inference time, frame count, scene summary, activity, and confidence score. This gave immediate feedback on both what the model saw and how the application was performing. This was important because local AI systems need performance visibility. It is not enough to know that the model responded. You need to know how long inference took, how frequently the system is analyzing frames, and whether the application is still responsive.
Python
overlay = frame.copy()
cv2.rectangle(
overlay,
(10, 10),
(625, 465),
(0, 0, 0),
-1,
)
frame = cv2.addWeighted(overlay, 0.70, frame, 0.30, 0)
v0.3 — Scene Memory
The next problem was memory.
The model could describe the current frame, but the application did not know how that frame differed from the previous one. Without memory, each analyzed frame is isolated. I added a memory layer that stores the previous scene and compares it with the current scene. The memory layer tracks a scene summary, activity, and normalized objects. The comparison uses set logic. Objects present in the current frame but not the previous frame are treated as additions. Objects present in the previous frame but not the current frame are treated as removals. This allowed the agent to generate raw changes such as: phone appeared mug disappeared activity changed But it also exposed a real AI engineering problem: model inconsistency. The same object might be called “cup,” “mug,” or “coffee mug.” Without normalization, the system would treat those as different objects. Scene State
Python
current = {
"summary": result.get("scene_summary", ""),
"activity": result.get("activity", ""),
"objects": self.normalize_objects(result.get("objects", [])),
}
Object Comparison
Python
previous_objects = set(self.previous.get("objects", []))
current_objects = set(current.get("objects", []))
added = current_objects - previous_objects
removed = previous_objects - current_objects
Normalization
Python
aliases = {
"coffee mug": "mug",
"cup": "mug",
"coffee cup": "mug",
"cell phone": "phone",
"mobile phone": "phone",
"smartphone": "phone",
}
v0.4 — Meaningful Event Engine
Raw scene changes were useful, but they were too noisy.
A vision model may describe the same room differently between frames. One frame might include a chair, another might omit it. One frame might mention the rug, another might not. Those are not meaningful events. To address that, I added an event engine. The event engine filters raw changes into meaningful events. It ignores stable background objects like doors, walls, rugs, shelves, desks, chairs, and windows. It focuses instead on watched objects that are more likely to matter, such as phones, laptops, mugs, notebooks, packages, keys, and bags. This turned noisy raw changes into cleaner operational events. Ignored Terms
Python
ignored_terms = {
"room",
"wall",
"floor",
"door",
"chair",
"desk",
"rug",
"shelf",
"window",
"painting",
"office furniture",
"computer monitor",
}
Watched Terms
Python
watched_terms = {
"phone",
"laptop",
"notebook",
"book",
"pen",
"mug",
"bottle",
"bag",
"package",
"wallet",
"keys",
"headphones",
}
v0.5 — Event Stability Engine
Even after adding event filtering, the system still had one more problem: model flicker.
A local vision model may detect an object in one frame, miss it in the next frame, and detect it again later. That can create noisy event output. For example:
In reality, the phone may not have moved at all. The model may simply have been uncertain from one analysis cycle to the next. To solve that, I added a stability engine. The stability engine does not immediately trust a single detection. Instead, it requires an object state to remain consistent across multiple analysis cycles before confirming the event. The logic is simple:
This changed the app from reacting to every one-frame observation to confirming events only when there was repeated evidence. The confirmation threshold is configurable. In this version, I used two analysis cycles. That means if the AI sees a phone once, the system does not immediately report it. If the phone remains present on the next analysis cycle, then the system confirms the event. Stability Configuration
Python
STABILITY_CONFIRMATION_FRAMES = 2
Internally, the stability engine keeps track of stable objects, presence counts, and absence counts.
The stable object list represents what the system currently believes is truly present. The presence counter tracks possible new objects. The absence counter tracks possible removals. Stability State
Python
self.stable_objects = set()
self.presence_counts = {}
self.absence_counts = {}
When an object appears in the current scene but is not already part of the stable object set, the system increments a presence count. Only when the count reaches the confirmation threshold does it emit a confirmed event.
Confirming Object Appearance
Python
self.presence_counts[obj] = self.presence_counts.get(obj, 0) + 1
if self.presence_counts[obj] >= STABILITY_CONFIRMATION_FRAMES:
self.stable_objects.add(obj)
event = self.create_event(
"object_added_confirmed",
f"{obj} appeared",
)
The same approach is used for removals. A missing object is not immediately treated as gone. It must remain missing across the configured number of analysis cycles before the system confirms that it disappeared.
This changed the overlay from showing raw or meaningful events to showing confirmed events. Confirmed Events OutputConfirmed Events: - phone appeared - laptop appeared - mug disappeared Adding People-Count Stability
The next improvement came from a real test.
Someone walked into the room while the app was running. The model detected the second person in the frame, but the first version of the stability engine did not treat that as a confirmed event. That exposed an important design gap. The system could confirm objects, but it was not yet confirming people-count changes. So I extended the stability engine to track the number of people in the scene. Instead of only comparing objects, the app now also tracks the stable people count. If the people count changes and remains stable across multiple analysis cycles, the system emits a confirmed person event. People-Count Concept1 person → 2 people = person entered 2 people → 1 person = person left
The people count is extracted from the model response. If the model returns a list of detected people, the system counts the list. If it returns a number, the system uses that number directly.
Extracting People Count
Python
def extract_people_count(self, result):
people = result.get("people", [])
if isinstance(people, list):
return len(people)
if isinstance(people, int):
return people
return 0
The app then compares the current people count against the stable people count. If the new count remains consistent across the confirmation window, the system confirms either a person-entered or person-left event.
People Event Logic
Python
if new_count > previous_count:
description = "person entered"
elif new_count < previous_count:
description = "person left"
This became one of the most useful capabilities in the lab.
The system was no longer just saying that a person was visible. It was confirming that the room state had changed. A person entered. A person left. That is much closer to how an operational vision system should behave. Current Runtime Behavior
At version 0.5, the application behaves like a complete local vision pipeline.
The camera remains live. The AI model analyzes frames on a timed interval. The system compares scene state, filters noise, confirms stable events, and updates the overlay. Runtime Flow1. Open webcam 2. Display live video 3. Every N seconds, capture a frame 4. Send the frame to Gemma 3 Vision through Ollama 5. Parse the model result 6. Compare current scene to previous scene 7. Generate raw changes 8. Filter changes into meaningful events 9. Confirm events using stability logic 10. Render confirmed events on the overlay 11. Write structured logs
The overlay now displays both runtime metrics and confirmed event output.
Overlay OutputAI Factory Lab - Vision Agent v0.5 Model: gemma3:4b FPS: 24.0 Inference: 3.78s Average: 3.97s Frames: 18 Summary: A home office setting with a man working at a desk. Activity: working at a desk Confidence: high Confirmed Events: - person left Recent Confirmed: 17:29:40 - person entered 17:30:13 - person left
That output is significantly more useful than the original frame description.
The first version could say:
That difference matters. The first is perception. The second is event understanding. What Was Actually Built
This was not just a webcam script. By version 0.5, the project had become a layered local AI application.
Each layer has a specific role. Perception Layer OpenCV + webcam + Gemma 3 Vision Memory Layer Tracks previous and current scene state Event Layer Filters raw changes into meaningful events Stability Layer Confirms events across multiple analysis cycles Presentation Layer Renders metrics and confirmed events on video overlay Logging Layer Writes structured event data for review
The perception layer captures and interprets frames. The memory layer compares state over time. The event layer determines which changes matter. The stability layer decides whether those events are trustworthy. The presentation layer makes the output visible. The logging layer preserves what happened.
That architecture is the real value of the project. The model provides perception, but the application provides structure. Why This Matters for Enterprise AI
This lab is small, but the architecture pattern is highly relevant.
A lot of AI conversations focus on the model. Which model?
Those things matter, but they are not the full system. A model response by itself is not an application. The application needs state management, normalization, filtering, confirmation, logging, and a user interface. That is true whether the input is a webcam, a document, a support ticket, a security event, or a retail shelf. AI Application PatternRaw model output ↓ Normalize ↓ Compare state ↓ Filter noise ↓ Confirm stability ↓ Log event ↓ Present to user
That pattern applies far beyond this webcam project.
For a retail shelf, the question becomes:
For a warehouse, the question becomes:
For physical security, the question becomes:
For enterprise operations, the question becomes:
That is where the engineering around the model becomes critical. Lessons Learned
There were several practical lessons from this build.
First, local vision works, but latency shapes architecture. The app should not expect real-time model inference on every frame. A better design is to keep video live and run AI inference on a controlled interval. Second, model output must be normalized. Vision models are descriptive, not deterministic. The same object may be called phone, smartphone, mobile phone, or cell phone. Without normalization, those become false changes. Third, raw changes are too noisy. A simple set comparison between previous and current objects creates too many false positives. Background objects need to be filtered out. Fourth, stability is required for trust. One model response should not always become an event. Requiring repeated confirmation makes the system more reliable. Fifth, people-count tracking is more useful than generic activity text. Activity descriptions vary too much. People entering or leaving is a clearer and more actionable state change. Key LessonsLessons learned: 1. Local vision works, but inference latency matters. 2. Model output must be normalized. 3. Raw scene changes are too noisy. 4. Stable events require repeated confirmation. 5. People-count changes are more actionable than generic activity text. 6. Version control matters, even in a lab. GitHub and Release Discipline
I treated the project like a real software build, not a throwaway script.
Each major milestone was committed, pushed, merged, and tagged in GitHub. That made the lab recoverable and easier to reason about. v0.1.0 - AI Factory Lab foundation v0.2 - Vision overlay and metrics v0.3.0 - Scene memory v0.4.0 - Event engine v0.5.0 - Event stability engine
That release discipline mattered when an experiment did not fit.
At one point, I tested keyboard shortcuts for querying event history. Technically, it worked. But it did not feel like the right user experience. A vision agent should not require the user to memorize keys. It should eventually support a real chat interface. Because the project was under version control, I was able to revert that experiment cleanly and return to a stable version. That is the difference between tinkering and engineering. What Comes Next
The next direction is not just adding more overlay text.
There are two strong paths forward. The first is a chat interface where the user can ask questions about confirmed events. Examples:
The second direction, and the one I am most interested in next, is turning this into a retail shelf monitoring demo. Instead of using the camera to watch a room, I want to build a small fake grocery store shelf and use the Vision Agent to detect shelf state changes. The agent could monitor a miniature shelf and confirm:
This would turn the project from a general-purpose vision demo into a specific retail operations use case. Next Architecture DirectionNext possible versions: v0.6.0 - Retail shelf monitoring mode v0.7.0 - Shelf event history v0.8.0 - Chat with shelf inventory v0.9.0 - Web dashboard v1.0.0 - Retail edge AI demo
The retail shelf direction is especially interesting because it maps to real enterprise use cases.
Retailers care about shelf availability, planogram compliance, out-of-stock detection, restocking workflows, and edge inference. A small-scale version of that can be built in a home lab with a webcam, a small shelf, and a few grocery products. The core question becomes:
That is a much more concrete use case than simply asking the model what it sees. Closing Thought
This project started with a webcam and a local vision model.
But the real work was not getting the model to describe an image. The real work was building the layers around the model that made the output useful. The model provides perception. The application adds memory. The event engine decides what matters. The stability engine decides what can be trusted. The overlay and logs make the result usable. That is the larger lesson from this lab. AI applications are not just models. They are systems. And the difference between a demo and a useful system is often everything you build around the model. Final Architecture SummaryModel ↓ Perception Application ↓ Memory ↓ Event filtering ↓ Stability confirmation ↓ Logging ↓ User experience Result ↓ A local AI vision system that understands confirmed change
0 Comments
Your comment will be posted after it is approved.
Leave a Reply. |
Categories
All
Recognition
|
