|
AI Collab Score: 7 / 3
Last year, I documented my first NVIDIA AI Workbench installation on a repurposed workstation. That project was about validating CUDA, cuDNN, PyTorch, and GPU acceleration. It proved that an older workstation could become a capable local AI development platform.
This rebuild had a different purpose. I did not want to recreate a collection of global Windows installs and hope every framework, runtime, and dependency remained compatible over time. I wanted to build a modular local AI lab that more closely resembles the technical layers enterprise teams must eventually design, secure, operate, and scale. The result is a lean open-source AI stack built around Windows 11, WSL2, Docker Desktop, Ollama, and Open WebUI. More importantly, it gave me a practical answer to a question many people have: How far can a 12 GB RTX 3060 realistically go with modern local models before VRAM capacity, power draw, or responsiveness becomes the limiting factor? The Lab at a Glance
This is not a purpose-built AI server. It is a repurposed workstation configured as a controlled testing ground for local AI workloads.
The architecture is intentionally simple:
Windows 11 → NVIDIA Driver → WSL2 Ubuntu → Docker Desktop → Ollama → Open WebUI → Local Models on the RTX 3060 The lab does not yet include a vector database, agent framework, notebook platform, or observability stack. That is intentional. Those components will come later, once there is a real workload and a clear reason to introduce them. Why Build This After NVIDIA AI Workbench?
NVIDIA AI Workbench remains a useful way to accelerate project-based AI development. My first build was valuable because it taught me how to validate GPU acceleration, install the required components, and benchmark the workstation.
This second build is not a replacement story. It is a learning progression. AI Workbench abstracts many of the operational layers required to make AI projects productive. This open-source rebuild exposes those layers:
That distinction matters. At enterprise scale, the hardware may change from a single RTX 3060 to a cluster of high-end GPUs. But the underlying architecture questions remain the same. The Build Process: Validate Each Layer Before Adding the Next1. Rebuild the Windows and WSL Foundation
The workstation had been completely reinstalled, so the first task was rebuilding the foundation.
The initial WSL installation exposed a Windows registration issue. Rather than layering workarounds on top of a broken base, I repaired the Windows image, updated WSL, and installed Ubuntu cleanly as a WSL2 distribution. That early issue reinforced an important lesson: AI tooling depends on a stable platform. If Windows, WSL2, virtualization, or drivers are inconsistent, every layer above them becomes harder to troubleshoot. 2. Use Docker Desktop as the Container Platform
For this Windows-based lab, Docker Desktop with WSL2 integration became the practical container platform.
Ubuntu remains the Linux-native shell and project environment. Docker Desktop provides the container runtime, Docker Compose support, persistent volumes, and GPU-enabled container access. That separation keeps the host simple while allowing the application stack to remain modular. 3. Validate GPU Access Before Downloading Models
Before deploying Ollama or downloading any local models, I validated that the RTX 3060 was visible inside a GPU-enabled container.
That confirmed the real execution path: Windows NVIDIA driver → WSL2 GPU virtualization → Docker Desktop → CUDA container → RTX 3060 This was the go/no-go test. A local AI lab should not begin with model downloads. It should begin by proving that the host, operating system, container runtime, and GPU path are stable under load. 4. Deploy the Minimum Useful Stack
The first deployment included only two services:
Both services use persistent Docker volumes. Open WebUI is exposed only through localhost, while Ollama remains internal to the container stack. The initial architecture was intentionally small: Browser → Open WebUI → Ollama → RTX 3060 No vector database. No agent framework. No observability stack. The objective was to prove local inference, establish a benchmark baseline, and identify the practical limits of the GPU before adding more complexity. Model Testing: 4B, 8B, and 14B
Once the platform was stable, I tested three practical model tiers.
The goal was not to find the largest model that could technically start.
The goal was to determine which model sizes remained useful in an interactive workflow. Baseline 1: Qwen3 4B
Qwen3 4B established the initial text-inference baseline.
During a structured infrastructure prompt, the RTX 3060 showed:
This was a healthy result. The model was responsive, GPU utilization was high, and the card was operating well below its thermal and power limits. For fast local chat, initial prompt development, and general day-to-day assistance, the 4B tier is highly capable. Baseline 2: Gemma 3 4B for Vision and Rack Review
The next test was a multimodal workflow using Gemma 3 4B.
The use case was intentionally practical: image-based infrastructure review. I tested screenshots and rack imagery with prompts asking the model to identify visible components, describe likely architecture, avoid inventing details, and clearly separate observations from “Need to Validate” items. The GPU results remained healthy:
This demonstrated that a local 12 GB GPU can support lightweight visual-review workflows without requiring major infrastructure investment. It also reinforced a practical lesson: model capability matters. A text-only model cannot process an image attachment simply because it is available in the same interface. Mixed-content workflows require the right model for the right task. That may sound basic, but it is exactly the type of operational detail teams need to understand before building document, vision, RAG, or agent workflows. Qwen3 8B: The Daily Reasoning Model
Qwen3 8B produced the most balanced result.
The model ran with:
The model was fully GPU-resident with no CPU offload. That matters. A model that fits entirely in VRAM avoids the unpredictable latency and performance penalties that can occur when workload execution spills into system memory or CPU processing. For the architecture prompt used in this lab, Qwen3 8B completed end to end in: 59.34 seconds
That made it the clear default model for local infrastructure reasoning.
It offered stronger structured reasoning than the 4B tier while maintaining meaningful VRAM headroom, moderate power draw, healthy thermals, and a usable interactive response time. Qwen3 14B: Finding the Practical Boundary
The most interesting test was Qwen3 14B.
A 14B model on a 12 GB RTX 3060 is where the question changes from: Will it run?
To:
Will it remain practical?
The answer was better than expected.
Qwen3 14B ran with:
The model reported as fully GPU-resident. That is a meaningful result for this class of workstation. The model was not partially offloaded to CPU processing. It remained fully GPU-resident. However, it was clearly operating near the practical edge of the card. The RTX 3060 was using nearly all available VRAM and drawing essentially its full 170W power limit. Thermals remained healthy, but capacity headroom was minimal. The same architecture prompt completed in: 93.80 seconds
That was 34.46 seconds longer than Qwen3 8B, or approximately 58% slower end to end.
This was not a standardized benchmark or a controlled quality evaluation.
It was a practical comparison using the same architecture prompt in the same local environment. Response time can vary with output length, prompt complexity, context, and model generation behavior. But it still answered the operational question that matters: Does the larger model provide enough additional value to justify the additional wait, power consumption, and VRAM pressure?
For this lab, the answer is clear.
The Practical Model Strategy
The results produced a simple operating model for the workstation.
Qwen3 8B is the best default.
It is fully GPU-resident, consumes substantially less VRAM than the 14B model, runs cooler, draws less power, and completes the same structured workload more quickly. Qwen3 14B is still useful. It should simply be treated as a selective deep-analysis model rather than the default choice for every interaction. Lessons Learned From Building It Twice
The first build taught me how to establish GPU acceleration.
The second build taught me how to think about an AI environment as an operational stack. 1. The Host Is Not the Application Environment
Windows should remain stable.
Drivers, virtualization, WSL2, Docker Desktop, storage, and networking belong on the host. AI services should live in containers. That boundary reduces dependency drift and makes the environment easier to reproduce. 2. Start With the Smallest Useful Stack
Ollama and Open WebUI were enough to prove the platform.
Adding a vector database, notebook environment, agent framework, and observability platform on Day One would have created complexity before there was a defined workload for each component. 3. Measure Before Expanding
The benchmark results changed the next design decision.
Without testing, it would be easy to assume the 14B model was too large for a 12 GB RTX 3060. The actual result showed that it could run fully on the GPU. The measurements also showed why Qwen3 8B should remain the default model. 4. Vision Is a Legitimate Local Use Case
A small multimodal model can be useful for screenshots, rack imagery, diagrams, and visual-review workflows.
It will not replace an enterprise computer-vision platform. But it is enough to prototype the workflow, identify the limitations, and determine where the value exists. 5. Local AI Is About Learning the Operational Layers
The value of a local lab is not simply privacy or avoiding cloud cost.
It is learning. A local lab makes the infrastructure decisions visible:
Those are the same questions that appear at enterprise scale. The hardware changes. The architecture thinking does not. What Comes Next
This build is intentionally not complete.
The next phase is not to install every available AI tool. The next phase is to add the first real workload: an AI Infrastructure Copilot. That will introduce:
That is where a vector database, retrieval workflows, and eventually observability become useful. The workstation has now proven it can support the model layer. The next question is whether it can support a disciplined retrieval workflow that improves infrastructure design work without creating false confidence. That will be the next build.
0 Comments
Your comment will be posted after it is approved.
Leave a Reply. |
Categories
All
Recognition
|
