Chunking — Complete Guide

Introduction

Chunking — Complete Guide is essential for teams building Enterprise Local AI Platform — Toolliyo's 100-article Microsoft Agent Framework with Ollama path covering open-source LLMs, Ollama ops, ASP.NET Core integration, Semantic Kernel, AutoGen, local RAG, AI security, K8s/GPU deployment, SaaS patterns, and enterprise projects (CRM copilot, ERP, hospital assistant, multi-agent).

Regulated industries and cost-conscious teams choose local inference for chunking — this lesson shows production patterns, not desktop demos.

After this article you will

• Explain Chunking for local/open-source agent stacks with Ollama and Semantic Kernel
• Apply chunking to Enterprise Local AI Platform (Hospital Assistant)
• Compare cloud-only vs hybrid local inference for cost, privacy, and latency
• Answer interviews on Ollama, SK, AutoGen, local RAG, and AI security
• Connect to Article 44 in the 100-lesson path

Prerequisites

• Software: .NET 8 SDK, Ollama installed, Docker (optional GPU)
• Knowledge: ASP.NET Core, ASP.NET Core Agentic AI
• Previous: Article 42 — Embeddings — Complete Guide
• Time: 24 min reading + Ollama hands-on

Concept deep-dive

Level 1 — Analogy

Chunking on Enterprise Local AI Platform teaches Ollama + Microsoft agent patterns for chunking.

Level 2 — Technical

Chunking grounds Hospital Assistant on local embeddings — chunk/index in pgvector or Qdrant, hybrid BM25+vector, citation-required prompts, tenant filters.

Level 3 — Local agent flow

[User / Internal App / Edge Device]
       ▼
[ASP.NET Core Agent API + Auth]
       ▼
[Semantic Kernel / AutoGen Orchestrator]
       ▼
[Ollama Runtime — phi/llama/mistral/qwen]
       ▼
[RAG Memory — pgvector / Qdrant (local)]
       ▼
[Tools · MCP · Queue Workers · Audit Log]
       ▼
[Project: Hospital Assistant]

Ollama operations

• Models: Pin tags (llama3.2, phi3, mistral) — avoid floating latest in prod
• Health: GET /api/tags before routing traffic; circuit-break to queue
• Hardware: Match model size to GPU VRAM; batch jobs off interactive path
• Hybrid: Policy-based cloud fallback only when data classification allows

Hands-on implementation — Hospital Assistant

Run Chunking on Enterprise Local AI Platform for Hospital Assistant: Ollama local models + Semantic Kernel/AutoGen, RAG with pgvector, tool sandboxing, and offline-capable agent workflows.

1. Install Ollama and pull model (llama3, phi3, mistral, or qwen) for this lesson.
2. Wire Semantic Kernel AddOllamaChatCompletion in ASP.NET Core Program.cs.
3. Implement plugins/tools with read-only defaults and tenant-scoped RAG.
4. Run local golden-task eval — latency and quality vs cloud baseline.
5. Containerize with Docker (API + Ollama sidecar or dedicated GPU node).

Anti-pattern (huge model on CPU, no health check, cloud leak, no eval)

// ❌ BAD — cloud leak, no eval, wrong hardware
var openAi = new OpenAIClient(key); // sends regulated docs to cloud
var huge = await ollama.Generate("llama3.1:405b", prompt); // on laptop CPU — timeout
// No health check, no model version pin, no audit log

Production-style Ollama + Semantic Kernel agent

// ✅ PRODUCTION — Chunking (Hospital Assistant) local stack
builder.Services.AddKernel()
    .AddOllamaChatCompletion("llama3.2", new Uri(_config.OllamaEndpoint));

public class LocalAgentOrchestrator
{
    public async Task<AgentResult> RunAsync(AgentRequest req, CancellationToken ct)
    {
        await _health.EnsureOllamaReadyAsync(ct);
        var kernel = _kernelFactory.Create(req.TenantId, model: "llama3.2");
        kernel.ImportPluginFromObject(new DocsRagPlugin(_pgvector), "docs");
        using var span = ActivitySource.StartActivity("LocalAgent");
        var result = await _agent.InvokeAsync(req.Message, ct);
        await _audit.LogAsync(req, result, modelVersion: "llama3.2");
        return result;
    }
}

Complete example

// Chunking — Enterprise Local AI Platform (Hospital Assistant)
builder.Services.AddScoped<ILocalAgentOrchestrator, LocalAgentOrchestrator>();

Local AI enterprise examples

HIPAA local hospital assistant

Clinical summaries on Ollama inside hospital DC; no PHI leaves network; immutable audit.

Hospital Assistant edge clinics

Small phi model on clinic server; syncs anonymized metrics only when online.

Enterprise Local AI Platform — Hospital Assistant · Article 43

Evaluating local agents

[Fact]
public async Task LocalAgent_PassesGoldenTasks()
{
    var result = await _eval.RunGoldenTasksAsync("hospital-assistant-v1");
    Assert.True(result.SuccessRate >= 0.80);
    Assert.True(result.P95LatencyMs < 8000);
}

Common errors & fixes

• Running Ollama on CPU for 70B models in production — Right-size model to hardware; use GPU nodes or smaller phi/mistral for interactive latency.
• No fallback when Ollama is down — Health checks, queue backlog, optional cloud fallback with data policy gates.
• Embedding locally but sending docs to cloud LLM — Keep full RAG pipeline local when data residency requires — match inference and embedding locality.
• Skipping eval because "it is local" — Golden-task suite still required — local models drift with version bumps and quant changes.

Best practices

• 🟢 Pin Ollama model versions; document in ADR
• 🟢 Keep RAG embeddings and inference on same trust zone
• 🟡 Right-size models — phi/mistral for chat, larger for batch
• 🟡 Monitor GPU utilization and queue depth
• 🔴 Never send regulated data to cloud without explicit policy
• 🔴 Never skip eval because inference is local

Interview questions

Mid level

Q1: Why use Ollama for Chunking instead of Azure OpenAI?
A: Data residency, predictable TCO, offline/air-gap — trade lower model capability for privacy and cost control on Hospital Assistant.

Q2: How do you connect Semantic Kernel to Ollama?
A: AddOllamaChatCompletion with endpoint http://ollama:11434; pin model tags; health-check before invoke.

Q3: Local RAG architecture?
A: Embed with nomic-embed or local model; store in pgvector/Qdrant on-prem; retrieve then Ollama generate with citations.

Senior / architect level

Q4: GPU sizing for production Ollama?
A: Interactive: 7B–13B on single GPU; batch: queue workers on multi-GPU; CPU-only for dev/demo not prod SLA.

Q5: Hybrid cloud/local routing?
A: Policy engine routes sensitive tenants to Ollama; general queries to cloud; log routing decisions for audit.

Q6: Eval local models?
A: Same golden-task suite as cloud — compare success rate, latency p95, and hallucination rate per model version.

Summary & next steps

• Article 43: Chunking — Complete Guide
• Module: Module 5: RAG and Vector Databases · Level: INTERMEDIATE
• Project module: Hospital Assistant

Previous: Embeddings — Complete Guide
Next: Semantic Search — Complete Guide

Practice: Pull one Ollama model and wire SK — commit with feat(ollama-agent): article-043.

FAQ