AI Middleware — Complete Guide

Introduction

AI Middleware — Complete Guide is essential for teams building Enterprise Local AI Platform — Toolliyo's 100-article Microsoft Agent Framework with Ollama path covering open-source LLMs, Ollama ops, ASP.NET Core integration, Semantic Kernel, AutoGen, local RAG, AI security, K8s/GPU deployment, SaaS patterns, and enterprise projects (CRM copilot, ERP, hospital assistant, multi-agent).

Regulated industries and cost-conscious teams choose local inference for ai middleware — this lesson shows production patterns, not desktop demos.

After this article you will

• Explain AI Middleware for local/open-source agent stacks with Ollama and Semantic Kernel
• Apply ai middleware to Enterprise Local AI Platform (Security AI)
• Compare cloud-only vs hybrid local inference for cost, privacy, and latency
• Answer interviews on Ollama, SK, AutoGen, local RAG, and AI security
• Connect to Article 24 in the 100-lesson path

Prerequisites

• Software: .NET 8 SDK, Ollama installed, Docker (optional GPU)
• Knowledge: ASP.NET Core, ASP.NET Core Agentic AI
• Previous: Article 22 — Semantic Kernel Integration — Complete Guide
• Time: 24 min reading + Ollama hands-on

Concept deep-dive

Level 1 — Analogy

AI Middleware on Enterprise Local AI Platform teaches Ollama + Microsoft agent patterns for ai middleware.

Level 2 — Technical

AI Middleware connects ASP.NET Core to Ollama — SK AddOllamaChatCompletion, streaming SignalR hubs, DI-scoped kernels, and JWT tenant sessions for Security AI.

Level 3 — Local agent flow

[User / Internal App / Edge Device]
       ▼
[ASP.NET Core Agent API + Auth]
       ▼
[Semantic Kernel / AutoGen Orchestrator]
       ▼
[Ollama Runtime — phi/llama/mistral/qwen]
       ▼
[RAG Memory — pgvector / Qdrant (local)]
       ▼
[Tools · MCP · Queue Workers · Audit Log]
       ▼
[Project: Security AI]

Ollama operations

• Models: Pin tags (llama3.2, phi3, mistral) — avoid floating latest in prod
• Health: GET /api/tags before routing traffic; circuit-break to queue
• Hardware: Match model size to GPU VRAM; batch jobs off interactive path
• Hybrid: Policy-based cloud fallback only when data classification allows

Hands-on implementation — Security AI

Run AI Middleware on Enterprise Local AI Platform for Security AI: Ollama local models + Semantic Kernel/AutoGen, RAG with pgvector, tool sandboxing, and offline-capable agent workflows.

1. Install Ollama and pull model (llama3, phi3, mistral, or qwen) for this lesson.
2. Wire Semantic Kernel AddOllamaChatCompletion in ASP.NET Core Program.cs.
3. Implement plugins/tools with read-only defaults and tenant-scoped RAG.
4. Run local golden-task eval — latency and quality vs cloud baseline.
5. Containerize with Docker (API + Ollama sidecar or dedicated GPU node).

Anti-pattern (huge model on CPU, no health check, cloud leak, no eval)

// ❌ BAD — cloud leak, no eval, wrong hardware
var openAi = new OpenAIClient(key); // sends regulated docs to cloud
var huge = await ollama.Generate("llama3.1:405b", prompt); // on laptop CPU — timeout
// No health check, no model version pin, no audit log

Production-style Ollama + Semantic Kernel agent

// ✅ PRODUCTION — AI Middleware (Security AI) local stack
builder.Services.AddKernel()
    .AddOllamaChatCompletion("llama3.2", new Uri(_config.OllamaEndpoint));

public class LocalAgentOrchestrator
{
    public async Task<AgentResult> RunAsync(AgentRequest req, CancellationToken ct)
    {
        await _health.EnsureOllamaReadyAsync(ct);
        var kernel = _kernelFactory.Create(req.TenantId, model: "llama3.2");
        kernel.ImportPluginFromObject(new DocsRagPlugin(_pgvector), "docs");
        using var span = ActivitySource.StartActivity("LocalAgent");
        var result = await _agent.InvokeAsync(req.Message, ct);
        await _audit.LogAsync(req, result, modelVersion: "llama3.2");
        return result;
    }
}

Complete example

// AI Middleware — Enterprise Local AI Platform (Security AI)
builder.Services.AddScoped<ILocalAgentOrchestrator, LocalAgentOrchestrator>();

Local AI enterprise examples

SOC log summarization (local)

SIEM exports processed on-prem; mistral summarizes incidents without exfiltration.

Security AI red-team eval

Prompt injection test suite against local agents before production rollout.

Enterprise Local AI Platform — Security AI · Article 23

Evaluating local agents

[Fact]
public async Task LocalAgent_PassesGoldenTasks()
{
    var result = await _eval.RunGoldenTasksAsync("security-ai-v1");
    Assert.True(result.SuccessRate >= 0.80);
    Assert.True(result.P95LatencyMs < 8000);
}

Common errors & fixes

• Running Ollama on CPU for 70B models in production — Right-size model to hardware; use GPU nodes or smaller phi/mistral for interactive latency.
• No fallback when Ollama is down — Health checks, queue backlog, optional cloud fallback with data policy gates.
• Embedding locally but sending docs to cloud LLM — Keep full RAG pipeline local when data residency requires — match inference and embedding locality.
• Skipping eval because "it is local" — Golden-task suite still required — local models drift with version bumps and quant changes.

Best practices

• 🟢 Pin Ollama model versions; document in ADR
• 🟢 Keep RAG embeddings and inference on same trust zone
• 🟡 Right-size models — phi/mistral for chat, larger for batch
• 🟡 Monitor GPU utilization and queue depth
• 🔴 Never send regulated data to cloud without explicit policy
• 🔴 Never skip eval because inference is local

Interview questions

Mid level

Q1: Why use Ollama for AI Middleware instead of Azure OpenAI?
A: Data residency, predictable TCO, offline/air-gap — trade lower model capability for privacy and cost control on Security AI.

Q2: How do you connect Semantic Kernel to Ollama?
A: AddOllamaChatCompletion with endpoint http://ollama:11434; pin model tags; health-check before invoke.

Q3: Local RAG architecture?
A: Embed with nomic-embed or local model; store in pgvector/Qdrant on-prem; retrieve then Ollama generate with citations.

Senior / architect level

Q4: GPU sizing for production Ollama?
A: Interactive: 7B–13B on single GPU; batch: queue workers on multi-GPU; CPU-only for dev/demo not prod SLA.

Q5: Hybrid cloud/local routing?
A: Policy engine routes sensitive tenants to Ollama; general queries to cloud; log routing decisions for audit.

Q6: Eval local models?
A: Same golden-task suite as cloud — compare success rate, latency p95, and hallucination rate per model version.

Summary & next steps

• Article 23: AI Middleware — Complete Guide
• Module: Module 3: ASP.NET Core AI Fundamentals · Level: INTERMEDIATE
• Project module: Security AI

Previous: Semantic Kernel Integration — Complete Guide
Next: AI APIs — Complete Guide

Practice: Pull one Ollama model and wire SK — commit with feat(ollama-agent): article-023.

FAQ