Why should you care?

AIPredict batch re-scoring of millions of rows may need GPU for deep ONNX, not CPU FastTree.

See it live — copy this example

Use a .NET console or Web API project with Microsoft.ML. Run dotnet run after pasting.

var sessionOptions = new SessionOptions();
sessionOptions.AppendExecutionProvider_CUDA(gpuDeviceId: 0);
var gpuPipe = ml.Transforms.ApplyOnnxModel(
    modelFile: "models/embedding-gpu.onnx",
    outputColumnNames: new[] { "output" },
    inputColumnNames: new[] { "input" },
    gpuDeviceId: 0);
var batch = gpuPipe.Fit(schema).Transform(millionRows);

What happened?

• Use GPU for heavy neural scoring batches; keep lightweight tree models on CPU in the API.
• Monitor VRAM.

Practice next

1. Enable CUDA EP in ONNX.
2. Transform large batch.
3. Compare rows/sec vs CPU.
4. Fall back to CPU if GPU unavailable.
5. Profile batch size 512 vs 2048.

Remember

GPU for deep batch. CPU for tree API. Watch VRAM.

AIPredict GPU batch

Nightly embedding job uses CUDA.

Outcome: Overnight window shrinks from 6h to 90m.