What does private AI mean?
“Private AI” is used for different product categories. Google describes Private AI Compute as an architecture that connects on-device intelligence with cloud models while, according to Google, providing privacy and security assurances through its stated design. That is a current vendor claim, not a universal definition of private AI (Google). VMware says VCF Private AI Services is intended to help organizations balance AI gains with privacy and compliance needs and offers flexibility to run a range of AI platforms for an environment. This is VMware’s description of its service, not a guarantee or a universal definition of private AI (VMware).
The variation is useful. The label “private” does not by itself identify every computing boundary. A buyer should not try to settle the word in the abstract. The buyer should turn it into a deployment map.
For a boutique consulting firm, that map should cover the working system, not only the model. An AI workforce may include an interface, retrieval, agents, integrations, identity controls, logs, and approval steps. Our guide to AI agent architecture explains how those parts fit together. The private AI decision begins when the firm assigns a location and operator to every part.
Which data crosses a boundary?
Start with a single client task. Follow its data from entry to output. Do not begin with the product’s headline. Begin with what actually moves.
Buyer boundary matrix
Use this matrix to record what the evidence says about each component. It is a buying aid, not a legal, privacy, security, or compliance assessment. Fill every cell from product documentation, configuration records, contracts, and direct inspection where available.
| Component | Location | Operator | Retained data | Evidence |
|---|---|---|---|---|
| User interface | Browser, desktop, firm server, or vendor service | Firm, software vendor, or hosting provider | Prompts, drafts, uploads, or none declared | Architecture page, settings export, data terms |
| Retrieved files | Firm storage, managed database, or external service | Firm or named provider | Source files, chunks, indexes, metadata | Storage configuration, provider documentation |
| Model inference | User device, firm server, dedicated cloud, or shared service | Firm, cloud provider, or model provider | Inputs, outputs, telemetry, or none declared | Endpoint configuration, vendor statement, logs |
| Identity | Firm directory or external identity service | Firm or identity provider | User identity, group, session, access events | Identity configuration, access policy, audit export |
| System logs | Device, firm server, observability service, or multiple locations | Firm or service provider | Prompts, tool calls, errors, metadata, approvals | Logging configuration, sample event, retention setting |
| Admin access | Local console, remote support channel, or provider control plane | Named firm role or named provider role | Admin actions and support records | Role list, support terms, admin audit trail |
| Approval gate | Firm-controlled workflow or external workflow service | Named reviewer or workflow operator | Request, decision, timestamp, changed output | Workflow configuration, approval record |
The point is not to force every component into the same box. It is to make each crossing explicit. “No client files leave firm storage” is a narrower and more testable statement than “everything is private.” It still needs evidence from the actual configuration and relevant provider terms.
┌──────────────────── Firm-controlled boundary ────────────────────┐ │ │ │ Identity ──► Local workspace ──► Retrieved files │ │ │ │ │ ▼ │ │ Approval gate ──────────────► Action │ │ │ │ │ └────────► Firm logs │ │ │ │ Local model ◄──────── optional inference route ────────┐ │ └─────────────────────────────────────────────────────────┼────────┘ ▼ ┌─ External model boundary ─┐ │ Model endpoint │ │ Provider-side operations │ │ Declared retention │ └────────────────────────────┘
Can a firm run its own private AI?
Yes, a firm can operate a stack in which selected AI components run on infrastructure it controls. But that sentence is only a deployment description. It does not prove that every dependency is local, that no data leaves a boundary, or that the system satisfies a legal or privacy requirement.
The practical test is operational ownership. Who maintains the model endpoint? Who updates the retrieval service? Who controls user accounts? Who can change the logging destination? Who approves an action when an agent is about to send, publish, or modify something?
A firm does not need to treat local operation as an all-or-nothing identity. It can keep files and logs under its control while using an external inference endpoint, provided that this is an explicit boundary decision. It can also operate model inference locally while using an external identity provider. Each is a mixed deployment, and each needs its own completed table.
That framing also answers the objection, “Private means no data can ever leave the box.” That may be a firm’s chosen requirement, but the label alone does not demonstrate it. Inspect each boundary. If the requirement is no external inference, confirm the endpoint. If the requirement is local logs, confirm the log destination. If the requirement covers identities and support access too, inspect those separately.
Which deployment patterns exist?
The patterns below are comparison categories, not product rankings or security grades. A real system may combine them.
| Deployment pattern | Model inference | Files and retrieval | Identity and logs | Buyer’s boundary question |
|---|---|---|---|---|
| Local workstation | On a user-controlled device | On the device or firm storage | May be local or external | Which background services still connect outside the device? |
| Firm-operated server | On infrastructure operated by the firm | On firm-selected storage | Firm-selected services | Which maintenance, update, and support paths permit external access? |
| Dedicated hosted environment | In a hosted environment assigned to the firm | In selected hosted storage | Firm or provider services | Which control-plane roles remain with the provider? |
| External model endpoint | At a model provider | Files may remain local while selected context is sent for inference | Identity and logs may be split | Exactly what request content and metadata reach the endpoint? |
| Managed private AI service | As described by the service architecture | As configured across firm and provider systems | Shared operational responsibility | Which parts does “private” cover in the vendor’s documentation? |
| Hybrid stack | Local for some tasks, external for others | Split by workflow or data class | Split across operators | What rule selects the route, and where is that decision recorded? |
Do not compare these patterns using a single “privacy” column. Compare locations, operators, retained data, and evidence. If a workflow moves between patterns, document the routing decision. The AI agent workflow guide can help you separate the steps before assigning boundaries.
See the AI Jungle OS cockpit to see how workflows, approvals, and operating visibility can be organized around explicit deployment boundaries.
What should a privacy claim specify?
A useful claim has a subject, boundary, operator, data type, and evidence. “Prompts are not retained by the model endpoint under the selected service terms” is structurally more useful than “enterprise-grade privacy,” but the buyer must still verify the applicable terms and configuration.
NIST calls its Privacy Framework a voluntary tool intended to help organizations identify and manage privacy risk (NIST). Separately, this article’s boundary matrix is not a NIST assessment or certification.
Ask a provider to specify:
- the component covered by the claim;
- the data covered, such as prompts, files, outputs, account data, or telemetry;
- the processing and storage locations it is willing to state;
- the operator of each relevant service;
- the retained data and the applicable retention statement;
- the roles that can obtain administrative or support access;
- the configuration on which the claim depends; and
- the document, setting, or record that supports the claim.
Use the provider’s answers to complete the boundary matrix and mark any unsupported answer as unknown.
Red flags in a private AI pitch
Treat these as reasons to ask another question, not proof of a defect:
- “Private by design” with no named component or boundary.
- “Runs in your environment” with no explanation of identity, logging, updates, or support access.
- “Your data never leaves” with no definition of data, endpoint, telemetry, or operational exception.
- “No retention” without a named service, applicable terms, and configuration.
- A diagram that shows the model but omits retrieval, integrations, logs, and administrators.
- A self-hosted label attached to an interface that still calls an unspecified external endpoint.
- A privacy framework named in a way that makes a product claim sound like a certification.
- One answer for production, support, diagnostics, and backups when those functions use different operators or locations.
Who can administer the system?
Administration is a separate boundary. For infrastructure in a firm-controlled account, check hosting controls, support channels, identity systems, deployment tools, and update mechanisms for other access paths. Whether those paths exist is a configuration and service question, not something the word “private” answers.
Create an administrator inventory for every component. Record the named role, employing organization, access path, scope, approval mechanism if present, and available evidence. Include emergency and support roles if the relevant provider documents them. Do not infer access from a logo or hosting location.
For an AI workforce, connect administrative access to action authority. A model endpoint may only produce text, while an integration can send an email, update a CRM, or alter a client file. Map those permissions in the AI agent integration guide, then decide where a human approval gate belongs.
What should be logged?
Log design should begin with a purpose. A consulting firm may choose to record who requested an action, which workflow ran, which tools were called, whether approval was required, and what final action occurred. Treat prompts, excerpts, outputs, identifiers, and error details as candidate log fields to verify. Give every log destination its own location and operator in the matrix.
Use an event inventory rather than “log everything.” For each event type, write down:
- why the event is recorded;
- which fields it contains;
- where it is stored;
- who operates that store;
- who can read or export it;
- what retention setting applies; and
- which record proves the active configuration.
If logs go to an external observability service, add that service to the same boundary table. If prompts are excluded, verify the actual event payload. If approval decisions are recorded elsewhere, link the two records by a workflow identifier that the firm defines.
How should a buyer compare private AI options?
Compare options against one workload first. “Draft a client memo from an approved document set” is easier to inspect than “deploy private AI across the firm.” Draw the workflow, complete one boundary table per option, and mark unknown cells as unknown. Do not award implied credit for marketing language.
Buyer checklist
- Define the workload, users, source files, outputs, integrations, and approval point.
- List every component that processes or records the workload.
- Record the location and operator for prompts, retrieval, inference, identity, logs, and administration.
- Ask what data each component retains and which evidence supports the answer.
- Separate vendor-stated architecture from configuration-specific evidence. Google’s Private AI Compute description, for example, is Google’s statement about its architecture, not a definition that transfers to another service (Google).
- Check what the vendor means by “private” for the service being offered, and do not transfer that description to a different deployment.
- Identify every administrative and support role.
- Inspect the active inference endpoint, storage destination, identity configuration, and log destination.
- Record unknowns instead of converting them into assumptions.
- Re-run the worksheet when a model, integration, provider, or routing rule changes.
- Ask qualified advisers for legal, contractual, privacy, or security conclusions.
Once the boundary is clear, compare how each option supports the actual workflow: the actions it must perform, the evidence a reviewer needs, and the points where a person approves an external effect. If the firm is deciding between an internal build and managed help, the AI automation services guide provides a separate way to frame delivery ownership.
FAQ
Is private AI the same as self-hosted AI?
Not necessarily. “Self-hosted” describes who hosts some part of the stack. It does not, by itself, locate every model call, identity service, integration, log store, update path, or support channel. Complete the boundary table for the deployed configuration.
Does private AI mean no data leaves the firm?
Only if that is the stated requirement and the evidence supports it across every relevant boundary. Inspect prompts, retrieved files, inference, identities, logs, administration, integrations, and support paths separately. This is an inspection method, not a confidentiality or privacy guarantee.
Can private AI use an external model?
It can describe a mixed architecture in which an external model processes selected requests. Google, for example, uses “Private AI Compute” for an architecture that Google says connects on-device intelligence with cloud models (Google). A buyer still needs to inspect the specific service, data, terms, configuration, and evidence.
Is an on-premises model automatically private?
The model’s location answers only one row of the table. Retrieval, identity, logging, integrations, administration, updates, and support may follow different paths. “On premises” should be treated as a location statement for named components, not a conclusion about the full system.
Does this checklist prove legal compliance or security?
No. The matrix helps make deployment boundaries explicit; it does not prove legal compliance or security. Use qualified professionals and the applicable authoritative materials for legal, privacy, or security conclusions.
What is the first document to request from a vendor?
Ask for an architecture description that names components, locations, operators, retained data, and administrative paths. Then request evidence for the active configuration. A generic diagram is useful only if you can map it to the exact service and settings under evaluation.
Private AI becomes a workable buying category when the noun stops carrying the whole claim. Map the system. Name the boundary. Name the operator. Record retained data. Demand evidence for the deployed configuration. Keep unknowns visible.
See the AI Jungle OS cockpit and examine the operating layer for an AI workforce built around explicit workflows, approvals, and boundaries.
Written by Tileo, who operates a portfolio of internet businesses on this same cockpit.
