Building an AI Assistant That Does Not Trust Its Inputs
A security-analysis assistant reads exactly the material most likely to be messy, contradictory, and outside its control. Architecture documents, copied tickets, vendor manuals, retrieved pages, and previous model output may all contain text that is wrong or text crafted to influence the model. Treating every sentence as an instruction would give the least trustworthy input the most authority.
DeviceProphet AI Assistant therefore starts from a simple rule: documents are evidence, not policy.
Consider a common claim: “secure boot is enabled.” The same evidence set may show only that the SoC supports secure boot. It may omit the production fuse state, the trusted key hash, or authentication of a later boot stage. The assistant must preserve that uncertainty instead of completing the story on the product’s behalf.
Untrusted evidence stays evidence
System policy, customer evidence, retrieved content, model output, and tool results remain separate across explicit data, model, and tool boundaries. Text inside a document cannot replace system policy, promote itself to a trusted fact, or grant a tool permission. That is true whether the text says “ignore previous instructions” directly or reaches the workflow indirectly through a retrieved source.
NIST defines prompt injection as an attack in which an adversary crafts input to cause unintended model behavior. There is no honest “prompt-injection-proof” switch. The assistant’s direct and indirect injection defenses are exercised against a regression corpus as continuing evaluation. New failures become new cases; passing the corpus does not establish immunity.
Retrieval-augmented generation follows the same rule. Retrieval can find a relevant fuse table or boot-flow paragraph, but selection does not turn that paragraph into an instruction. Retrieved material keeps its source attribution and remains evidence to be evaluated.
Important facts get an independent check
Free-form prose is useful for explanation, but it is a poor boundary between a model and a security analysis. The assistant validates model output into typed facts: the claim, its source references, the affected component, and a validation state. Unknown or unsupported details remain unknown instead of being filled with plausible text.
For the secure-boot example, an extracted fact might say that the platform supports signature verification and cite the relevant manual section. It must not silently upgrade that fact to “production devices enforce secure boot.” Consequential extracted facts go through an isolated verification pass against the original evidence before they enter the structured analysis context. The pass may reject the fact, narrow it, or leave it for human confirmation.
Independent checking is a boundary, not a magic number of models. It does not always require two different providers, and it does not make the accepted fact infallible. Its purpose is to stop an initial extraction from becoming trusted merely because the same workflow produced it.
The invocation record keeps the model and version, source context, retrieved evidence, tool calls, structured output, and validation decision needed to reconstruct the path. That makes “why did this appear in the analysis?” a question that can be answered from evidence rather than memory.
Tools receive narrow authority
Connecting a model to tools raises the stakes: a bad answer is one problem; a bad answer with broad authority is another. DeviceProphet AI Assistant uses allowlisted, typed, least-privilege tool access that is read-only by default. It provides no arbitrary shell, network, or filesystem authority.
The same principle applies when tools use the Model Context Protocol. MCP’s official security guidance emphasizes authorization boundaries and confused-deputy risks; adopting the protocol does not make a tool safe by itself. The implementation still constrains what can be called, validates arguments and results, and records the invocation. Document text cannot add a new tool to the allowlist.
In the running example, a retrieval tool may read an approved evidence collection. It cannot burn fuses, rotate keys, change an HSM policy, publish firmware, or browse an arbitrary network because a document or model asks it to.
Deployment boundaries must be explicit
The assistant workflow runs locally and supports multiple LLM APIs. With customer-local inference, supplied documents remain in the customer’s environment. If the customer selects a remote API, the context sent to that API enters the provider’s trust boundary. Calling the surrounding workflow local does not erase that transfer.
For sensitive environments, enclave-isolated delivery can be arranged with an isolated workload and customer-specific secrets. Hardware-backed isolation or attestation is claimed only when the chosen platform actually provides it. Retention, operator access, regional processing, and other guarantees must likewise come from the deployed configuration and engagement terms, not from a generic enclave label.
At the end of the secure-boot example, the assistant can state what the evidence supports, what remains missing, and which contradiction needs attention. It cannot certify enforcement from an unsupported sentence. Deterministic checks and an engineer retain final authority over the consequential conclusion.
If this security model fits an evidence-heavy device review, ask about DeviceProphet AI Assistant access.