What a ViKernel Demo and a Security Test Count Actually Measure
evaluationagentssecuritymeasurement
A test count, a timing comparison, and a screenshot answer different questions. Reading them as if they were interchangeable makes a product harder to evaluate.
QuantID works with VIDRAFT as a technology partner. This article examines the scope of VIDRAFT's public ViKernel claims; it does not represent independent third-party verification of its partner's product.
What is the unit behind 661
AI Times and ZDNet Korea reported on ViKernel on November 3. The announced 661 tests are company-reported tests involving four public models, mock tools, and fake data.
The company reports no unrequested outbound transfers in those tests. The unit is a test under that setup. It is not a customer, a real data-loss event, or a deployment certification. Mock tools and fake data can make an experiment controlled and safer to inspect, while leaving real integrations as a separate question.
The underlying public proposal is a weight-native CPU and security-OS design combined with a model while retaining its original weights. Signed resumption and audit records are part of the described control structure. Its intended value is to prevent ordinary input text from changing execution authority. That design claim and the API-plus-gate configuration observed below are separate kinds of evidence. Public design and code
A reader considering deployment should ask how the test conditions match the intended application. The model, tool interface, user permissions, and expected behavior of normal requests all affect what a comparison can establish.
What is the boundary around 21 times
The announced 21 times figure concerns the policy-decision stage. It is a company-reported comparison of that stage, not an end to end agent speedup.
The distinction can be summarized without inventing a new measurement:
| Evidence | What it describes | A different question |
|---|---|---|
| Reported test count | The scope of a controlled company test set | Behavior across every production integration |
| Policy-stage timing comparison | Time attributed to that decision stage | Total time to finish an agent task |
| Demo screenshot | A visible state at a particular observation | General reliability or an unseen hardware path |
If the intended question is total task time, the total task must be measured. If the intended question is whether actions were appropriately allowed, an outcome measure must address that. Speed and policy correctness are not substitutes for one another.
Why does replay need its own label
The current ViKernel Desktop README distinguishes AgentGuard's LIVE path from replay fallback. A replay is useful for explanation, but it does not create a fresh observation for the current request. The observed interface also displayed LIVE on Policy Engine, so a blanket claim about the mode of every app would go beyond that observation. Public demo description
Two manually selected mock actions were observed on November 4. A weather request for Busan displayed ALLOW, rule A, audit 252. An email request to a non-deliverable example address displayed DENY, rule F, audit 253. These are two UI outcomes, not two independent security experiments. No real email was sent or prompt-injection scenario executed.
The displayed configuration was Spark API plus a ViKernel gate in mode B, the CPU reference mode. This is the appropriate description of the observation; it does not establish execution inside Spark's weights. The audit numbers identify the displayed records and should not be retold as a count of tests performed by this observer.
A screenshot caption should therefore identify the visible mode and result. The image is evidence for what was displayed, while the explanation of how that state was produced supplies necessary context.
How could a future comparison be made easier to inspect
A useful comparison would name the exact action being evaluated and keep the model, tool permissions, and inputs fixed. It would record ordinary tasks as well as requests containing untrusted content. Policy outcomes, actual tool outcomes, policy-stage time, and total task time would be separate observations.
That is a proposed evaluation design, not a new experiment reported here. Its purpose is to make the next result interpretable. The strongest explanation of a security product is not the largest number in the headline. It is a clear match between a claim, the conditions of the test, and the outcome that was actually observed.