What the semantic layer guarantees

The semantic model library publishes three metric views and the Genie space instructions around them. The natural next question is what all of that actually guarantees — and the honest answer is: one narrow thing, enforced well, surrounded by conventions that nothing checks. This page draws that line precisely, because the line is where analytics-with-AI projects go wrong: teams assume the governed layer is doing work it never claimed to do.

Everything here is authored house convention. Where the engine enforces a rule, the row says so and says how far the enforcement reaches. Where it does not, the practice states the discipline it uses instead — and a convention is a choice, so each one names what it protects against.

Provenance.Rows whose basis reads “verified in one workspace” were tested hands-on on 2026-08-27— one workspace, one date, one serverless SQL warehouse, not a vendor commitment. Rows whose basis reads “documentation” are read from Databricks product documentation retrieved the same day; product names and pages move, so treat every row as perishable and re-verify before relying on it. The Snowflake section is documentation-only throughout, and says so.

If you take one thing: run the fourteen questions in the acceptance checklist against your own space, with answers you computed first. That run — not this page, and not any benchmark — is the evidence that your layer is doing its job.

Six statements the practice holds as the outer boundary of any natural-language analytics system, whatever the platform. Every row and checklist item on this page is one of these boundaries made specific.

1. Platform benchmarks are not enterprise correctness evidence. A benchmark measures the tasks in the benchmark — its schemas, its questions, its grader. Little of that transfers to your schema, your vocabulary, your access model, or your definitions. The only evaluation that counts is a run on your own semantic layer with answers you computed independently first, which is what the checklist below exists for.
2. SQL visibility is not semantic validation. Showing the generated SQL is audit evidence: it lets a reader check the query after the fact. It does not make the query right. A visible query reassures; it does not verify — reading SQL back is a skill, and checking a query after the fact is a different act, done by a different person, from asking the question. Treat the SQL as the trail, never as the proof.
3. A governed semantic layer narrows interpretation. It cannot guarantee correct questions, joins, data, or answers. The layer fixes definitions, which removes the largest source of being confidently wrong — a model inferring a metric from raw columns. Still open after that: whether the right question was asked, whether the rows in scope are the rows the asker meant, whether the join path fans out, whether the data underneath is right, and whether the generated query reached for the governed measure at all. Every row in the table below is one of those gaps, located.
4. Confidence without calibration is not probability. A fluent answer carries no per-answer correctness score — none is documented for Genie responses — and a model's tone is not a probability. Never present an unvalidated self-score as one. Where no calibrated signal exists, use a rule-based status — verified, unverified, failed — instead of a percentage that means nothing.
5. User satisfaction is not accuracy. A thumbs-up measures whether the answer was pleasing and plausible. Silent reliance on a wrong answer looks identical to satisfied use of a right one, and silent reliance produces no complaint at all. Sample successful-looking answers for review; do not monitor complaints and call it quality.
6. “Produced an answer” is not success. For many questions the right behaviour is a clarifying question, a refusal, or a declared abstention — an ambiguous scope, a restricted value, a failed query. A system that always answers is failing exactly those cases, fluently. The checklist scores clarify, refuse and abstain as passes where they are the right outcome.

Fourteen questions to point at your own Genie space — or any natural-language interface over your semantic layer — before anyone trusts it. The list is response-free on purpose: it publishes no expected numbers, because the right answers come from your data. Each item names the wrong construction it targets and the probe to ask; you supply the scope where you already hold the correct answer.

How to run it. Compute each gold answer independently first. Write down the expected behaviour class per item — for several of the fourteen a pass is not a number but a clarifying question, a refusal, or a declared abstention. Then ask, verbatim where you can, adapted where your vocabulary differs. Add two or three ordinary, plainly-authorized questions from your own reporting alongside the fourteen, so that over-refusal has somewhere to show up. Grade against what you wrote down, not against how the answer felt.

A run of the checklist is only as honest as its scorecard. Five rules, each one existing because the convenient alternative hides a failure mode.

• Score security separately, and never average it. A restricted-data leak and a formatting miss are not two points on one scale. Report SG-11 on its own line; one failure there is a finding, whatever the other thirteen say. An accuracy figure that has a security failure averaged into it is not an accuracy figure.
• Report coverage beside correctness. A system that clarifies, refuses or abstains on everything scores clean and answers nothing. Count how many answerable items got answers: high accuracy at low coverage is a different result from high accuracy at full coverage, and only reporting both makes refuse-everything visible.
• A perfect score is a statement about your space, on these questions, on that day. Not about a model, not about a platform, and not about next month. Definitions change, models update without notice, and generation is documented as nondeterministic — re-run the list on every semantic, schema or model change, and run it more than once even when nothing changed.
• Compute the gold answers first, independently. The system's own output never becomes its own gold label. If you did not hold the right answer before the question was asked, the run tested your patience, not the system.
• Write down the expected behaviour class per item, then grade against it. For several of the fourteen the right response is not a number. Decide before the run which items must clarify, refuse or abstain, so that “it gave an answer” can be scored as the failure it is on those items.

The Cortex Analyst translation in the semantic model library shows the concrete YAML this section is talking about.

• Genie & Cortex Semantic Models — The three metric views these guarantees are scoped to — the definitions, the space instructions, and the switch ledger against the dictionary.
• Unity Catalog Governance Patterns — tested limits — The same one-workspace discipline, from the same 2026-08-27 run, applied to grants, masks, tags and lineage — including the policy-visibility gaps the table above points at.
• Supply Chain KPI Dictionary — The definitions themselves. A semantic layer can only be as right as the definition it stores.

Everything on this page is free to copy, adapt, and run — no attribution required, no email needed. If a claim here disagrees with what your own workspace does, your workspace is the evidence: this page carries its dates so you can tell which of us is out of date.