- Home
- /
- AI Navigator
- /
- Getting Started
- /
- SAS AI Navigator Policy Creation: LLM Prompting Guide
- RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
SAS AI Navigator Policy Creation: LLM Prompting Guide
- Article History
- RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
|
Policy creation SAS® AI Navigator Policy Creation: LLM Prompting Guide Step-by-step LLM workflow for building custom policy packs |
|
Last updated |
2026-09-25 |
|
What’s Being Built Back to top ↑ |
A policy pack is two CSV files – a Policy Metadata CSV and a Policy Questions CSV – imported into SAS AI Navigator (AIN) to define a governance policy and its compliance assessment questions. This workflow also produces a third CSV, the audit file, which is retained outside AIN as a traceability record. This guide walks through using a large language model (LLM) to analyze source material, such as an organization’s internal AI use policy, data governance policy, or model risk management standard, propose metadata values, and draft assessment questions.
This guide is a script for a back-and-forth with an LLM – not a document to read straight through. Paste a prompt, review the model’s output, and correct anything wrong before continuing – an interactive, multi-turn session that ends with two importable CSV files plus one audit file.
The user of this guide acts as the reviewer, not the spectator. In some areas, the LLM proposes; the user makes the ultimate decision.
When the LLM places [CONFIRM] in a metadata field, the source does not provide enough information to verify the value. Replace [CONFIRM] with a confirmed value or deliberately leave the field blank before proceeding to Step 3. [CONFIRM] is used only for unresolved metadata fields and must not appear in assessment questions or finished CSV files.
How the guide is structured:
- Steps 1-6 are the workflow. Follow them in order.
- Light blue boxes contain the prompts. Copy them and paste them to the LLM.
- [ALL CAPS BRACKETS] are placeholders. After copying a prompt into the LLM chat, replace them in the pasted copy before sending it.
- “Before moving on, check” lists are quick self-checks between steps.
- Appendices are references – refer to them for further support.
The interaction rhythm: paste a prompt → LLM responds → user reviews and corrects → LLM waits for additional user input before moving on. The prompts are written to make the model stop and wait at each stage. This is intentional: it keeps the model from racing ahead on an unconfirmed value or generating questions too early.
Files produced
|
File |
Columns |
Purpose |
|
[PolicyName]_METADATA.csv |
9 |
Imported into SAS AI Navigator – defines the policy |
|
[PolicyName]_IMPORT.csv |
8 |
Imported into SAS AI Navigator – the assessment questions |
|
[PolicyName]_AUDIT.csv |
10 |
Kept for records – the traceability record, not imported |
The metadata CSV is produced in Step 3 and the audit CSV in Step 4. The import CSV is produced last, in Step 5, only after the question set has been reviewed – so the only version of the import file that ever exists is the reviewed version.
Filename convention
[PolicyName] is placeholder notation. The square brackets do not appear in the finished filenames.
A filename-safe prefix is established when the metadata CSV is created. Reuse that exact prefix for all three policy-pack files:
- [PolicyName]_METADATA.csv
- [PolicyName]_AUDIT.csv
- [PolicyName]_IMPORT.csv
Filename normalization applies only to the filenames and is used for LLM prompting guide workflow purposes only. It does not change the confirmed Policy Name that will appear in SAS AI Navigator or any other value stored inside a CSV. Follow the filename rules in the metadata-file step when establishing the prefix.
|
What’s Needed Back to top ↑ |
- Source material: the regulation, standard, framework, or internal policy that needs to be codified (PDF, document, or URL). Pasting or attaching the actual text (PDF or document) is strongly preferred. A URL is a last resort: many LLMs cannot reliably fetch URLs and will quietly answer from training instead of the live page. If a URL is used, Step 1 should confirm the model retrieved it before anything it reports about the contents is trusted.
For sources distributed across multiple documents or webpages: Provide every framework document, implementation guide, taxonomy, register, appendix, or linked resource needed for the intended policy pack. If the supplied source refers to missing material needed to interpret its requirements, the LLM will flag the omission and stop.
- Two blank CSV templates attached to this article (also available for download from the ‘Add Policy’ wizard within SAS AI Navigator): “Policy_Metadata_Template.csv” and “Policy_Questions_Template.csv”
- Reference file to upload to the LLM session (attached to this article): “Policy_CSV_Template_Instructions.md”
- An LLM – Microsoft Copilot, Claude, ChatGPT, Gemini, etc.
A note on LLM choice
- This workflow leans on three things smaller models do poorly: verbatim quoting from a long source, whole-document gap checks, and strict formatting discipline.
- Use a strong, long-context model for anything large or high-stakes.
- On a smaller or local model, run the source one section at a time (Step 4b, Option 2) and check every quoted passage by hand – don’t rely on the model’s self-review.
|
Key Terms Back to top ↑ |
Policy Classifications (optional)
A policy can optionally categorize a use case’s risk level. These risk levels are defined as Classification Options, which can use in-house terminology, and each one – however many a policy defines – maps to one of the three fixed AIN Classification Types:
|
Type |
Meaning |
|
prohibited |
Cannot be deployed under any circumstances |
|
high-risk |
Highest risk that still permits deployment |
|
standard-risk |
Deployable with appropriate governance documentation |
Question Levels
|
Level |
Where It Appears |
Who Answers |
|
organization |
Once; shared across all use cases with this policy |
Administrator / Compliance Officer |
|
use-case |
Each use case this policy is applied to |
Asset Owner / Contributor |
|
model |
Each model linked to a use case with this policy |
Asset Owner / Contributor |
|
agent |
Each agent linked to a use case with this policy |
Asset Owner / Contributor |
SAS AIN does not currently support multi-step questions, also called conditional branching. In other words, if a specific answer to a question (like answering “Yes” to “Is the model high-risk?”) reveals additional questions that are conditionally relevant, AI Navigator cannot support its structure for now. Once a policy is applied to a use case, all of its questions appear on that use case and its linked assets. This is why conditional questions use an N/A path (see Step 4).
Answer Types
|
Type |
Options Format |
|
boolean |
Exactly 2, pipe-delimited: Yes|No |
|
multiple-choice |
3 or more, pipe-delimited: Yes|No|Partial |
|
text |
Leave answer options blank |
Choosing an answer type
Choose the Answer Type that provides the information needed to assess compliance and risk without creating unnecessary response burden.
Use boolean when:
- the complete requirement supports a genuinely binary compliance determination;
- Yes means the complete stated requirement is satisfied and No means it is not;
- no meaningful partial state or applicability path is needed; and
- a narrative response would add little value to the risk decision.
State the substantive requirement. Avoid vague questions such as “Does a process exist?”
Use multiple-choice when:
- the source defines three or more categories, states, tiers, roles, or implementation approaches;
- partial implementation is a meaningful risk state; or
- an N/A option is needed because AIN does not use skip logic.
Use source-defined options when available. Do not invent unsupported categories, maturity levels, or risk tiers. Yes|No|N/A is multiple-choice, not boolean.
Use text when:
- the assessor needs to identify or describe an owner, control, process, rationale, scope, cadence, method, evidence location, exception, gap, or mitigation; or
- narrative information materially improves assessment of implementation, operation, risk exposure, or follow-up action.
Use paired questions selectively. Paired questions are those that are related – for example, one question – a status question – may whether the organization has a certain process, and the next question – an explanation question - asks to describe the process. Use only when they support distinct governance decisions. Do not automatically pair every status question with an explanation question.
Metadata Review Flag
[CONFIRM] – This is a placeholder. Every [CONFIRM] must be replaced with a real value or deliberately left blank before the CSV is built – no [CONFIRM] should remain in a finished file.
Governing Body is the most common example, because it refers to a committee in the user’s organization, not necessarily in the source document.
|
The Workflow Back to top ↑ |
At a glance: the LLM proposes the classification mapping and metadata (Steps 1–2), produces the metadata CSV (Step 3), then generates the assessment questions and produces the audit table and audit CSV (Step 4). Next, review the pack against the source, confirm the findings, and produce the import file (Step 5), then import the two SAS AI Navigator files (Step 6). Nothing enters AI Navigator until Step 6.
|
1 |
Upload Files and Introduce Source Material Back to top ↑ Steps 1 through 3 focus only on metadata CSV creation. The LLM must not begin question generation until the metadata is validated. Replace all [ALL CAPS IN BRACKETS] placeholders before pasting a prompt. Optional – reuse the setup with a project. If the LLM supports projects/custom GPTs/agents, create one and attach Policy_CSV_Template_Instructions.md along with the blank templates there once. Then start a new conversation inside the project for each regulation. This allows repeated access to the reference files automatically without re-uploading, while keeping each policy's questions in their own clean context. Two cautions: (1) start a fresh conversation per regulation so questions from one policy don't bleed into another, and (2) still run the Step 5 review in a brand-new conversation (or a second model) that sees only the source and the question list. Before sending the prompt
Download the two blank CSV templates and reference instruction file attached to this article (templates are also available for download from the 'Add Policy' wizard in SAS AI Navigator).
If the browser adds a duplicate-download suffix such as (1), (2), or -2, rename the files to match the filenames above. Confirm that the CSVs are the current blank templates, not example files, completed templates, or earlier downloads. Upload these three reference files. Then upload the complete source material, including any appendices, schedules, amendments, or separate attachments needed to interpret the policy.
Copy the complete prompt in the light blue box below and paste it into the LLM chat.
In the pasted copy:
List only source-material filenames between the markers. Do not list the three reference files or blank CSV templates there. Do not send the prompt until all three replacements have been made.
Before moving on, check:
|
|
2 |
Confirm Classification Mapping and Metadata Back to top ↑ This is an important step. Classification types cannot easily be corrected in bulk after import. Get this right before the metadata CSV is built. Review the proposed prompting block from Step 1. Re-typing everything isn’t necessary. Correct only what’s wrong, then ask the LLM to restate the complete, corrected block to get one clean, final copy to carry into Step 3. (Step 1 results are a proposal shown in the chat – not a file, and nothing has been imported into AI Navigator yet.) Because the classification mapping is the part that is hardest to undo later, double-check each classification entry and its type-value pairing rather than assuming the first proposal is right. If making a few edits, use the short correction block below. If you have more and editing a full block is preferred over listing corrections, fill out and paste back the optional template prompt below the short correction block. In either path, do not continue to Step 3 until the model has restated the complete finalized block and all [CONFIRM] values have been resolved.
Optional template prompt:
️ The model will stop here and wait. This is intentional. Send any corrections, or if the values are right, continue to Step 3 by pasting the prompt there. Before moving on, check:
|
|
3 |
Build the Metadata CSV Back to top ↑ Once the values from Step 2 are confirmed, have the LLM produce the file – paste the prompt below. Then download it and verify it using the checklist that follows.
If the metadata CSV is unusable: If columns are shifted, values are changed, quoting is broken, or the file will not open correctly, use the confirmed metadata block to populate row 2 of Policy_Metadata_Template.csv manually. Save the file as CSV UTF-8 (Comma delimited). Do not alter the confirmed metadata values while rebuilding the file. After downloading, open the file and confirm:
|
|
4 |
Generate the Questions Back to top ↑ This is where the LLM adds the most value – drafting comprehensive assessment questions from the source material. For thorough coverage, generate questions iteratively by section rather than all at once. A note on long sources (read before starting Step 4)
A note on multi-category policies Some policies screen a system against several parallel categories. For example, "is it Generative AI? High risk? Requires transparency?" If the source works this way, keep a few things in mind:
|
|
Jump to a sub-step 4a. Outline the regulation's structure 4c. Run a gap check |
|
4a |
Outline the regulation's structure Back to top ↑ Ask the LLM to map out the regulation before writing any questions:
Re-check the outline before generating questions. The outline determines how question generation is organized. Step 4c then checks both the outline and the reattached source, so omissions in the outline can still be detected before finalization. Paste the prompt below as a separate turn. (A model re-reading its own work is a useful filter, not independent verification – it catches structural omissions better than judgment errors, so the checks below still apply.)
Before moving on, check:
|
|
4b |
Generate Questions Back to top ↑ Note: If the source material isn’t organized into chapters or articles, the Step 4a outline will produce logical topic areas instead. For the first section, start with the prompt below (it includes three worked examples to show the format). After reviewing the first section’s output — to confirm the format, levels, and coverage look right before they repeat across the rest — continue in one of the two ways described after the prompt. Large policy packs: If the question set may be too large to reproduce reliably from chat history, maintain an external 10-column working table as each section is generated. Preserve all fields and Question Numbers exactly, and attach the completed working table in Step 4d for consolidation and audit-file creation.
Continuing after the first section: Paste Prompt 4b once to generate questions for the first outline section classified as primary question-source. Review that section using the spot check below. Then choose Option 1 or Option 2 for the remaining primary question-source sections. Option 1, single pass: Generate questions for all remaining primary question-source sections in outline order. This is faster and works best when the source is shorter and the model is following the instructions consistently. Option 2, section by section: Repeat Prompt 4b for each remaining primary question-source section, changing only the requested section. This gives tighter control and is safer for dense, lengthy, or complex sources. Switch to Option 2 if quotes become shortened or paraphrased, later sections become thinner, requirements are compressed, topics drift, fields are omitted, numbering becomes inconsistent, or output is repeatedly interrupted. If switching, restart with the first section whose output was unreliable. Spot check the first section: Confirm that all required fields are present and correctly formatted. Verify that every item in the Key Requirements / Suggested Actions block has at least one corresponding question. Resolve any gap before continuing. Option 1 — Let the LLM continue in one pass Best for: Shorter sources and modern high-capability models Why it works: Capable models can hold long source material in context and generate comprehensive question sets in a single, cohesive pass – often with better cross-section consistency than working section by section. The prompt asks the model to chunk the material internally, though how reliably it does so varies by model and source. How to use it: After reviewing the first section from the outline, paste the prompt below. The model will continue from where it left off and generate questions for every remaining section.
What to expect: On shorter sources this usually produces a complete set in one pass. On longer ones, expect the response to be cut off by output limits (see below) and watch for the drift symptoms listed above – the further into a single pass, the more likely format and quoting discipline slips. Note on output limits: If the response is cut off before all sections are covered (usually due to per-turn output limits, not a compression problem), simply paste:
Option 2 — Work through sections one at a time Best for: Dense or lengthy sources, smaller or in-house models, packs where tight control is needed over each section – or any point where Option 1's output shows the drift symptoms listed above. Why use it: Section-by-section prompting gives smaller models a narrower focus per turn, which improves depth and reduces compression. The prompt below re-states the key rules each time rather than relying on the model to remember them across turns – this is what keeps format and quoting discipline consistent. It also enables spot-checking quality between sections rather than at the end. How to use it: Paste this prompt for each remaining section, filling in the section name and article range from the outline, and repeat until every section from the outline has been covered:
|
|
4c |
Run a gap check Back to top ↑ After generating questions for all sections – and after re-attaching the source so the model works from the full text, not a fading memory of it – run a gap check:
Step 4c.5 – Rationalize the draft question set before finalizing:
|
|
4d |
Review the complete set Back to top ↑ If the Source → Question mapping identifies one or more remaining gaps: The LLM will propose the questions or revisions needed to address them and wait for approval. Approve or correct each proposal, then instruct the LLM to continue. It will incorporate only the approved changes, recheck the revised set, and restart consolidation at Part 1 before producing the audit file.
Using and retaining the output: Download [PolicyName]_AUDIT.csv and carry it into Step 5. It retains the Legal Citations and Verbatim Source Quotes for included questions because SAS AI Navigator does not store those fields. The Source → Question mapping checks the complete source separately and is reconstructed during Step 5a. Save the Step 5a findings when the rationale for intentional exclusions must be retained. The import file is not produced until Step 5b, after review and approval. If the audit CSV is unusable: If columns are shifted, values are truncated, quoting is broken, or the file will not open correctly, create a new spreadsheet with the exact 10-column audit header from Step 4d. Copy the finalized audit table into that spreadsheet and save it as CSV UTF-8 (Comma delimited). Do not use the eight-column Policy_Questions_Template.csv to rebuild the audit file. Do not alter substantive question content while troubleshooting the file. |
|
5 |
Review, Validate, and Build the Import File Back to top ↑ This step produces the import file. Step 4 produced the audit table and audit CSV only; the import file is built here, after the final review. |
|
Jump to a sub-step |
|
5a |
Have the output reviewed Back to top ↑ The model’s self-review below is a useful first pass – but it is not independent verification. The same model that wrote a fabricated question is unreliable at catching it, and after a long session it may be reviewing from a fading copy of the source. The strongest check runs this review with the source freshly attached, and ideally in a new chat (or a second model) that sees only the source plus the question list, not the conversation that produced them. When re-attaching the source, rename the file first – models don’t always use the most recent version of a same-named upload. What to bring into this review. The reviewing model needs two things: the source material, and the question set with its Legal Citation and Verbatim Source Quote fields. It will reconstruct the Source → Question mapping itself – that is a stronger check than reading one supplied from an earlier step, because a mapping that omits a section looks internally consistent. The import CSV won't work here; it doesn't exist yet, and it has no citations to verify. For the question set, either form works:
If the CSV is attached and the model reports that columns are garbled or run together, paste the table instead. The prompt below asks it to confirm this before reviewing.
Then verify independently. The reviewing model’s statement that a quote is accurate is not enough. Search each question’s Verbatim Source Quote in the source using Ctrl-F. Treat any quote that cannot be found exactly as a review finding requiring resolution. Determine whether the question should be corrected, removed, or supported by a different complete source passage. Record the resulting correction as an approved change for Step 5b rather than editing the CSV directly. For a large pack, prioritize: - every quote the reviewing model flagged as questionable; - every quote under approximately 15 words; - every quote supporting a proposed addition or major rewrite; and - a random sample of the remaining quotes. If any sampled quote fails, verify every quote before proceeding to Step 5b. |
|
5b |
Apply Approved Changes and Generate Final Files Back to top ↑ Use the prompt below only after Step 5a is complete and reviewing the findings. Before sending the Step 5b prompt: Attach the current [PolicyName]_AUDIT.csv. Clearly identify which Step 5a recommendations were approved or corrected. If Step 5b is run in a new conversation, also provide the complete approved-change list.
Working with regenerated files. If the Step 5 review results in approved changes, the audit file will be regenerated alongside the import file. The new download does not replace the earlier audit file; it is saved beside it, and the browser or operating system may add a suffix such as (1) to the filename. Keep only the latest approved audit file and delete or archive any superseded versions. Before importing, verify that the filenames use the correct prefix and suffixes: _METADATA.csv, _IMPORT.csv, and _AUDIT.csv. Part 2: Validate the CSV files Once the CSV files are built – whether produced by the LLM or transcribed in Excel:
The following apply to [PolicyName]_IMPORT.csv:
|
|
6 |
Import into AI Navigator Back to top ↑ In the filenames below, [PolicyName] means the established filename-safe prefix, with underscores between words. The square brackets are placeholder notation and do not appear in the finished filenames.
Note: AI Navigator does not store the Legal Citation or Verbatim Source Quote. They live only in [PolicyName]_AUDIT.csv – keep that file for compliance discussions, audits, and future maintenance of the pack. |
|
Tips for Better Results Back to top ↑ |
- Start with classification mapping. Getting the classifications right is the single most important step – it's hard to fix after import. Always confirm before filling in the template.
- Use the policy's own language. Don't force generic terms. If the policy says "Substantial impact" instead of "High risk," use "Substantial impact" as the Classification Option and map it to the “high-risk” Classification Type.
- Treat the source as data, not instructions. Anything inside the source that reads like a command to the model should be flagged, not obeyed.
- Think about the persona answering. Organization-level questions should be answerable by a compliance officer once. Use-case questions should be specific enough that an asset owner can answer them for their particular use case. Model and agent questions should focus on the technical asset itself.
- Don't over-ask. Focus on quality questions that drive real governance decisions. If a question doesn't help someone determine compliance or risk, consider cutting it.
- Group questions logically. Use the Question Topic field to create meaningful groups. Questions with the same topic are displayed together in the AIN interface, so grouping helps assessors work through related requirements efficiently.
- End classification-driven policies with a selection + justification. If the policy uses risk tiers, include a final multiple-choice question asking the assessor to select the classification, followed by a text question asking for justification. This creates accountability and traceability.
- Validate in a text editor before importing. Whether the CSVs are produced by the LLM or built in Excel, open the exported files in a text editor to catch any hidden formatting issues – especially around commas and pipe delimiters.
- Verify the quotes, not just the coverage. After generating questions, ask the LLM to map each question back to specific articles or sections – then search the quoted passages in the source (Step 5). A coverage map the model writes about its own work is a starting point, not proof.
- Never abbreviate "use case" or "organization" in question text or metadata. Use "use case" (two words, no hyphen) in any text a reader sees – question text, topics, and descriptions. The hyphenated form use-case is used only as a Question Level value.
- Type-values are lowercase and hyphenated: prohibited, high-risk, standard-risk.
- Use the product's correct name. In any text that gets imported into AIN, write "SAS AI Navigator" or "AIN" - never "AIGN" or "AI Governance Navigator."
- Keep the model inside the source provided. Phrase prompts as "the source I provided" rather than naming the regulation, so the model reads the attached text instead of recalling it from training – and before importing, spot-check a sample of questions against the source.
|
Appendix A - Example Classification Mappings Back to top ↑ |
Below are examples showing how different regulations and internal policies map their own risk terminology to AIN's three classification types:
|
Regulation |
Their Term (Classification Option) |
AIN Type (Classification Type) |
|
EU AI Act |
Unacceptable risk |
prohibited |
|
EU AI Act |
High risk |
high-risk |
|
EU AI Act |
Limited risk |
standard-risk |
|
EU AI Act |
Minimal risk |
standard-risk |
|
Internal Policy |
Prohibited use |
prohibited |
|
Internal Policy |
Restricted use |
high-risk |
|
Internal Policy |
Approved use |
standard-risk |
Key rules:
- Classifications are optional — a policy can have zero classification options
- If a Classification Option is populated, its corresponding Classification Type must also be populated
- Multiple Classification Options can map to the same Classification Type (e.g., both Limited risk and Minimal risk → standard-risk)
- Enter classifications in descending order of severity: prohibited first, then high-risk, then standard-risk. When multiple source-defined options map to the same Classification Type, retain the source's descending severity order among those options.
|
Appendix B - Common Mistakes to Avoid Back to top ↑ |
|
Mistake |
Correct Approach |
|
use_case (underscore) in Question Level |
use-case (hyphen) |
|
Standard Risk (capitalization and missing hyphen) in Classification Type |
standard-risk (lowercase and hyphenated) |
|
Space in classification Option Label and type-value pair |
Low risk:standard-risk (no spaces around colon) |
|
External or external in Policy source |
EXTERNAL (all uppercase) |
|
Boolean question with 3 answer options |
Boolean must have exactly 2 options |
|
Multiple choice question with only 2 options |
Multiple choice needs at least 3 options |
|
Classification entry missing its type-value |
Every entry must be Option Label:type-value |
|
Classification Types in ascending severity order |
List classifications in descending severity: prohibited first → high-risk → standard-risk last. |
|
Spaces around pipes: Yes | No |
No spaces: Yes|No |
|
"If yes, describe..." conditional phrasing |
Reword to standalone: "Describe X or indicate N/A if not applicable." |
|
Using Complete as an assessment state |
The correct state is Submitted (lifecycle: Incomplete → Submitted → Approved) |
|
Applying policies directly to models or agents |
Policies are applied only to use cases; model/agent questions surface via the linked use case |
|
Pipe character (|) used within question text |
Never use the pipe character in question text; it breaks answer option parsing |
|
Marking conditional questions as optional so assessors can skip them |
Keep them required with an N/A option; optional cannot distinguish between ‘doesn’t apply’ from |
|
Leaving [PolicyName] in the finished filename or using inconsistent filename prefixes |
Replace [PolicyName] with one concise filename-safe prefix. Use underscores between words, preserve source-supported official identifiers, omit the square brackets, and reuse the exact same prefix for the metadata, import, and audit files. Normalize the filename only; preserve the confirmed Policy Name inside the metadata CSV. |
|
Making every question text because narrative evidence is generally helpful |
Use text when narrative information materially improves assessment of implementation, ownership, evidence, risk exposure, gaps, or mitigation. Use boolean for complete binary determinations and multiple-choice for meaningful alternatives, partial states, or N/A. |
|
Converting questions between answer types simply to create a more balanced distribution |
Select each Answer Type based on the source requirement, governance decision, and respondent burden. A policy pack may legitimately contain more of one type than another. |
|
Appendix C - Troubleshooting Import Errors Back to top ↑ |
If the AIN import wizard returns a validation error, check these common causes:
|
Error Type |
Likely Cause |
Fix |
|
Classification validation error |
An entry is missing its type-value, has spaces around the colon, or uses an invalid type-value |
Ensure each entry is formatted as Option Label:type-value |
|
Answer option error |
A boolean question has more or fewer than 2 options, or a multiple choice has fewer than 3 |
Check pipe-delimited options and answer type match |
|
Question Level error |
Used use_case (underscore) instead of use-case (hyphen) |
Replace all underscores with hyphens in the Question Level column |
|
Parsing/delimiter error |
A value containing a comma isn't wrapped in double quotes, or answer options have spaces around pipes |
Open the CSV in a text editor and fix the quoting/spacing |
|
Import Rejected/File not accepted |
Extra or misnamed columns (often leftover Legal Citation/Verbatim Source Quote) |
Use the 8-column _IMPORT file; AIN requires exact column match |
|
Filename displays %20 or is difficult to read |
The filename contains spaces or the full Policy Name was reproduced instead of using the established concise filename-safe prefix |
Replace spaces with underscores and reuse the same concise prefix for _METADATA.csv, _IMPORT.csv, and _AUDIT.csv. Do not change the Policy Name stored inside the metadata CSV. |
For additional support with unresolved errors, contact a SAS AI Navigator representative.
|
Appendix D - Worked Example – SR 26-2 (Model Risk Management) Back to top ↑ |
This appendix shows a complete run of the workflow on a real, public source: the interagency Supervisory Guidance on Model Risk Management (SR 26-2, issued by the Federal Reserve, FDIC, and OCC, April 17, 2026). It illustrates the expected output at each step – including the judgement calls the workflow is designed to surface.
Because it follows the workflow in order, the same field can look different at different stages. Governing Body is the clearest case: the model proposes [CONFIRM] at D.1, which gets resolved at D.2, and the resolved value is what appears in the file at D.3. That progression is the point of the example, not an inconsistency.
Fields marked with * are required. All others are optional and may be left blank.
D.1 — Metadata Proposal
Output of the Step 1 metadata prompt, prior to any review. Fields are shown as rows here for readability. In the actual CSV, they are formatted as 9 columns described above.
|
Field |
Proposed Value |
|
Policy Name* |
Supervisory Guidance on Model Risk Management (SR 26-2) |
|
Policy Description* |
Interagency supervisory guidance from the Federal Reserve, FDIC, and OCC setting forth sound principles for effective model risk management at banking organizations — covering model development and use, validation and monitoring, and governance and controls — using a risk-based approach tailored to an organization’s model risk profile and the size and complexity of its operations. |
|
Policy Source* |
EXTERNAL |
|
Geographies |
United States |
|
Classifications |
|
|
Tags |
model-risk|model-validation|model-governance|banking|supervisory-guidance |
|
Governing Body |
[CONFIRM] |
|
Policy Provider |
Board of Governors of the Federal Reserve System; Federal Deposit Insurance Corporation; Office of the Comptroller of the Currency |
|
Penalty For Non-Compliance |
Non-enforceable guidance — “non-compliance with this guidance will not result in supervisory criticism against a banking organization,” though “supervisory action may result for any violations of law or unsafe or unsound practices stemming from insufficient management of model risk.” |
Note: The penalty is taken from the source rather than invented — the guidance is explicitly non-enforceable.
The Three Judgement Calls: This proposal demonstrates all three states an optional field can take.
A proposed value - Penalty For Non-Compliance.
The source addresses this directly, so the field is populated. Note that the value records what the source says - that the guidance is non-enforceable - rather than inventing a penalty regime. "No penalty" is a finding, not a gap.
[CONFIRM] - Governing Body.
This field refers to the department or committee inside the relevant organization that is accountable for the policy. The source names the issuing agencies, but those belong in Policy Provider. Source documentation does not necessarily identify the internal governing body, so this field is flagged. It will almost always be [CONFIRM] at this stage.
BLANK - Classifications.
SR 26-2 defines no risk tiers, categories, or classification scheme of any kind. The field is inapplicable, so it is left empty. Do not enter filler values such as "none", "N/A", or "(none)" – literal text will import as a value.
D.2 — What gets resolved
[CONFIRM] behaves as a review flag, not a value that gets preserved. Every [CONFIRM] must be resolved by the user before the CSV is written – either replaced with a real value or purposely left blank. No [CONFIRM] should appear in a finished file.
In this example, there is one flag to resolve for the Governing Body value:
- Proposed by LLM: [CONFIRM]
- Resolved: Model Risk Committee
The resolved value goes into the Step 2 correction block. Everything else in D.1 is accepted as proposed.
D.3 — What gets written
Final confirmed metadata values are entered into the sheet, which is later saved as a CSV file to upload into AIN.
Enter values plainly, precisely as they should read. Do not add quotation marks around a whole value to mark it as a field; Excel does that automatically when saving any value containing a comma. In this example that applies to Policy Description and Penalty For Non-Compliance.
|
Field |
Final Value |
|
Policy Name* |
Supervisory Guidance on Model Risk Management (SR 26-2) |
|
Policy Description* |
Interagency supervisory guidance from the Federal Reserve, FDIC, and OCC setting forth sound principles for effective model risk management at banking organizations — covering model development and use, validation and monitoring, and governance and controls — using a risk-based approach tailored to an organization’s model risk profile and the size and complexity of its operations. |
|
Policy Source* |
EXTERNAL |
|
Geographies |
United States |
|
Classifications |
|
|
Tags |
model-risk|model-validation|model-governance|banking|supervisory-guidance |
|
Governing Body |
Model Risk Committee |
|
Policy Provider |
Board of Governors of the Federal Reserve System; Federal Deposit Insurance Corporation; Office of the Comptroller of the Currency |
|
Penalty For Non-Compliance |
Non-enforceable guidance – “non-compliance with this guidance will not result in supervisory criticism against a banking organization,” though “supervisory action may result for any violations of law or unsafe or unsound practices stemming from insufficient management of model risk.” |
D.4 – Classifications Syntax
SR 26-2 defines no classification scheme, so the run above leaves that field blank. The example below shows the syntax for a framework that does define tiers.
Example source with classifications: a regulation defining four risk tiers.
Classifications value:
Unacceptable risk:prohibited|High risk:high-risk|Limited risk:standard-risk|Minimal risk:standard-risk
What this example demonstrates:
- Option Label comes from the regulation. "Unacceptable risk" and "Limited risk" are the regulation's own terms. Do not replace with generic labels.
- The type-value comes from AIN's fixed vocabulary. Only three values are valid: prohibited, high-risk, standard-risk. They must be lowercase and hyphenated.
- No spaces around the colon. Write High risk:high-risk, not High risk : high-risk.
- Descending severity order: prohibited first, then high-risk, then standard-risk. Many-to-one mapping is allowed. Both "Limited risk" and "Minimal risk" map to standard-risk. A regulation with more tiers than AIN has types is normal and expected.
D.5 — Structure outline (Step 4a)
• §I Introduction — framing; non-enforceable. No assessment questions.
• §II Purpose & Scope — $30B applicability; definition of “model”; generative/agentic AI excluded. → organization, use case.
• §III Overview of Model Risk — inherent risk, exposure, purpose, materiality; effective challenge; aggregate risk. → use case, organization.
• §IV Model Development & Use — development aligned to purpose; testing; use beyond intended purpose. → model, use case.
• §V Validation & Monitoring — validation before first use; conceptual soundness; outcomes analysis; ongoing monitoring. → model.
• §VI Governance & Controls — policies; roles & responsibilities; model inventory; documentation. → organization.
• §VII Vendor & Third-Party Products — validation, monitoring, and documented customization of vendor models. → model, use case.
• Agent level — out of scope (the source excludes generative and agentic AI), so no agent-level questions are generated.
D.6 — Final Assessment Questions After Step 5 Review
The questions were initially generated in Steps 4b-4d and finalized after the Step 5 review. 24 questions across organization, use case, and model levels. No agent-level questions — the source places generative and agentic AI out of scope, so the agent minimum is recorded as not applicable rather than padded. Conditional questions use Yes|No|N/A, and three weak attestations were changed to text (marked in bold) because their original yes/no format could conceal meaningful gaps in implementation, ownership, or evidence.
Note: Conditional questions use a N/A option rather than being marked optional, so completeness is preserved while still allowing 'does not apply'
Note: Required and Version are omitted from the table below only to save width. Both are uniform in this example: every question is required, Version 1.0.
|
# |
Question Text |
Type |
Options |
Topic |
Level |
|
1 |
Does the organization apply model risk management consistent with this guidance, recognizing it is most relevant to banking organizations with over $30 billion in total assets (and may also apply to smaller organizations with significant model risk exposure)? |
boolean |
Yes|No |
Applicability & Scope |
organization |
|
2 |
Does this use case rely on one or more models as defined in this guidance — a complex quantitative method, system, or approach that applies statistical, economic, or financial theories to process input data into quantitative estimates? |
boolean |
Yes|No |
Applicability & Scope |
use-case |
|
3 |
Describe how model materiality was determined for the models supporting this use case, based on model exposure and model purpose. |
text |
(blank) |
Materiality Assessment |
use-case |
|
4 |
Does the organization assess model risk both individually and in aggregate, accounting for interactions, dependencies, and shared assumptions, data, or methodologies across models? |
boolean |
Yes|No |
Aggregate Risk |
organization |
|
5 |
Describe who performs effective challenge for these models and how their expertise, independence, and standing to effect change are ensured. |
text |
(blank) |
Effective Challenge |
use-case |
|
6 |
Does model development begin with a clear statement of purpose and stay aligned with the model’s intended use, business use, and banking organization policy? |
boolean |
Yes|No |
Model Development |
model |
|
7 |
Has the model been tested to evaluate whether it performs as intended, with testing rigor commensurate with the model’s complexity and materiality? |
boolean |
Yes|No |
Model Testing |
model |
|
8 |
Has the organization assessed whether any model supporting this use case is being used beyond its intended purpose, and reviewed existing controls to manage the resulting risk? |
boolean |
Yes|No |
Model Use |
use-case |
|
9 |
Has the model been validated to evaluate whether it performs as expected — including its reliability and limitations — generally prior to first use? |
boolean |
Yes|No |
Model Validation |
model |
|
10 |
Does validation assess the model’s conceptual soundness — its design, key modeling choices, assumptions, qualitative judgments, data selection, construction, and developmental testing? |
boolean |
Yes|No |
Conceptual Soundness |
model |
|
11 |
Describe the outcomes analysis performed for this model, comparing model outputs to corresponding real-world outcomes against established performance thresholds. |
text |
(blank) |
Outcomes Analysis |
model |
|
12 |
Is there an ongoing monitoring plan evaluating whether the model continues to perform as expected given changes in products, exposures, activities, clients, data relevance, or market conditions? |
boolean |
Yes|No |
Ongoing Monitoring |
model |
|
13 |
Identify the organization’s model risk management policy and describe how it defines risk management expectations and the framework for applying practices commensurate with model risk. |
text |
(blank) |
Governance & Policy |
organization |
|
14 |
Describe how roles, responsibilities, and accountability are defined across the model lifecycle, including how conflicts of interest between model development and validation groups are managed. |
text |
(blank) |
Roles & Responsibilities |
organization |
|
15 |
Describe the information captured in the model inventory and how it supports understanding model risk at the individual and aggregate levels. |
text |
(blank) |
Model Inventory |
organization |
|
16 |
Is model risk management supported by adequate documentation, including the tracking of recommendations, responses, and exceptions? |
boolean |
Yes|No |
Documentation |
organization |
|
17 |
For each vendor or third-party model, has the organization validated the product and developed an understanding of its conceptual soundness, design, development data, and performance? |
boolean |
Yes|No |
Vendor & Third-Party Models |
model |
|
18 |
Is ongoing monitoring and outcomes analysis conducted on vendor models to assess whether they are accurate, remain fit for purpose, and continue to be reliable? |
boolean |
Yes|No |
Vendor & Third-Party Models |
model |
|
19 |
Where vendor models are customized for this use case’s specific business needs, are the adjustments appropriately documented, justified, and evaluated as part of model validation? |
multiple-choice |
Yes|No|N/A |
Vendor & Third-Party Models |
use-case |
|
20 |
When a model must be used before validation is completed (e.g., an urgent business need), are its limitations communicated to relevant stakeholders and appropriate controls determined, such as limits on use or closer monitoring? |
multiple-choice |
Yes|No|N/A |
Model Validation |
model |
|
21 |
Where internal audit is part of model risk management, does it evaluate whether the practices are rigorous and effective and whether related policies are implemented, rather than duplicating model development or validation? |
multiple-choice |
Yes|No|N/A |
Roles & Responsibilities |
organization |
|
22 |
For external resources used to help manage model risk, does the organization maintain proper oversight and integrate that work into its broader model risk management activities? |
boolean |
Yes|No |
Governance & Policy |
organization |
|
23 |
For models supporting this use case that are deemed immaterial, are they identified and their performance monitored for conditions under which their use may become material? |
multiple-choice |
Yes|No|N/A |
Materiality Assessment |
use-case |
|
24 |
Describe how input from model users and business managers is incorporated into model development, including how questions about methods or assumptions are addressed. |
text |
(blank) |
Model Development |
model |
Distribution: organization 8 · use case 6 · model 10 · agent 0 (out of scope) · 24 total. Answer types: 13 boolean · 7 text · 4 multiple-choice (Yes|No|N/A). Every question is required, Version 1.0. The questions in bold were strengthened from a boolean attestation to evidence-seeking text during the Step 5 review.
Answer Types are selected requirement by requirement, not to meet a quota. Boolean is used where the complete requirement supports a meaningful yes/no determination. Multiple-choice is used where an N/A path or another material state is needed. Text is used where the assessor must describe implementation, ownership, evidence, rationale, or operation to reveal the underlying risk. The mix in this example is illustrative, not a required distribution for other policy packs.
D.7 — Traceability (sample)
Every question in this pack maps to a specific passage in SR 26-2 via a Legal Citation and Verbatim Source Quote — the annotations that make the pack auditable. Three questions are shown below with the full annotations that would accompany them in the LLM's Step 4b output. In an actual workflow, every question in the pack carries these two review-only fields; only three are shown here for length.
|
# |
Verbatim source passage + locator |
|
6 |
Question 6 — Does model development begin with a clear statement of purpose and stay aligned with the model's intended use, business use, and banking organization policy? |
|
11 |
Question 11 — Describe the outcomes analysis performed for this model, comparing model outputs to corresponding real-world outcomes against established performance thresholds. |
|
19 |
Question 19 — Where vendor models are customized for this use case's specific business needs, are the adjustments appropriately documented, justified, and evaluated as part of model validation? |
The remaining questions in this pack follow the same annotation pattern. Each Legal Citation identifies the smallest meaningful subdivision of SR 26-2 (§II, §III, §IV, etc.), and each Verbatim Source Quote is the exact passage from that section supporting the question. These annotations are review artifacts — they help the reviewer confirm each question is grounded in source text — and are not imported into AI Navigator.
SAS® AI Navigator — Policy Prompting Guide — September 2026
Call for content now open!
It's your turn to help shape SAS Innovate 2027. Share your expertise and inspire the SAS community.
