Implement generative AI and agentic solutions 2.1 Build generative applications by using Foundry
A coding assistant generates Python and SQL snippets.
Study guide skill: Evaluate models and apps, including detecting fabrications, relevance, quality, and safety
Question B8-017
A coding assistant generates Python and SQL snippets. Before release, you want a built-in evaluator that flags generated code with security problems such as SQL injection or code injection. Which evaluator should you use?
Select one answer.
Answer: B
Code vulnerability measures whether AI-generated code has security vulnerabilities such as code injection and SQL injection across languages including Python and SQL. Protected materials detects copyrighted text, and Ungrounded attributes detects ungrounded inferences about personal attributes. Indirect Attack measures whether a response fell for an indirect jailbreak injected into context.
Deep dive
How it works
Among Foundry's risk and safety evaluators, Code vulnerability is the one for generated code. It measures whether AI generates code with security vulnerabilities, such as code injection, tar-slip, SQL injections and stack trace exposure, across Python, Java, C++, C#, Go, JavaScript and SQL. It applies to both models and agents. The evaluator outputs a fail if any of its listed vulnerability subclasses is present, and a pass if none are detected.
For a coding assistant that writes Python and SQL, it covers both languages and both of the named problems.
How the distractors differ:
- Indirect Attack (XPIA), model only: measures to what extent a response fell for an indirect jailbreak injected into the context of a document or source.
- Ungrounded attributes: text with ungrounded inferences about people's personal attributes.
- Protected materials: text that's under copyright, such as song lyrics, recipes and articles.
The other options, and when you'd use them
- Indirect Attack (XPIA): For responses that followed injected instructions from a document, not insecure code.
- Ungrounded attributes: For ungrounded inferences about demographics or emotional state.
- Protected materials: For copyrighted text.
Exam tip
Insecure generated code (SQL injection, code injection) = Code vulnerability. Fell for injected instructions = Indirect Attack.
References
Last reviewed: