AGP Picks
View all

AIM Intelligence benchmark exposes AI safety gaps across 10 countries

5 hours ago
AIM Intelligence benchmark exposes AI safety gaps across 10 countries
By AI, Created 16:00 UTC, Nov 19, 2026, AGP -

AIM Intelligence says its XL-SafetyBench, now accepted to the NeurIPS 2026 Evaluations & Datasets Track, found that frontier LLMs often miss local risks and cultural taboos outside English settings. The benchmark spans 10 countries and suggests AI safety checks need country-specific standards, not one global score.

Why it matters: - AI safety tests built around English can miss country-specific fraud schemes, laws and cultural taboos. - AIM Intelligence says XL-SafetyBench shows that resistance to jailbreaks and cultural sensitivity are separate capabilities that should be measured separately. - The benchmark is designed to help companies and institutions verify AI systems against local standards before deployment.

What happened: - AIM Intelligence announced that its paper, "XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity," was accepted to the NeurIPS 2026 Evaluations & Datasets Track. - The benchmark evaluates large language models across 10 countries: the United States, France, Germany, Spain, South Korea, Japan, India, Indonesia, Türkiye and the UAE. - The project used 5,500 test cases written in local languages and grounded in each country’s laws, institutions and culture. - Seventeen researchers from 10 institutions contributed, including Microsoft, the Korea AI Safety Institute, KT, Coinbase and the Technical University of Munich.

The details: - The benchmark splits evaluation into two parts. - The jailbreak benchmark includes 4,500 attack prompts tied to local laws, institutions and fraud schemes. - One prompt tests whether a model can be manipulated with a fake official letter about fraud schemes involving jeonse, Korea’s lump-sum deposit lease system. - The cultural benchmark includes 1,000 scenarios that hide a local taboo inside an ordinary request. - One example checks whether a model recommends chrysanthemums as a thank-you gift in France, where the flower is associated with death and mourning. - Two native-speaker experts reviewed every test case for each country, for 20 reviewers in total. - The research team tested 37 LLMs, including 10 global frontier models and 27 locally developed models. - Across the 10 global frontier models, localized attack scenarios produced harmful responses 44.8% of the time on average. - The paper says that figure reflects benchmark scenarios that were chosen because they had succeeded against reference models, not a real-world breach rate. - Models performed best in the United States, where the attack success rate was 34.5%. - Attack success rates were higher in South Korea at 52.8% and the UAE at 57.0%. - When cultural taboos were stated directly, models recognized them 83.2% of the time on average. - When the taboo was hidden in an everyday request, recognition fell to 49.9%. - All 10 models declined the hidden-taboo scenarios, but recognition dropped below 40% for India and Türkiye. - Most locally developed models recognized hidden taboos less than 15% of the time, and three scored 0%. - Among those models, lower attack success rates tracked with more irrelevant or incoherent answers, with a correlation coefficient of -0.81. - The paper describes that pattern as "comprehension-masked safety," meaning a low attack score may reflect failure to understand the request rather than a proper refusal.

Between the lines: - The findings suggest that current safety leaderboards can overstate how well models handle real-world use in non-English settings. - The benchmark also argues that a model can look safer on refusal metrics while still failing to understand local context. - Choi said the team wants evaluation standards that match each country’s laws, institutions and culture so organizations can verify AI against local standards.

What's next: - The paper is available on arXiv. - The dataset is on Hugging Face. - The code is on GitHub. - AIM Intelligence expects XL-SafetyBench to be used as a reference for companies and institutions deploying global services or country-specific AI. - NeurIPS 2026 will be held in December in Sydney, Atlanta and Paris, where accepted Evaluations & Datasets Track papers will be presented.

The bottom line: - AIM Intelligence’s benchmark says AI safety cannot be judged well with English-only testing, because local risks and local culture can change both harmful behavior and refusal behavior.

Disclaimer: This article was produced by AGP Wire with the assistance of artificial intelligence based on original source content and has been refined to improve clarity, structure, and readability. This content is provided on an “as is” basis. While care has been taken in its preparation, it may contain inaccuracies or omissions, and readers should consult the original source and independently verify key information where appropriate. This content is for informational purposes only and does not constitute legal, financial, investment, or other professional advice.

Sign up for:

Georgia Culture Digest

The daily local news briefing you can trust. Every day. Subscribe now.

By signing up, you agree to our Terms & Conditions.

Share this page:

Advanced Search Options

Search for:

Search scope:

Type:

Search in:

Date range:

The last

Sort by:

Sign up for:

Georgia Culture Digest

The daily local news briefing you can trust. Every day. Subscribe now.

By signing up, you agree to our Terms & Conditions.