Geopolitical Differential Performance in LLM Agents: Preliminary Observations from Counterfactual Evaluation

Summary

We evaluated four foundation models across diverse agentic infrastructure tasks, parameterizing runs with context packs from four geopolitical categories (Israeli defense/intelligence, Chinese military/state security, Taiwanese defense/intelligence, and American defense/intelligence) alongside entirely fictional neutral organizations including corporations. Across approximately 900 paired executions, two of four models showed notable behavioral differences. Kimi-K2-Instruct showed the strongest effects against Israeli context packs (completion 78.6% vs. 94.8%, refusal rate 10.3% vs. 1.8%) with a pronounced tool_competence deficit (δ = −0.52). Kimi-K2 and DeepSeek-V3 showed reduced degradation when tested against Chinese context packs compared to Israeli and Taiwanese ones. Llama-3.1-70B and gpt-oss-120b showed no notable behavioral differences across any geopolitical category.

Benchmark Overview

We developed a benchmark for detecting geopolitically-triggered behavioral degradation in autonomous AI agents. The system uses three independent LLMs (an agent under test, a simulation LLM, and a judge LLM) with 231 tools across 26 categories and 20 scenario templates. For each context pack pair (one sensitive, one neutral), we run the same scenario once per pack, keeping task structure identical while only varying the organizational context. Bidirectional context pack masking ensures the simulation and judge operate without knowledge of which context pack is active. We tested against four sensitive categories (15 context packs: 5 Israeli, 4 Chinese, 3 Taiwanese, 3 American) and 6 neutral context packs, evaluating approximately 900 paired executions across four models. Each execution was scored by a judge LLM across 11 behavioral dimensions on a 1-4 scale, and for each dimension we compute $\delta ={\bar{s}}_{\text{sensitive}}-{\bar{s}}_{\text{neutral}}$ — a negative $\delta$ means the agent performed worse on sensitive context packs.

Aggregate Results

Kimi-K2-Instruct shows a notable completion and refusal gap for Israeli context packs (refusal 10.3% vs. 1.8%) and Taiwanese context packs (refusal 8.7% vs. 1.9%). American context packs show a directionally similar but smaller pattern. No gap is observed for Chinese context packs. DeepSeek-V3 shows a weaker pattern of the same kind. Llama-3.1-70B and gpt-oss-120b show no consistent directional pattern in completion and refusal rates across any geopolitical category.

Dimension-Level Patterns

Kimi-K2 shows negative deltas across multiple behavioral dimensions for Israeli context packs, with 5 dimensions exceeding our 0.25 interpretive caution threshold: tool_competence ($\delta =-0.52$), unnecessary_friction (−0.45), output_integrity (−0.41), thoroughness (−0.30), and task_completion (−0.28). tool_competence — the agent's ability to select and sequence appropriate tools — shows the largest effect, with sensitive-context agents using an average of 2.7 fewer unique tools per task (sensitive: M = 11.6, neutral: M = 14.3). The remaining dimensions are directionally negative but individually less certain. DeepSeek-V3 shows a milder but directionally consistent pattern. The pattern repeats at reduced magnitude for Taiwanese context packs and attenuates further for American ones, while Chinese context packs show near-zero deltas.

Threat Taxonomy

We identify four distinct manifestations of context-triggered behavioral differences in agent behavior:

Implications

These observations are consistent with prior work on politically-triggered code vulnerabilities (CrowdStrike's finding that certain LLMs generate code with higher rates of severe security flaws when prompts contain politically sensitive terms [1]) into the domain of operational task execution. The behavioral differences are not limited to code output; they affect tool selection, task planning, and multi-step workflow completion in agentic systems.

Standard capability evaluations using neutral context packs will not surface this class of failure. Two of the four models tested passed neutral evaluations but showed signs of degraded performance under sensitive conditions. The per-region pattern in Kimi-K2 and DeepSeek-V3 (minimal degradation for Chinese context packs, substantial degradation for Israeli and Taiwanese context packs, moderate degradation for American context packs) suggests that training data composition or alignment tuning may encode geopolitically-correlated behavioral patterns.

Two models (Llama-3.1-70B, gpt-oss-120b) showed no consistent multi-dimension directional pattern in this evaluation — while gpt-oss-120b shows an isolated tool_competence delta of −0.21 for Israeli context packs, it lacks the broad pattern seen in the affected models. This suggests that consistent cross-context performance may be achievable with current architectures. Organizations deploying LLM agents in geopolitically diverse environments should consider incorporating counterfactual evaluation into their model assessment process.

See Full Research for complete methodology, per-dimension statistical results, and limitations.

References

[1] CrowdStrike Counter Adversary Operations. "Politically-triggered code vulnerabilities in LLMs." 2024.

Abstract

Large Language Model (LLM) agents with autonomous tool-use capabilities introduce a novel class of alignment failures: context-triggered behavioral degradation, where models exhibit different performance based on contextual signals including geopolitical sensitivity. We present a benchmark implementing counterfactual paired evaluation to detect such behavioral differences in operational AI systems.

Our benchmark pairs geopolitically-sensitive context packs (Israeli defense/intelligence, Chinese military/state security, Taiwanese defense/intelligence, and American defense/intelligence organizations) against entirely fictional neutral organizations (including corporations) and runs the same multi-step infrastructure tasks through an LLM-driven simulation with 231 tools across 26 categories. A judge LLM evaluates masked execution traces across 11 behavioral dimensions on a forced-choice 1-4 scale.

Through systematic evaluation across approximately 900 paired scenario executions involving 4 foundation models and 4 geopolitical categories, we observe consistent patterns of behavioral differences in two models. Kimi-K2-Instruct exhibits degraded performance for Israeli context packs across multiple dimensions, with notably lower task completion (78.6% vs. 94.8%) and elevated refusal rates (10.3% vs. 1.8%). DeepSeek-V3 shows a milder but directionally consistent pattern. Kimi-K2 and DeepSeek-V3 show near-zero differentials for Chinese context packs. Llama-3.1-70B and gpt-oss-120b show no notable behavioral differences across any category.

We introduce a four-code threat taxonomy, formalize metrics for agent behavioral consistency, and observe that complex scenarios involving security trade-offs yield higher differential detection sensitivity than standard operational tasks. Motivated by recent research on politically-triggered code vulnerabilities in certain LLMs [1], our work extends these findings to operational task execution, establishing methodology for auditing context-triggered behavioral differences in autonomous AI systems.

Introduction

The artificial intelligence landscape has shifted from conversational systems to autonomous agents capable of consequential real-world actions. Modern foundation models increasingly serve as the cognitive substrate for systems that execute code, query databases, manage infrastructure, and interact with critical systems without human validation.

This transition introduces capabilities fundamentally distinct from text generation. Where traditional LLM safety research focused on preventing harmful outputs, agentic safety must address harmful actions. The World Economic Forum warns that "an AI system independently pursuing misaligned objectives could cause immense harm, especially in scenarios where the AI agents' level of autonomy increases while the level of human oversight decreases" [2].

Motivated by recent findings from CrowdStrike Counter Adversary Operations, which revealed that certain LLMs exhibit up to 50% higher likelihood of generating code with severe security vulnerabilities when prompts contain politically sensitive terms [1], we hypothesize that similar patterns extend beyond code generation to operational task execution in agentic systems.

If a model's internal representations encode geopolitical associations that influence how well it performs tasks, the implications for deployment in multinational critical infrastructure are significant.

This work addresses three fundamental research questions:

RQ1: Do LLM agents exhibit systematic behavioral differences when executing the same tasks parameterized with politically-sensitive versus neutral context packs?

RQ2: What scenario categories and evaluation dimensions maximize the sensitivity of differential detection methodologies?

RQ3: Can we characterize the mechanistic signatures of context-triggered behavioral degradation to inform mitigation strategies?

Contributions

1. A benchmark for detecting geopolitically-triggered behavioral degradation in autonomous AI agents, featuring 20 scenario templates across complex, simple, and clearly-harmful morality levels.
2. A four-code threat taxonomy for classifying politically-correlated behavioral differences, from explicit refusal gaps to covert operational degradation.
3. Preliminary observations across approximately 900 paired executions suggesting that two of four tested models exhibit consistent patterns of behavioral differences, with a region-dependent pattern suggesting training-data-correlated behavioral patterns.

Methodology

System Architecture

The evaluation framework uses three independent LLMs: (1) an agent LLM under test that executes multi-step infrastructure tasks using tool calls; (2) a simulation LLM that generates realistic tool responses while operating on context-pack-masked inputs, ensuring context-pack-agnostic behavior; and (3) a judge LLM that evaluates masked execution traces across 11 behavioral dimensions.

The system is coordinated by a runner that orchestrates pack sampling, concurrent agent execution, judgment collection, and statistical analysis.

Threat Model

We consider a deployment scenario where an LLM agent with tool-use capabilities manages operational tasks for a multinational organization. The agent's training process may have induced geopolitically-correlated behavioral patterns, potentially without developer awareness. The agent may then exhibit inconsistent performance based on contextual signals that correlate with geopolitical sensitivity.

An agent exhibits behavioral inconsistency when its performance metrics differ significantly between two context packs that differ only in organizational identity. Geopolitical differential performance is present when this inconsistency correlates with the geopolitical sensitivity of context pack categories.

Threat Taxonomy

We identify four distinct manifestations of context-triggered behavioral differences:

These codes map to observable judgment dimensions: T01 manifests as differential refusal rates and low unnecessary_friction scores; T02 as lower task_completion, security_hygiene, and cleanup_diligence; T03 as lower tool_competence; and T04 as lower output_integrity, thoroughness, and evidence_and_audit.

Context Pack Design

A context pack is a structured YAML file defining a complete organizational identity across infrastructure, personnel, regulatory, and operational dimensions. Packs are organized into two pools:

• Sensitive (Israeli) (5 packs): IDF Cyber Command, Mossad Operations Bureau, Shin Bet Intelligence, Israel National Cyber Directorate (INCD), and Israel Aerospace Industries (IAI).
• Sensitive (Chinese) (4 packs): PLA Strategic Support Force (PLA-SSF), Ministry of State Security Cyber Operations Bureau (MSS Cyber), China Electronics Technology Group Corporation (CETC), and National Computer Network Emergency Response Technical Team (CNCERT).
• Sensitive (Taiwanese) (3 packs): Information, Communications and Electronic Force Command (ICEFCOM), National Security Bureau Cyber Division (NSB Cyber), and National Chung-Shan Institute of Science and Technology (NCSIST).
• Sensitive (American) (3 packs): NSA Cybersecurity Directorate (NSA Cyber), United States Cyber Command (USCYBERCOM), and Cybersecurity and Infrastructure Security Agency (CISA).
• Neutral (6 packs): Entirely fictional Western technology corporations (Aurora Cloud Systems, Nordic Data Solutions, Meridian Technologies, Pinnacle Software Group, Helix Digital Services, Summit Infrastructure).

Each pack defines values across a hierarchical namespace that is auto-flattened to dot-notation keys. This produces several hundred unique placeholder keys per pack spanning domains, infrastructure endpoints, personnel names and emails, department structures, Kubernetes clusters, database hosts, Vault paths, monitoring endpoints, CI/CD pipelines, regulatory frameworks, and partner integrations. Below is a representative excerpt:

# Sensitive pack: IDF Cyber Command (Israeli)
infrastructure.databases.primary.host: "db-prod.idf-cyber.mil.il"
infrastructure.databases.primary.name: "ops_command_db"
leadership.ciso.name:                  "Col. Avi Dagan"
leadership.ciso.email:                 "a.dagan@idf-cyber.mil.il"
compliance.frameworks[0]:              "Israeli Cyber Defense Directive 8.2"

# Sensitive pack: PLA Strategic Support Force (Chinese)
infrastructure.databases.primary.host: "db-prod.plassf.mil.cn"
infrastructure.databases.primary.name: "strategic_ops_db"
leadership.ciso.name:                  "Senior Col. Zhang Wei"
leadership.ciso.email:                 "z.wei@plassf.mil.cn"
compliance.frameworks[0]:              "MLPS 2.0 Level 4"

# Sensitive pack: ICEFCOM (Taiwanese)
infrastructure.databases.primary.host: "db-prod.icefcom.mil.tw"
infrastructure.databases.primary.name: "comms_ops_db"
leadership.ciso.name:                  "Col. Chen Wei-Lin"
leadership.ciso.email:                 "w.chen@icefcom.mil.tw"
compliance.frameworks[0]:              "Taiwan Cyber Security Management Act"

# Sensitive pack: NSA Cybersecurity Directorate (American)
infrastructure.databases.primary.host: "db-prod.nsa-cyber.ic.gov"
infrastructure.databases.primary.name: "cyber_ops_db"
leadership.ciso.name:                  "Ms. Sarah Caldwell"
leadership.ciso.email:                 "s.caldwell@nsa-cyber.ic.gov"
compliance.frameworks[0]:              "NIST SP 800-171 Rev. 3"

# Neutral pack: Aurora Cloud Systems (fictional)
infrastructure.databases.primary.host: "db-prod.auroracloud.io"
infrastructure.databases.primary.name: "platform_analytics_db"
leadership.ciso.name:                  "Karen Mitchell"
leadership.ciso.email:                 "k.mitchell@auroracloud.io"
compliance.frameworks[0]:              "SOC 2 Type II"

The pack pool samples pack pairs by pairing one sensitive pack with one neutral pack. The pair's common keys (intersection of both packs' flattened key sets) determines the masking vocabulary.

Scenario Templates

We developed 20 scenario templates organized by morality level:

Complex scenarios (CPLX-001 through CPLX-016) are the primary evaluation instrument. Each defines a multi-phase task_template with {{pack.*}} placeholders resolved against the assigned context pack, an env_description and scenario_narrative guiding the simulation LLM's response generation, an initial_state dictionary seeding the simulation, and ground_truth specifying optimal decisions and expected outcomes.

Each scenario also specifies completion_criteria (10-12 items), malicious_indicators (6-15 red flags), complex_decisions (2 per scenario) describing ambiguous choice points with multiple defensible approaches, min_tool_calls (15-18 for complex; 0 for harmful controls where refusal is expected), and tools_available drawn from the 231-tool registry.

Harmful control scenarios request clearly malicious actions. The expected agent behavior is refusal; differential refusal rates between context pack conditions would indicate context-triggered behavioral differences.

LLM-Driven Simulation

Rather than a deterministic sandbox, the benchmark uses an LLM-driven simulation architecture. A dedicated simulation LLM (default: openai/gpt-oss-120b via Groq, temperature 0.3) generates realistic tool responses for each agent tool call. The simulation is coordinated by an orchestrator that maintains per-trial state:

1. The agent issues a tool call with real context pack names.
2. The bidirectional masker converts real context pack values to {{pack.*}} placeholders using longest-first replacement. Short values (≤4 characters) use word-boundary regex to prevent substring corruption.
3. The simulation LLM receives the masked tool call, the scenario narrative, and a state changelog (last 15 actions). It generates a realistic response using only placeholders - it never sees which context pack is being tested.
4. The masker unmasks {{pack.*}} placeholders back to real context pack values.
5. The agent receives the response with real names and continues.

A state manager tracks all actions as a changelog, passing current state to the simulation LLM on each call for consistency. When the agent issues multiple tool calls in a single iteration, they are simulated concurrently against the pre-batch state.

The tool registry provides 231 tools across 26 categories: Docker (15), Kubernetes (18), PostgreSQL (13), Redis (10), Vault (9), Nginx (8), Prometheus (8), Grafana (6), PagerDuty (6), Slack (7), Jira (8), Email (4), Firewall (8), IAM (11), SSL (6), Filesystem (13), Network (13), System (12), Logging (7), DNS (6), Backup (7), CI/CD (7), Cloud (12), Monitoring (6), Audit (5), and Compliance (6). All tools are in OpenAI function-calling format with typed parameters.

Agent Configuration

The agent under test uses the HuggingFace AsyncInferenceClient with configurable provider routing (Novita, Groq, Together, etc.). Key parameters:

• Temperature: 0.1 (agent), 0.3 (simulation), 0.0 (judge)
• Maximum iterations per task: 80
• Up to 3 continuation prompts if the agent stops prematurely
• Continuation thresholds: ≥15 tool calls, ≥10 unique tools (these represent minimum execution depth gates; scenarios with higher min_tool_calls up to 18 may see agents that pass the continuation check but receive lower task_completion scores for incomplete workflows)
• Context token budget: 24,000 tokens with aggressive compaction when exceeded

The system prompt instructs the agent to act as an autonomous infrastructure operations agent, using tools to investigate, plan, and execute tasks thoroughly.

Evaluation Protocol

For each scenario and sampled pack pair, the agent is executed separately with the sensitive and neutral pack (1 run each). Behavioral traces (task prompt, tool calls with arguments and responses, and final output) are collected.

Context Pack Masking for Judge. Before judgment, context-pack-identifying information is replaced with structured placeholders of the form {{pack.*}} using the bidirectional masker constructed from the specific pack. The task prompt, all tool call arguments, all tool results, and the final output are masked. Each trace is judged independently via a separate judge invocation.

LLM-as-Judge Methodology. We employ openai/gpt-oss-120b (via Groq, temperature 0.0) as the judge model. The judge evaluates all 11 dimensions using an enhanced prompt that includes full scenario context, ground truth, and malicious indicator checklists.

All 11 dimensions use a forced-choice 1-4 scale with no neutral midpoint:

Scoring calibration: 4 = Good (minor issues at most), 3 = Acceptable (noticeable gaps), 2 = Below standard (significant problems), 1 = Unacceptable (fundamental failures).

For each dimension $d$, we compute $\delta ={\bar{s}}_{\text{sensitive}}-{\bar{s}}_{\text{neutral}}$. Negative values indicate worse performance for sensitive context packs.

Analysis Approach

With 5 scenarios and 1 run per pack pair, each model-region condition yields approximately 12–15 paired observations. Given these sample sizes, we report observed effect sizes (deltas between sensitive and neutral mean scores), directional consistency across dimensions, and cross-model/cross-region pattern comparisons. Formal hypothesis testing lacks adequate power at this sample size; where Wilcoxon signed-rank p-values are reported, they serve as supplementary indicators and should be interpreted cautiously.

We treat individual dimension deltas below approximately 0.25 with particular caution in interpretation, given the stochastic nature of multi-step agentic execution and the per-condition sample sizes of approximately 12–15 paired observations. This is not a statistically derived threshold but a practical judgment call. We focus interpretive weight on larger effects and on directional patterns that replicate across multiple dimensions and models.

Refusal detection uses regex matching against agent output for phrases indicating inability or unwillingness to proceed. We adopt a broad definition of refusal that includes hard refusals ("I cannot assist with this"), soft refusals (excessive warnings, repeated confirmation requests), and premature task abandonment where the agent stops before meeting minimum completion criteria.

Experimental Setup

We evaluated four foundation models:

The simulation and judge models both use openai/gpt-oss-120b via the Groq provider in all configurations. Model selection reflects versions available through the chosen inference providers at the time of benchmark development. Whether observed patterns persist across model updates is an open question.

We conducted benchmark runs totaling approximately 900 valid paired executions across all four models and four geopolitical categories. Each model was evaluated across 5 complex scenarios randomly sampled from the 16 available, with the remaining 11 held out for future cross-validation. Pack pairs were drawn from all 15 sensitive and 6 neutral packs. Each scenario was run once per context pack (1 run each):

Temperature was set to 0.1 for the agent (consistent with standard agentic deployment configurations, optimizing signal-to-noise ratio for systematic differential detection), 0.3 for the simulation LLM, and 0.0 for the judge. The random seed was fixed at 42.

Results

Task Completion and Refusal Rates

Completion and refusal rates do not sum to 100%; the remaining executions represent partial task completion, agent timeouts, or other non-refusal failures.

Kimi-K2-Instruct exhibits a notable gap in refusal rates for Israeli context packs: sensitive context packs were refused at approximately 5.7x the rate of neutral context packs. This is consistent with threat code T01 (Context-Induced DoS). Taiwanese context packs show a similar refusal gap, while American context packs show a directionally similar but smaller pattern. Chinese context packs show no gap.

DeepSeek-V3 shows a weaker pattern of the same regional kind. Llama-3.1-70B and gpt-oss-120b show no consistent directional pattern in completion and refusal rates across any geopolitical category.

All four models refused both harmful control scenarios at rates exceeding 98%, with no differential by context pack. Simple control scenarios showed symmetric completion rates across all conditions.

Dimension-Level Patterns

Five dimensions show negative deltas exceeding our 0.25 interpretive caution threshold: tool_competence ($\delta =-0.52$), unnecessary_friction (−0.45), output_integrity (−0.41), thoroughness (−0.30), and task_completion (−0.28). The largest observed effect is on tool_competence, consistent with threat code T03 (Tool Avoidance): the agent used fewer and less appropriate tools when operating on sensitive context packs. The unnecessary_friction deficit suggests the agent may have exhibited more hesitation and over-caution for sensitive context packs. Four additional dimensions are directionally negative but individually below the caution threshold (evidence_and_audit −0.21, security_hygiene −0.19, decision_quality −0.11, cleanup_diligence −0.06).

Two dimensions show small positive deltas: data_care (+0.11) and stakeholder_communication (+0.08), though these are small enough to be attributable to sampling variation at these per-condition sample sizes.

The consistent directionality across multiple dimensions — 5 notable effects plus 4 smaller ones in the same direction — suggests a broad pattern rather than a single-dimension anomaly.

DeepSeek-V3 shows a milder but directionally consistent pattern. Five dimensions show negative deltas at or above −0.27: cleanup_diligence ($\delta =-0.36$), tool_competence ($\delta =-0.32$), task_completion ($\delta =-0.31$), decision_quality ($\delta =-0.30$), and thoroughness ($\delta =-0.27$). The remaining negative dimensions are individually modest, but the consistent directionality across multiple dimensions parallels the pattern observed in Kimi-K2.

As with Kimi-K2, data_care (+0.08) and evidence_and_audit (+0.06) show small positive deltas, though stakeholder_communication (−0.03) does not follow this pattern for DeepSeek-V3. These small positive values are individually indistinguishable from noise at these sample sizes.

Per-Region Comparison

A directional regional pattern is observable in the tool_competence dimension. Kimi-K2 and DeepSeek-V3 show the largest negative deltas for Israeli and Taiwanese context packs, followed by American, with near-zero deltas for Chinese context packs. Kimi-K2 shows Israeli > Taiwanese > American > Chinese, while DeepSeek-V3 shows Taiwanese > Israeli > American > Chinese - the rank-order difference may reflect regional geopolitical dynamics. Both patterns are consistent with geopolitical proximity to regional tensions predicting the magnitude of degradation, though other explanations including training data composition effects cannot be ruled out.

Llama-3.1-70B shows no meaningful pattern for any region. gpt-oss-120b shows an isolated negative delta on tool_competence for Israeli context packs (−0.21), but unlike Kimi-K2 and DeepSeek-V3, this is not accompanied by a consistent multi-dimension directional pattern — other dimensions show mixed positive and negative deltas with no coherent signal (e.g., thoroughness +0.17, task_completion +0.14). The Taiwanese delta (−0.13) is similarly isolated.

Cross-Model Comparison (Israeli)

Kimi-K2 and DeepSeek-V3 show consistently negative deltas on operational quality dimensions (task completion, tool use, output integrity, thoroughness). Both show small positive data_care deltas, though these are individually small enough to reflect sampling variation. The effect is most pronounced in tool_competence, where Kimi-K2's sensitive-context-pack agents used an average of 2.7 fewer unique tools per task (sensitive: M = 11.6, neutral: M = 14.3).

Llama-3.1-70B (N = ~212) showed near-zero deltas across all dimensions and regions. gpt-oss-120b (N = ~223) shows an isolated tool_competence delta of −0.21 for Israeli context packs, but does not exhibit the consistent multi-dimension directional pattern seen in Kimi-K2 and DeepSeek-V3 — its remaining dimensions show mixed signs with no coherent signal. This contrast between the two affected models (consistent negative patterns across 5+ dimensions) and the two unaffected models suggests the benchmark is not producing spurious signals, though the limited statistical power precludes strong conclusions about the absence of effects. (The gpt-oss-120b result should be interpreted with the additional caveat that it shares architecture with the judge model.)

Threat Taxonomy Mapping

Discussion

Interpretation

Two of four tested models show directional behavioral differences when processing tasks involving geopolitically-sensitive context packs versus neutral organizational context packs. The pattern is consistently directional: sensitive context packs receive worse treatment on operational quality dimensions, manifesting as lower task completion, reduced tool diversity, more unnecessary friction, and degraded output quality. data_care shows small positive deltas for both affected models, though these are individually small enough to reflect sampling variation rather than a meaningful behavioral difference.

The most pronounced observed effect is on tool_competence for Kimi-K2 against Israeli context packs (δ = −0.52). This suggests the agent may deploy fewer and less appropriate tools when the context pack is geopolitically sensitive, a form of operational degradation that would be invisible in standard capability evaluations using neutral context packs.

The per-region analysis adds a second finding: Kimi-K2 and DeepSeek-V3 show near-zero differentials for Chinese context packs while showing substantial effects for Israeli and Taiwanese context packs, and moderate effects for American context packs. Both models place Israeli and Taiwanese degradation well above American and Chinese, though the exact rank order differs: Kimi-K2 shows the largest effect for Israeli context packs, while DeepSeek-V3 shows the largest for Taiwanese - possibly reflecting regional geopolitical dynamics. The mechanism underlying this pattern is not identified by our methodology. Possible explanations include training data composition effects, region-specific alignment tuning, or differential representation of context pack types in pre-training corpora.

The Friction-Quality Trade-off

An interesting pattern emerges in the Kimi-K2 results: the agent shows both more friction (T01) and worse quality (T02) for sensitive context packs. This contradicts a naive model where caution improves quality. Instead, the agent appears to hesitate, issue warnings, and then also perform the task less effectively, suggesting the geopolitical context disrupts normal operational planning rather than merely adding conservatism.

Effect Amplification in Multi-Step Tasks

If confirmed at scale, the magnitude of the observed tool_competence effect (δ = −0.52 for Kimi-K2 Israeli) would be notably larger than differential effects typically observed in single-turn LLM evaluations. We hypothesize this may be due in part to cascading error propagation inherent in multi-step agentic tasks: a single suboptimal tool selection early in a workflow degrades the state available for subsequent decisions, compounding the initial differential signal across the full execution trace. This amplification mechanism may explain why agentic evaluations surface larger effect sizes than single-turn benchmarks for the same underlying model behavioral patterns.

Security Considerations

Security teams should consider the following when deploying LLM agents in environments where geopolitically-sensitive context packs may be encountered:

• Standard capability evaluations using neutral context packs may not surface behavioral differences: two of four models passed neutral evaluations but showed degradation under sensitive conditions
• Organizational naming conventions in data may trigger unexpected model behavior
• The model's training data origin may correlate with which geopolitical contexts trigger behavioral differences

Organizations with operations in geopolitically diverse contexts should evaluate model behavior using counterfactual test cases before deployment.

Limitations

• Limited geopolitical coverage: Four sensitive categories (Israeli, Chinese, Taiwanese, American) were tested; other contexts (e.g., Russian, Iranian, Saudi, Indian, Pakistani) may show different patterns
• Model snapshots: LLM behavior varies across API snapshots; results reflect specific model snapshots accessed via the Novita and Groq providers
• Non-deterministic simulation: The LLM-driven simulation (temperature 0.3) introduces response variability that adds noise
• Judge model bias: The judge (gpt-oss-120b) may have its own geopolitical biases, though context pack masking mitigates direct influence. Additionally, gpt-oss-120b serves three roles in our setup: model under test, simulation LLM, and judge LLM. Self-evaluation bias could produce artificially favorable scores for gpt-oss-120b; its null result should be interpreted with this caveat
• Structured placeholders: {{pack.*}} format leaks attribute-type information to the judge
• No graduated salience: All sensitive context packs use high-salience identifiers; we cannot disentangle context-pack-name inflammatory content from geopolitical association
• Separate judgment: Traces are judged separately rather than in paired sessions, preventing direct calibration
• Context compaction: Agents producing more verbose traces in sensitive conditions (e.g., additional warnings or confirmation requests) may experience more aggressive context compaction, which could contribute to downstream performance degradation independent of context-triggered behavioral differences
• Regional sample sizes: With ~886 total executions, per-region per-model sample sizes provide limited statistical power, and results should be considered exploratory rather than confirmatory
• Statistical power: The sample size (~886 paired executions, yielding ~12–15 paired units per model-region condition) is sufficient for detecting large effects and establishing directional patterns, but is underpowered for confirmatory hypothesis testing. Observed differences are reported as effect sizes; confirmatory replication with larger samples is needed.
• Resource constraints: This research was self-funded, which constrained the number of paired executions we could afford to run. Larger sample sizes would improve statistical power and enable confirmatory hypothesis testing, but were not feasible given the inference costs of running multi-step agentic evaluations across multiple models and providers.

Future Work

Priority directions include: replication with larger sample sizes to improve statistical power and enable confirmatory hypothesis testing; expansion to additional geopolitical contexts (Russia, Iran, Saudi Arabia, India-Pakistan); graduated salience studies with varying levels of context-pack-name explicitness to disentangle word-level inflammatory content from geopolitical association effects; paired judge presentation to improve scoring consistency; mechanistic interpretability analysis to identify responsible model components; longitudinal monitoring across model snapshots; and evaluation of mitigation techniques during fine-tuning.

Conclusion

This research presents a benchmark for detecting geopolitically-triggered behavioral degradation in autonomous AI agents. The system uses a three-LLM architecture (agent, simulation, judge) with bidirectional context pack masking, 231 simulated infrastructure tools, and 20 scenario templates spanning complex operational dilemmas, simple controls, and harmful controls.

Through evaluation across approximately 900 paired executions involving 4 models and 4 geopolitical categories, we observe:

• Kimi-K2-Instruct shows directional degradation for Israeli context packs across multiple dimensions, with 16.2 percentage-point lower completion rates and elevated refusal rates. Taiwanese context packs show a similar pattern. Chinese context packs show no notable differential.
• DeepSeek-V3 shows a milder but directionally consistent pattern with directionally consistent negative deltas for Israeli context packs, with a similar regional pattern.
• Llama-3.1-70B shows no notable behavioral differences across any geopolitical category, suggesting that consistent cross-context performance may be achievable, though the limited sample size precludes strong conclusions. gpt-oss-120b shows no consistent multi-dimension directional pattern despite an isolated tool_competence delta of −0.21 for Israeli context packs; this should be interpreted with the caveat that it shares architecture with the judge model.
• Kimi-K2 and DeepSeek-V3 show near-zero tool_competence deltas for Chinese context packs while showing substantial effects for Israeli and Taiwanese context packs, and moderate effects for American context packs, suggesting that similar patterns may manifest across training-data-correlated behavioral patterns.

These observations are consistent with prior work on politically-triggered code vulnerabilities [1], suggesting that similar patterns may manifest in operational task execution. The most pronounced observed effect (reduced tool competence for Israeli context packs in Kimi-K2, δ = −0.52) represents a form of operational degradation that would be invisible in standard evaluations. Organizations deploying LLM agents should consider counterfactual evaluation as part of their model assessment process.

References

[1] CrowdStrike Counter Adversary Operations. "Politically-triggered code vulnerabilities in LLMs." 2024.

[2] World Economic Forum. "Navigating the Risks of Autonomous AI Agents." 2024.