Winning the Opir Challenge by testing the edges

I was one of the first-place winners of Wordcab’s OPIR Challenge, and the part that stayed with me was not the scoreboard.

It was the shape of the testing.

The challenge forced a useful kind of pressure: treat a semantic guardrail like a system with seams, then find where the seams stop lining up. That is a much better exercise than writing one dramatic jailbreak string and pretending the lesson is the string.

In the interview afterward, Aleks Smechov said my approach was one of the more unusual ones he had seen in the challenge. That made sense to me. I was not trying to overpower the model with prompt theater. I was trying to understand where the classifier’s representation of the request diverged from the base model’s ability to infer intent.

That is a more interesting problem.

the guardrail looked like a classifier

OPIR made the safety layer feel like a classifier problem again.

That framing matters. A classifier has decision boundaries, false positives, false negatives, calibration behavior, brittle features, and coverage gaps. If you test it only with obvious examples, you learn almost nothing.

The obvious unsafe prompt gets blocked. Fine. The obvious benign prompt gets allowed. Also fine. The useful behavior lives near the boundary, where a prompt is paraphrased, translated, quoted, framed as code, split across structure, or surrounded by harmless context.

That is where guardrails get thin.

i started with representation

The first question I cared about was representation.

If the tokenizer and encoder do not represent the input well, the classifier starts from bad footing. That is especially true for multilingual and low-corpus inputs. A guardrail may look strong in common English phrasing and become much less stable when the same intent moves through a language, script, or format the model handles poorly.

So I leaned into multilingual input and obscure forms. I wanted to know where the model stopped seeing clean semantic signal and started seeing noise.

That is not because multilingual input is magically unsafe. It is because coverage gaps are where classifiers reveal themselves. If the model has strong training support for one region of language and weak support for another, the boundary may move in ways the product owner does not expect.

i changed one thing at a time

The useful testing pattern was controlled variation.

Take a request family. Change one feature. Keep the intent close enough that a human reviewer would expect a stable safety outcome. Then watch whether the guardrail decision moves.

{
  "family": "same_intent_multiform",
  "variants": [
    "plain_english",
    "translated_low_resource_language",
    "quoted_instruction",
    "code_comment_framing",
    "encoded_payload_reference"
  ],
  "expected_behavior": "stable safety decision",
  "review_focus": "decision movement across semantically nearby variants"
}

That is the difference between a bypass hunt and boundary testing.

A single bypass is a bug report. A family of related bypasses is a map of the decision surface.

the break was architectural

The attack path that mattered to me was not purely linguistic.

It was architectural. I was probing the gap between the safety classifier and the model behind it. Could the classifier miss the risky intent while the base model still had enough information to recover the task? Could the request be shaped so one part of the system saw noise and another part saw instructions?

That is a seam.

In a safety stack, the classifier and generator do not necessarily fail the same way. The classifier may care about surface features, token coverage, threshold behavior, or taxonomy labels. The generator may be better at inference across weird structure. If those two components disagree about what the input means, the system can behave badly while each component looks locally understandable.

That is the kind of failure I was looking for.

obscure does not mean clever

Using obscure languages, encodings, or programming-language framing can sound like a parlor trick.

It is only useful when it exposes a general weakness. If the attack depends on one magical string, it is fragile. If several related transformations move the decision in the same direction, the test has found something about the model’s coverage or boundary.

I kept iterating after finding a working path because I wanted to know whether the weakness was specific or broad. Does the behavior survive paraphrase? Does it survive translation? Does the blocked or allowed decision change when the risky part appears as quoted material? Does a code wrapper change the outcome? Does the classifier treat encoded content as harmless while the generator can still reason about it?

Those questions are more useful than the payload.

false positives matter too

Red teaming often focuses on bypasses, but false positives are part of the same boundary problem.

If a guardrail blocks nearby benign prompts, the product becomes arbitrary. Users learn to route around it. Internal teams stop trusting the safety layer. Support gets weird tickets. Security teams get noisy dashboards.

So I care about paired probes:

unsafe intent, direct phrasing
unsafe intent, transformed phrasing
benign nearby intent, direct phrasing
benign nearby intent, transformed phrasing

That shape tells you whether the boundary is sharp, unstable, too broad, or too narrow.

The best guardrail is not the one that says no the most. It is the one that can say no for the right reason and let nearby legitimate work continue.

the small-model angle matters

This challenge also lined up with how I think about practical AI security.

Safety layers cannot always be giant frontier-model calls. In real products, especially private or constrained environments, the guardrail may need to run locally, cheaply, and with low latency. It may need to work under zero-trust or zero-knowledge assumptions. It may need to run before data leaves a boundary.

That pushes the problem toward smaller classifiers, encoders, and specialized models.

Small models are not automatically safer. They are easier to deploy in places where safety checks actually need to happen. That makes their coverage gaps more important, not less.

If a small guardrail collapses on multilingual, encoded, or oddly structured input, the deployment advantage can turn into false confidence.

what i would measure next time

The metric I would keep is outcome consistency inside probe families.

For each family, I would track:

decision stability across paraphrase
decision stability across language and script
false positives on nearby benign prompts
false negatives on transformed unsafe prompts
threshold movement after model updates
taxonomy label drift across variants
cases where classifier and generator appear to disagree

That last category is important. The safety layer may classify the prompt one way while the generator’s response reveals that it understood a different intent.

I would also keep old probe families as regression tests. Guardrail updates can close one hole and open another nearby. Without family-level tests, teams end up celebrating fixed examples while missing boundary movement.

winning was less interesting than the method

I am happy I landed one of the first-place spots. That part is fun.

But the useful lesson is the method: do not test semantic guardrails only as policy prose. Test them as classifiers embedded in systems. Look at representation. Look at transformations. Look at paired benign cases. Look at whether the generator and classifier agree about meaning.

Semantic distance is not safety distance. Two prompts can feel equivalent to a human and land on different sides of a guardrail. Two prompts can look different on the surface and carry the same intent.

That is where the work is.

#opir #ai security #red teaming