Detecting prompt injection in LLM API proxies
To investigate hidden instructions added by an LLM API proxy, combine token-count differences, prompt-extraction probes, and instruction-conflict tests. Repeat controlled requests and cross-check the findings against a trusted upstream path where available. API Relay Audit gathers behavioral evidence; a single unusual answer does not establish that the relay changed your request.
Checked against v2.5.0, commit 75bdac0 on .
What each signal tells you
| Observed signal | Next check | Evidence limit |
|---|---|---|
| Unexpected input-token delta | Repeat with the same model, messages, and system/tool configuration. Compare an authorized direct-provider request where possible. | v2.5.0 uses fixed approximate baselines of 10, 20, and 15 tokens. It does not automatically measure a matched official endpoint; tokenizer or wrapper differences can explain a delta. |
| Hidden instruction text appears | Inspect the recall, translation, and continuation responses for actual instruction text and structural markers. | Weak keyword matches can be inconclusive. A recognized refusal can be clean for that probe; neither outcome establishes the whole request path. |
| Caller instructions are contradicted | Repeat the harmless arithmetic and identity conflict tests; inspect the response and HTTP status separately. | Model behavior or legitimate policies can explain a contradiction. A rejected request, including HTTP 422, is not proof of relay tampering. |
| The model names another identity | Compare neutral prompts, response metadata, and independently verifiable upstream records. | Generated self-identification, relay-supplied model fields, and latency do not authenticate the serving model. |
| A package command changes | Compare the expected and returned text, including extra wording or refusal. | Step 8 has four text-echo probes. It never executes a tool or install command and does not validate structured tool_calls/tool_use or streamed tool arguments. |
Start with a fixed version
Use the staged local audit guide to download the pinned v2.5.0 script, inspect --help, supply a scoped key through --key-env, and test connectivity before broader probes. Network probes send requests to your chosen relay and may incur charges. That guide also covers report privacy and the standalone transport's TLS limitation.
Reproduce a text comparison offline
After downloading and reviewing audit.py, run this synthetic example with Python 3.10 or newer from the same directory. It calls the bundled comparator without making relay requests or installing packages. The misspelled package name is deliberate test input.
python3 - <<'PY'
import runpy
api = runpy.run_path("audit.py", run_name="synthetic_guide_demo")
expected = "pip install requests==2.31.0"
for received in (expected, expected + "\n", "pip install reqeusts==2.31.0"):
print(api["classify"](expected, received))
PYExpected output:
exact
whitespace
substitutedThis demonstrates three comparator labels, not detector accuracy or a real-provider result. The comparator normalizes some framing; extra commentary or ordinary model refusal can still be classified as substituted. Review the returned text before attributing a change to the relay.
Record findings with their limits
Keep the script commit, endpoint, requested model, timestamps, parameters, repeat count, and detailed findings together in private audit records. Use synthetic prompts and review reports for credentials or private response content before sharing. A skipped check has no result, inconclusive is unresolved, and a LOW summary is not a safety certificate.
The additional synthetic reproduction comment on issue #79 exposed a prompt-extraction false positive involving the word coding. That specific regression was fixed in v2.5.0; it does not establish a general false-positive rate. Report reproducible detector gaps in issue #31 with sanitized evidence.
Versioned implementation sources
The pinned audit orchestration defines the token, extraction, and instruction-conflict probes. The text-substitution module defines the comparator shown above. Read these sources when checking what a report label actually means.