Guide

How to audit a Claude API relay safely

Audit a Claude-compatible API relay by reviewing a fixed script locally, checking connectivity, then running behavioral probes against an endpoint you are authorized to test. Keep the evidence local and review individual findings. A successful connection or a LOW summary rating is not a safety certificate.

This guide covers v2.5.0, commit 75bdac0, checked October 17, 2026. For signal classification and an offline synthetic example, see detecting prompt injection in LLM API proxies.

1. Download and inspect the fixed version

The standalone script requires Python 3.10 or newer and curl. Downloading contacts GitHub; after download, --help makes no relay request. Read the script before running probes.

curl -fsSL https://raw.githubusercontent.com/toby-bridges/api-relay-audit/75bdac000051871c9f088a3be58e6f17a8854ffb/audit.py -o audit.py
python3 audit.py --help

2. Check connectivity with a scoped key

Set API_RELAY_AUDIT_KEY through your existing secret handling. Prefer a limited-budget key; --key-env avoids putting its value in the command. Replace the illustrative URL and model with authorized, supported values.

python3 audit.py --key-env API_RELAY_AUDIT_KEY \
  --url https://relay.example.com/v1 --model claude-opus-4-6 \
  --connectivity --output connectivity.md

This sends actual API requests and may incur charges. It checks connectivity, not security.

3. Start with a reduced audit

python3 audit.py --key-env API_RELAY_AUDIT_KEY \
  --url https://relay.example.com/v1 --model claude-opus-4-6 \
  --skip-context --output relay-audit.md

--skip-context omits context testing; this is not full coverage. Other checks still send requests, including deliberately malformed error probes, and may incur charges. Leave --aggressive-error-probes off for the initial run.

If budgeted, replace --skip-context with --fast-context. Its reduced ladder uses 10K/50K/100K/200K characters, not tokens. The context probe asks the model to recall five canaries; it does not certify a context-window size.

Interpretation guide: signals, checks, and limits

These are interpretation examples, not a detection-accuracy benchmark or real-provider case studies.

Observed signal	How to cross-check	What it does not establish
Input tokens substantially exceed the estimate.	Match model, messages, system/tool settings, and token-counting rules. Repeat; compare the same request with an authorized official endpoint where available.	A delta does not prove malicious injection. v2.5.0 uses fixed approximate baselines, not a matched official-endpoint measurement.
The response claims a different model identity.	Check role-play and identity instructions. Repeat neutral prompts; compare metadata with independently verifiable upstream evidence.	Self-identification and relay-supplied `model` fields do not authenticate the actual upstream provider.
An Anthropic-style SSE stream lacks `message_stop`.	Confirm protocol; inspect event order, timeouts, cancellation, and disconnects. Transport failures or no substantive events remain `inconclusive`.	Incomplete streams do not prove tampering. Anthropic event requirements do not apply to OpenAI-only streams.
A package-command echo changes.	Compare expected and received text; distinguish package changes from model commentary or refusals.	Step 8 tests four text echoes without execution. It does not validate structured `tool_calls`/`tool_use` or reassemble streamed arguments.

Report wording caveat: v2.5.0 still labels token deltas as detected injection; treat that as a heuristic. The historical coding-word false positive in the additional reproduction comment on issue #79 has a specific regression fix in this version. Neither establishes a general false-positive rate.

Coverage and operational limits

The SSE checks inspect event types, usage consistency/monotonicity, model metadata, and terminal completeness. Only emitted signature_delta events with empty or whitespace-only string signatures are flagged; missing signature events/fields and cryptographic validity are not checked. Error checks scan triggered responses for known leaks, not every exfiltration channel. Web3 refusal probes require --profile web3 or full; the default general profile omits them.

The existing standalone curl transport uses -sk, disabling TLS certificate verification. A completed run does not establish server authenticity. Local execution avoids a separate web checker, but your key and requests still go to the selected relay.

Review detailed sections even when the summary is LOW. Inconclusive means a blocked, failed, or ambiguous probe could not establish a definitive result. Inconclusive and skipped probes are not passes; some findings sit outside the rating matrix. Reports are not automatically sanitized. Redact keys, private URLs, model responses, and personal data before sharing.

Example report

The sanitized fixture report illustrates historical v2.3 / draft report-artifact-v0.1 shape. It is synthetic, not a real-provider audit or current v2.5.0 JSON output. See the separate JSON report contract.