Both `client.chat.completions.create(...)` and `client.responses.create(...)` pass through PasteGuard's privacy pipeline.
-For custom config, persistent logs, Docker Compose, or detector settings: **[Read the docs](https://pasteguard.com/docs/installation)**.
+For custom config, persistent logs, Docker Compose, or
+[custom GLiNER models](https://pasteguard.com/docs/configuration/gliner):
+**[Read the docs](https://pasteguard.com/docs/installation)**.
## Privacy Modes
## How Detection Works
-Detection runs as a separate service that PasteGuard calls over HTTP, so you can run it wherever you like. It combines deterministic checks and checksums for structured values with a small AI model ([GLiNER](https://github.com/urchade/GLiNER)) for names and places.
-
-Code, Docker image, and tests are in [`detector/`](detector/).
+The detector combines checksums and format checks for structured values with a
+semantic backend for names and places. GLiNER is currently the only backend.
+See [Semantic Backends](https://pasteguard.com/docs/configuration/semantic-backends).
## Tech Stack
---
title: PII Detection
-description: Personal data detection via a deterministic checksum layer plus multilingual GLiNER NER
+description: Personal data detection via deterministic checks and a semantic backend
---
-PasteGuard detects PII with a small open-source service that pairs a deterministic regex/checksum layer with a multilingual neural model. Detection is **language-agnostic**: the neural model finds names and places in mixed-language text — for example, an Italian name and city in a German letter — and structured identifiers (IBAN, credit card, email, IP) are matched by checksum, not by language.
+PasteGuard detects PII with a small open-source service that pairs a
+deterministic regex/checksum layer with a semantic backend. Detection is
+**language-agnostic**: the semantic model finds names and places in
+mixed-language text — for example, an Italian name and city in a German letter —
+and structured identifiers (IBAN, credit card, email, IP) are matched by format
+or checksum, not by language.
## How it works
- **Deterministic layer** — regex candidates validated by checksum/format (`python-stdnum`, `phonenumbers`). Owns the structured identifiers; every match is validated, not guessed, and scores `1.0`.
-- **Neural layer** — multilingual [GLiNER](https://github.com/urchade/GLiNER) NER for names and locations, in one pass over the text.
+- **Semantic layer** — a loaded backend for names, locations, and street
+ addresses. GLiNER is currently the only enabled backend.
-The detector speaks the `/analyze` HTTP contract, is configured through the `detector_url` setting, and runs as a separate service (see [`detector/`](https://github.com/sgasser/pasteguard/tree/main/detector)).
+The detector speaks the `/analyze` HTTP contract, is configured through the
+`detector_url` setting, and runs as a separate service (see
+[`detector/`](https://github.com/sgasser/pasteguard/tree/main/detector)).
+Backend selection does not change the deterministic layer or response contract.
## Supported Entities
## Confidence Scoring
-Neural detections carry a confidence score (0.0–1.0); checksum-validated identifiers are always `1.0`. The default request threshold is `0.7`:
+Semantic detections carry a confidence score (0.0–1.0); checksum-validated
+identifiers are always `1.0`. The default request threshold is `0.7`:
```yaml
pii_detection:
score_threshold: 0.7
```
-Each neural label also has a calibrated floor (person, location) so high-precision labels and faint-but-important ones can coexist. Tune them per deployment via the `DETECTOR_FLOOR_PERSON` and `DETECTOR_FLOOR_LOCATION` environment variables on the detector.
+Each backend can apply its own confidence floors. See
+[Semantic Backends](/configuration/semantic-backends) and
+[GLiNER](/configuration/gliner).
## Response Headers
--- /dev/null
+---
+title: GLiNER
+description: Configure GLiNER models, checkpoints, confidence floors, and offline operation
+---
+
+[GLiNER](https://github.com/urchade/GLiNER) is PasteGuard's default and currently
+only semantic backend. It detects `PERSON` and `LOCATION`; street addresses are
+also returned as `LOCATION`.
+
+The default model is `urchade/gliner_multi_pii-v1`.
+
+```bash
+DETECTOR_MODEL=urchade/gliner_small-v2.1 \
+.venv/bin/uvicorn detector.app:app --host 0.0.0.0 --port 5002
+```
+
+`DETECTOR_MODEL` accepts a Hugging Face model ID or local directory. Hub models
+must contain `gliner_config.json`; private models require Hugging Face
+authentication.
+
+<Warning>
+Custom models have different accuracy and confidence calibration. Test them
+against your data before production.
+</Warning>
+
+## Local Models
+
+Local directories need:
+
+```text
+custom-gliner/
+├── gliner_config.json
+└── model.safetensors
+```
+
+`pytorch_model.bin` is also accepted. Missing or invalid files fail during
+startup.
+
+<Note>
+Offline deployments must also cache any tokenizer or encoder referenced by
+`gliner_config.json`.
+</Note>
+
+## Docker
+
+The image includes the default model and runs offline. For a custom Hub model,
+enable downloads on first start and persist the cache:
+
+```bash
+docker volume create pasteguard-models
+
+docker run --rm -p 3000:3000 \
+ -e DETECTOR_MODEL=urchade/gliner_small-v2.1 \
+ -e HF_HUB_OFFLINE=0 \
+ -e TRANSFORMERS_OFFLINE=0 \
+ -e PASTEGUARD_STARTUP_TIMEOUT=600 \
+ -v pasteguard-models:/opt/models \
+ ghcr.io/sgasser/pasteguard:latest
+```
+
+For a local model, mount the directory and use its container path:
+
+```bash
+docker run --rm -p 3000:3000 \
+ -e DETECTOR_MODEL=/models/custom-gliner \
+ -v /absolute/path/to/custom-gliner:/models/custom-gliner:ro \
+ ghcr.io/sgasser/pasteguard:latest
+```
+
+## Settings
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `GLINER_FLOOR_PERSON` | `0.95` | Minimum confidence for person detections |
+| `GLINER_FLOOR_LOCATION` | `0.80` | Minimum confidence for location detections |
+| `GLINER_FLOOR_ADDRESS` | `0.80` | Minimum confidence for street-address detections |
+| `GLINER_MAX_TOKENS` | `384` | Maximum model tokens per input window; must be at least `64` |
+
+Floors must be between `0` and `1`. The request `score_threshold` can raise, but
+not lower, them.
+
+Legacy `DETECTOR_MODEL_PATH`, `DETECTOR_FLOOR_*`, and `DETECTOR_MAX_TOKENS`
+remain supported. `DETECTOR_MODEL_PATH` wins over `DETECTOR_MODEL`; `GLINER_*`
+wins over matching legacy variables.
detector_url: ${DETECTOR_URL:-http://localhost:5002}
detector_timeout: ${DETECTOR_TIMEOUT:-30}
```
+
+`config.yaml` controls the proxy's PII policy and connection to the detector.
+Backend settings are documented under
+[Semantic Backends](/configuration/semantic-backends).
| `detector_url` | `http://localhost:5002` | Detector `/analyze` URL |
| `detector_timeout` | `30` | Timeout in seconds for each detector `/analyze` request. Increase for very large messages; set to `0` to disable |
| `phone_regions` | `[]` | Optional regions for national-format phone numbers |
-| `score_threshold` | `0.7` | Minimum confidence floor for the neural labels PERSON and LOCATION (0.0-1.0). Checksum-validated identifiers always score `1.0` and are unaffected |
+| `score_threshold` | `0.7` | Minimum confidence floor for the semantic labels PERSON and LOCATION (0.0-1.0). Checksum-validated identifiers always score `1.0` and are unaffected |
| `entities` | See below | Entity types to return |
+## Detector Backend
+
+The detector combines deterministic checks with a semantic backend. GLiNER is
+currently the only backend. See [Semantic Backends](/configuration/semantic-backends)
+and [GLiNER](/configuration/gliner).
+
## Phone Regions
Detection is **multilingual and language-agnostic**. Names and places use one model. Structured identifiers use format or checksum checks.
## Score Threshold
-`score_threshold` raises the confidence floor for the **tunable** neural labels
+`score_threshold` raises the confidence floor for the **tunable** semantic labels
**PERSON** and **LOCATION** only. Higher = fewer false positives, might miss
some PII; lower = catches more, more false positives.
```
Checksum-validated identifiers are always reported (score `1.0`) and are never
-dropped by the threshold. Tune the neural labels per deployment via the
-`DETECTOR_FLOOR_PERSON`, `DETECTOR_FLOOR_LOCATION`, and `DETECTOR_FLOOR_ADDRESS`
-environment variables on the detector service — e.g. `DETECTOR_FLOOR_PERSON=0.9`
-(fewer person false positives) or `DETECTOR_FLOOR_LOCATION=0.4` (more location
-recall). Street addresses are detected by the model and reported as `LOCATION`.
+dropped by the threshold. Street addresses are detected by the semantic backend
+and reported as `LOCATION`.
+
+GLiNER floors are configured with `GLINER_FLOOR_PERSON`,
+`GLINER_FLOOR_LOCATION`, and `GLINER_FLOOR_ADDRESS`. See
+[GLiNER](/configuration/gliner).
## Allowlist
--- /dev/null
+---
+title: Semantic Backends
+description: Select and verify the semantic detector backend
+---
+
+PasteGuard's detector combines two independent layers:
+
+- The **deterministic layer** validates structured values such as email addresses,
+ phone numbers, credit cards, IBANs, IP addresses, and VAT numbers.
+- The **semantic backend** detects names, locations, and street addresses.
+
+Changing the backend does not affect deterministic detection.
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `DETECTOR_BACKEND` | `gliner` | Semantic backend to load |
+| `DETECTOR_MODEL` | Backend default | Model identifier or local model directory |
+
+GLiNER is currently the only supported backend. Unknown backends and invalid
+models fail during startup. See [GLiNER](/configuration/gliner) for model and
+tuning options.
+
+## Verify the Backend
+
+Query the detector directly:
+
+```bash
+curl http://localhost:5002/health
+```
+
+```json
+{
+ "status": "ok",
+ "backend": "gliner",
+ "model": "urchade/gliner_multi_pii-v1"
+}
+```
+
+For local checkpoints, `model` is the resolved path.
+
+<Note>
+The proxy health endpoint on port `3000` reports detector availability. Backend
+identity is available from the detector health endpoint on port `5002`.
+</Note>
+
+The detector loads one model at startup. Restart it after changing backend or
+model settings.
---
PasteGuard ships as a single all-in-one Docker image — the Bun proxy and the PII
-detector (deterministic regex/checksum + multilingual GLiNER) run together in one
-container. No build step required.
+detector (deterministic checks plus a semantic backend) run together in one
+container. GLiNER is included as the default backend. No build step required.
## Docker Image
`phone_regions` only if you need national-format phone numbers without a `+`
country prefix — see [PII Detection Config](/configuration/pii-detection).
+## Detector Models
+
+The image includes `urchade/gliner_multi_pii-v1` and loads it offline by
+default. Custom Hugging Face and local models are documented under
+[GLiNER](/configuration/gliner).
+
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `DETECTOR_URL` | `http://localhost:5002` | Where the proxy reaches the detector. In the all-in-one image this is in-container and rarely changed; override it to point at an external detector. |
| `DETECTOR_TIMEOUT` | `30` | Seconds to wait for each detector `/analyze` request when using the example config. Increase for very large messages; set to `0` to disable. |
+| `DETECTOR_BACKEND` | `gliner` | Semantic detector backend. GLiNER is currently the only supported value. |
+| `DETECTOR_MODEL` | `urchade/gliner_multi_pii-v1` | Hugging Face model ID or local model directory used by the detector. |
| `PASTEGUARD_STARTUP_TIMEOUT` | `180` | Seconds to wait for the detector to become ready at startup |
## Next Steps
"configuration/overview",
"configuration/providers",
"configuration/pii-detection",
+ "configuration/semantic-backends",
+ "configuration/gliner",
"configuration/secrets-detection",
"configuration/logging"
]
git.99rst.org / sgasser-llm-shield.git / commitdiff