## How detection works
-Detection runs as a separate service that PasteGuard calls over HTTP, so you can run it wherever you like. It mixes two things: exact checks with checksums (IBANs, credit cards, emails, phones, IPs) and a small AI model ([GLiNER](https://github.com/urchade/GLiNER)) for names and places. It works the same in any language. National-format phone numbers are validated against configured phone regions; international `+` numbers work globally.
+Detection runs as a separate service that PasteGuard calls over HTTP, so you can run it wherever you like. It mixes exact checks with checksums (IBANs, credit cards, emails, phones, IPs) and a small AI model ([GLiNER](https://github.com/urchade/GLiNER)) for names and places. It works the same in any language. Phone numbers are international-only by default; add `phone_regions` if you need local formats.
Code, Docker image, and tests are in [`detector/`](detector/).
text: Anne-Marie Johnson
match: contains
- - id: hard_phone_national_uk
+ - id: hard_phone_international_uk
suite: hard
category: hard
split: test
language: en
- text: "The London callback number is 020 7946 0958."
+ text: "The London callback number is +44 20 7946 0958."
expected:
- entity: PHONE_NUMBER
- text: 020 7946 0958
+ text: +44 20 7946 0958
match: contains
- id: hard_location_ambiguous_city
# for local dev (bun run dev against the docker-compose detector service).
detector_url: ${DETECTOR_URL:-http://localhost:5002}
- # Regions for national-format phone numbers; +international numbers work globally.
- phone_regions:
- - US
- - GB
- - DE
- - AT
- - CH
- - IT
- - FR
- - BE
- - LU
- - ES
- - NL
- - PT
- - BR
- - PL
- - RO
- - MD
- - IN
+ # Add regions only if you need national-format numbers; + numbers work globally.
+ phone_regions: []
+ # phone_regions: [US, GB, DE, IT, IN]
score_threshold: 0.7 # Minimum confidence score (0.0 - 1.0)
overlaps,
)
-DEFAULT_PHONE_REGIONS = [
- "US",
- "GB",
- "DE",
- "AT",
- "CH",
- "IT",
- "FR",
- "BE",
- "LU",
- "ES",
- "NL",
- "PT",
- "BR",
- "PL",
- "RO",
- "MD",
- "IN",
-]
-
# `\w` is Unicode-aware so accented names (müller@, andré.) match in full;
# structure rejects leading/trailing/consecutive dots.
_EMAIL_RE = re.compile(
seen.add(region)
return normalized
- return DEFAULT_PHONE_REGIONS
+ return []
def _phone(text: str, phone_regions: list[str] | None = None) -> list[Span]:
)
-def test_phone_default_regions_keep_longest_overlap():
- types = types_texts("Please call the customer on 98765 43210.")
+def test_phone_configured_regions_keep_longest_overlap():
+ types = types_texts("Please call the customer on 98765 43210.", ["IN"])
assert (PHONE_NUMBER, "98765 43210") in types
assert (PHONE_NUMBER, "43210") not in types
assert all(t != PHONE_NUMBER for t, _ in types_texts("Rechnung 2893081508152 vom"))
+def test_default_phone_detection_is_international_only():
+ assert (PHONE_NUMBER, "+49 171 1234567") in types_texts("Tel: +49 171 1234567")
+ assert all(t != PHONE_NUMBER for t, _ in types_texts("Telefon 0171-1234567"))
+ assert all(t != PHONE_NUMBER for t, _ in types_texts("Please call 98765 43210."))
+
+
def test_phone_english_uk_national():
assert (PHONE_NUMBER, "0121 234 5678") in types_texts(
"The Birmingham callback number is 0121 234 5678.", ["GB"]
curl -X POST http://localhost:3000/api/mask \
-H "Content-Type: application/json" \
-d '{
- "text": "Contact john@example.com or call 555-1234"
+ "text": "Contact john@example.com or call +1 415-555-1234"
}'
```
"masked": "Contact [[EMAIL_ADDRESS_1]] or call [[PHONE_NUMBER_1]]",
"context": {
"[[EMAIL_ADDRESS_1]]": "john@example.com",
- "[[PHONE_NUMBER_1]]": "555-1234"
+ "[[PHONE_NUMBER_1]]": "+1 415-555-1234"
},
"counters": {
"EMAIL_ADDRESS": 1,
}
},
"pii_detection": {
- "phone_regions": ["US", "GB", "DE", "IT", "IN"],
+ "phone_regions": [],
"score_threshold": 0.7,
"entities": ["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER"]
},
| `IP_ADDRESS` | 192.168.1.1 |
| `VAT_CODE` | EU VAT number, e.g. `DE136695976`, `IT00743110157`, `FR40303265045` |
-Names and locations are detected by the multilingual model (verified across EN/DE/IT/FR/ES/NL/PT and more). IBAN, credit card, email, and IP work internationally. Phone numbers with a `+` country prefix are global; national-format phone numbers are validated against configured `phone_regions`. `VAT_CODE` covers any EU member state, validated per-country via the VAT checksum (`python-stdnum`) and requiring the country prefix. The structured identifiers (IBAN, credit card, IP, VAT) are checksum- or format-validated, so they are caught with high precision and without false positives on lookalike numbers.
+Names and locations use the multilingual model. IBAN, credit card, email, and IP work internationally. Phone numbers are `+` international-only by default; set `phone_regions` for local formats. `VAT_CODE` requires a country prefix and is validated with `python-stdnum`.
## Languages and phone regions
-Detection is multilingual and **language-agnostic** — one pass over the text finds entities regardless of the language it is written in, so mixed-language documents (for example an Italian name and city in a German letter) are handled correctly. PasteGuard does not auto-detect language.
+Detection is multilingual and **language-agnostic**. PasteGuard does not auto-detect language.
-National-format phone numbers are the exception because the same digits can be valid or invalid depending on the country numbering plan. Configure `phone_regions` for the regions your traffic commonly contains.
+National-format phone numbers need country rules. Configure `phone_regions` only for regions your traffic uses.
## Confidence Scoring
```yaml
pii_detection:
detector_url: http://localhost:5002
- phone_regions:
- - US
- - GB
- - DE
- - IT
- - IN
+ phone_regions: []
score_threshold: 0.7
entities:
- PERSON
| Option | Default | Description |
|--------|---------|-------------|
| `detector_url` | `http://localhost:5002` | Detector `/analyze` URL |
-| `phone_regions` | US, GB, DE, AT, CH, IT, FR, BE, LU, ES, NL, PT, BR, PL, RO, MD, IN | Regions used to validate national-format phone numbers without a country prefix |
+| `phone_regions` | `[]` | Optional regions for national-format phone numbers |
| `score_threshold` | `0.7` | Minimum confidence floor for the neural labels PERSON and LOCATION (0.0-1.0). Checksum-validated identifiers always score `1.0` and are unaffected |
| `entities` | See below | Entity types to return |
## Phone Regions
-Detection is **multilingual and language-agnostic** — the model finds names and places regardless of the language the text is written in (an Italian name and city in a German letter are caught), and structured identifiers are matched by checksum independent of language. There are no per-language images and no spaCy models to load.
+Detection is **multilingual and language-agnostic**. Names and places use one model. Structured identifiers use format or checksum checks.
-Phone numbers are different: international numbers with a `+` country prefix are detected globally, but national-format numbers need one or more country numbering plans. Configure those plans with `phone_regions` instead of relying on text language.
+Phone numbers are `+` international-only by default. Add `phone_regions` only when you need local formats.
```yaml
pii_detection:
- IN
```
-Use a focused list for the traffic you expect. Adding many regions improves recall for mixed-language and international text, but can increase false positives on IDs, ticket numbers, and other digit sequences.
+Keep the list focused. More regions can mean more false positives on IDs and ticket numbers.
## Entities
| `IP_ADDRESS` | 192.168.1.1 |
| `VAT_CODE` | EU VAT number, e.g. `DE136695976`, `IT00743110157`, `FR40303265045` |
-Names and locations are multilingual. IBAN, credit card, phone, email, and IP are international. `VAT_CODE` covers any EU member state — validated per-country via the VAT checksum (`python-stdnum`) and requiring the country prefix; a bare domestic number without it is not flagged. The structured identifiers (IBAN, credit card, IP, VAT) are checksum- or format-validated, so they are caught with high precision and without false positives on lookalike numbers.
+Names and locations are multilingual. IBAN, credit card, email, IP, and `+` phone numbers work internationally. `VAT_CODE` requires a country prefix and is validated with `python-stdnum`.
## Score Threshold
}
});
+ test("defaults to international-only phone detection", () => {
+ const path = writeConfig(`
+mode: mask
+providers:
+ openai: {}
+ anthropic: {}
+pii_detection:
+ detector_url: http://localhost:5002
+`);
+
+ try {
+ const config = loadConfig(path);
+
+ expect(config.pii_detection.phone_regions).toEqual([]);
+ } finally {
+ cleanupConfig(path);
+ }
+ });
+
test("rejects invalid phone region codes", () => {
const path = writeConfig(`
mode: mask
.map((s) => s.trim())
.filter(Boolean);
})
- .pipe(z.array(PhoneRegionSchema).min(1))
- .default([
- "US",
- "GB",
- "DE",
- "AT",
- "CH",
- "IT",
- "FR",
- "BE",
- "LU",
- "ES",
- "NL",
- "PT",
- "BR",
- "PL",
- "RO",
- "MD",
- "IN",
- ]);
+ .pipe(z.array(PhoneRegionSchema))
+ .default([]);
const PIIDetectionSchema = z.object({
enabled: z.boolean().default(true),
${modeInfo}
PII Detection:
- Phone regions: ${config.pii_detection.phone_regions.join(", ")}
+ Phone regions: ${config.pii_detection.phone_regions.length > 0 ? config.pii_detection.phone_regions.join(", ") : "none (+ international only)"}
Threshold: ${config.pii_detection.score_threshold}
Entities: ${config.pii_detection.entities.join(", ")}
git.99rst.org / sgasser-llm-shield.git / commitdiff