mempalace

jason/mempalace

Fork 0

Commit Graph

Author	SHA1	Message	Date
MSL	a0b7ba005d	feat(privacy): treat Tailscale CGNAT range (100.64.0.0/10) as local 2 files changed, 60 insertions, 0 deletions. 2 new tests (RED-first). Follow-up to #1224's privacy warning. The URL-based heuristic in ``mempalace.llm_client._endpoint_is_local`` shipped without recognizing Tailscale's CGNAT range (100.64.0.0/10), so a user running LM Studio, Ollama, or any local LLM accessible via a Tailscale-assigned 100.x.x.x address would currently get a wrong privacy warning — Tailscale addresses are network-private (only reachable inside the user's Tailnet) but they're not RFC1918, so the heuristic was treating them as external. This PR adds CGNAT recognition: when the hostname starts with ``100.`` AND the second octet is between 64 and 127 inclusive, it's classified as local. Addresses in 100.x.x.x outside that range (i.e. second octet < 64 or > 127) are regular allocated public space and remain external, so a user pointing at a public 100.0.0.1 still gets the warning. Concrete user impact: Before: ``mempalace init --llm-provider openai-compat --llm-endpoint http://100.100.50.50:1234`` (LM Studio on Tailnet) → triggers privacy warning incorrectly. After: same command → no warning. data stays inside the user's Tailnet, which is what the warning is supposed to protect against. TDD: 2 tests added in ``tests/test_llm_client.py``, both RED-first. 1. ``test_openai_compat_provider_tailscale_cgnat_endpoint_is_local`` — covers three Tailscale CGNAT addresses (start, middle, near-end of the range) and pins they're all classified local. This was the RED that drove the implementation. 2. ``test_openai_compat_provider_outside_tailscale_cgnat_is_external`` — pins the boundary on both sides: addresses with second octet 0-63 and 128-255 stay external. Prevents future "treat all 100.x.x.x as local" overcorrection. Tests: 1388 total mempalace tests pass. 2 pre-existing environmental failures unrelated to this change (chromadb optional dep). Ruff check + format both clean. Backwards compatible: only widens the local-recognition set. Anything classified local before is still classified local; anything classified external before remains so unless it's specifically in the CGNAT range. Out of scope (tracked for future iteration based on real user feedback, not built speculatively): pre-init confirmation prompt before sending to external API, persistent ``private-only`` config flag that refuses external endpoints entirely, explicit cloud-provider name detection ("Using Anthropic's hosted API at ..." vs the current generic warning).	2026-04-26 15:31:44 -07:00
MSL	4400734867	feat(privacy): warn when LLM tier sends content to external API 4 files changed, 248 insertions, 0 deletions. 7 new tests (4 unit + 3 integration), all RED-first. Per @milla-jovovich's question to @igorls during PR #1221 review: users running `mempalace init` with an external LLM provider (Anthropic API, OpenAI hosted, etc.) need a clear, explicit warning that their folder content will be sent to the provider, that MemPalace doesn't control how the provider logs/retains/uses that data, and how to opt out. @igorls confirmed this should be a small follow-up PR scoped to the warning itself, before the v3.3.4 tag. This PR adds: - `_endpoint_is_local(url)` helper in `mempalace/llm_client.py` — URL-based heuristic returning True if the hostname is on the user's machine or private network. Covers: localhost, 127.0.0.1, ::1, hostnames ending in .local (mDNS/Bonjour), IPv4 RFC1918 ranges (10/8, 172.16-31/12, 192.168/16), and IPv6 unique-local addresses (fc00::/7). - `is_external_service` property on the `LLMProvider` base class. Subclasses inherit; the URL determines (no provider-specific hardcoding). This means: Ollama on localhost = local. LM Studio on LAN = local. Anthropic with default `https://api.anthropic.com` = external. A user proxying Anthropic through localhost (advanced setup) = local, no false-positive warning. - One-line warning print in `cmd_init` after successful provider acquisition, gated on `is_external_service`: ⚠ {provider_name} is an EXTERNAL API. Your folder content will be sent to the provider during init. MemPalace does not control how the provider logs, retains, or uses your data. Pass --no-llm to keep init fully local. The warning fires AFTER `LLM enabled: ...` so users see both that the LLM is engaged AND the privacy implications of where it lives, before Pass 0 / entity detection actually runs. LOCAL providers (Ollama on localhost, LM Studio on localhost or LAN, llama.cpp on localhost, vLLM on localhost) DO NOT trigger the warning — nothing leaves the user's machine/network in those configurations. TDD: 7 tests added across 2 files. Unit tests in `tests/test_llm_client.py` (4 tests, all RED-first): 1. test_ollama_provider_default_endpoint_is_local — pins that the default `http://localhost:11434` is classified local. 2. test_openai_compat_provider_localhost_endpoint_is_local — covers the LM Studio / llama.cpp / vLLM common case (localhost, 127.0.0.1, and 192.168.x LAN). 3. test_openai_compat_provider_cloud_endpoint_is_external — pins that pointing openai-compat at https://api.openai.com (or any non-local URL) classifies as external. 4. test_anthropic_provider_default_endpoint_is_external — pins that AnthropicProvider's default endpoint is external (the dominant user-facing case for `--llm-provider anthropic`). Integration tests in `tests/test_corpus_origin_integration.py` (3 tests, RED-first; 1 was the critical RED — the other 2 passed by accident since nothing printed "EXTERNAL API" before this PR): 5. test_init_prints_privacy_warning_when_provider_is_external — captures stdout from cmd_init with a mocked external provider, asserts the warning text contains "EXTERNAL API" + "--no-llm" + language about MemPalace not controlling provider behavior. 6. test_init_no_privacy_warning_when_provider_is_local — same flow with a mocked local provider, asserts the warning text does NOT appear. 7. test_init_no_privacy_warning_with_no_llm_flag — pins the --no-llm path: no provider acquisition attempted, no warning fires. Tests: 1382 total mempalace tests pass. 2 pre-existing environmental failures unrelated to this change (chromadb optional dep). Ruff check + format both clean. Backwards compatible: `is_external_service` is a new property; existing callers don't reference it. The warning is a new print statement that fires only when an external endpoint is acquired. The `--no-llm` opt-out existed before this PR and continues to work identically. Out of scope for follow-up (deliberately not in this PR per Igor's "small PR" guidance): Tailscale CGNAT (100.64.0.0/10) treatment, pre-init confirmation prompt, persistent privacy-mode config flag, explicit cloud-provider name detection. Tracked for future iteration.	2026-04-26 14:43:20 -07:00
Igor Lins e Silva	df6c7d0dc3	feat(llm): pluggable provider abstraction for entity refinement Three providers cover the useful space while keeping the zero-API default: - `ollama` (default): local models via http://localhost:11434. Works fully offline. Tag-matching check accepts both `model` and `model:latest` forms. - `openai-compat`: any /v1/chat/completions endpoint. Covers OpenRouter, LM Studio, llama.cpp server, vLLM, Groq, Together, Fireworks, and most self-hosted frameworks. API key falls back to $OPENAI_API_KEY. Endpoint normalization is forgiving about trailing `/v1`. - `anthropic`: Messages API v2023-06-01. API key falls back to $ANTHROPIC_API_KEY. Concatenates multi-block text responses. JSON mode is normalized across providers — Ollama uses `format: "json"`, OpenAI-compat uses `response_format`, Anthropic uses prompt-level instruction. Callers request JSON once; this module handles the provider-specific plumbing. No external SDK dependency; stdlib `urllib` throughout. HTTP errors are wrapped into a single `LLMError` class so callers don't need to distinguish transport, auth, and parse failures at the call site. 26 tests, all with mocked HTTP — suite runs offline with no real provider required.	2026-04-24 00:46:43 -03:00

Author

SHA1

Message

Date

MSL

a0b7ba005d

feat(privacy): treat Tailscale CGNAT range (100.64.0.0/10) as local

2 files changed, 60 insertions, 0 deletions. 2 new tests (RED-first).

Follow-up to #1224's privacy warning. The URL-based heuristic in
``mempalace.llm_client._endpoint_is_local`` shipped without recognizing
Tailscale's CGNAT range (100.64.0.0/10), so a user running LM Studio,
Ollama, or any local LLM accessible via a Tailscale-assigned 100.x.x.x
address would currently get a wrong privacy warning — Tailscale
addresses are network-private (only reachable inside the user's
Tailnet) but they're not RFC1918, so the heuristic was treating them
as external.

This PR adds CGNAT recognition: when the hostname starts with ``100.``
AND the second octet is between 64 and 127 inclusive, it's classified
as local. Addresses in 100.x.x.x outside that range (i.e. second octet
< 64 or > 127) are regular allocated public space and remain external,
so a user pointing at a public 100.0.0.1 still gets the warning.

Concrete user impact:

  Before: ``mempalace init --llm-provider openai-compat --llm-endpoint http://100.100.50.50:1234``
          (LM Studio on Tailnet) → triggers privacy warning incorrectly.

  After:  same command → no warning. data stays inside the user's
          Tailnet, which is what the warning is supposed to protect against.

TDD: 2 tests added in ``tests/test_llm_client.py``, both RED-first.

1. ``test_openai_compat_provider_tailscale_cgnat_endpoint_is_local``
   — covers three Tailscale CGNAT addresses (start, middle, near-end
   of the range) and pins they're all classified local. This was the
   RED that drove the implementation.

2. ``test_openai_compat_provider_outside_tailscale_cgnat_is_external``
   — pins the boundary on both sides: addresses with second octet 0-63
   and 128-255 stay external. Prevents future "treat all 100.x.x.x as
   local" overcorrection.

Tests: 1388 total mempalace tests pass. 2 pre-existing environmental
failures unrelated to this change (chromadb optional dep). Ruff check
+ format both clean.

Backwards compatible: only widens the local-recognition set. Anything
classified local before is still classified local; anything classified
external before remains so unless it's specifically in the CGNAT range.

Out of scope (tracked for future iteration based on real user feedback,
not built speculatively): pre-init confirmation prompt before sending
to external API, persistent ``private-only`` config flag that refuses
external endpoints entirely, explicit cloud-provider name detection
("Using Anthropic's hosted API at ..." vs the current generic warning).

2026-04-26 15:31:44 -07:00

MSL

4400734867

feat(privacy): warn when LLM tier sends content to external API

4 files changed, 248 insertions, 0 deletions. 7 new tests (4 unit + 3 integration), all RED-first.

Per @milla-jovovich's question to @igorls during PR #1221 review: users
running `mempalace init` with an external LLM provider (Anthropic API,
OpenAI hosted, etc.) need a clear, explicit warning that their folder
content will be sent to the provider, that MemPalace doesn't control
how the provider logs/retains/uses that data, and how to opt out.
@igorls confirmed this should be a small follow-up PR scoped to the
warning itself, before the v3.3.4 tag.

This PR adds:

- `_endpoint_is_local(url)` helper in `mempalace/llm_client.py` —
  URL-based heuristic returning True if the hostname is on the user's
  machine or private network. Covers: localhost, 127.0.0.1, ::1,
  hostnames ending in .local (mDNS/Bonjour), IPv4 RFC1918 ranges
  (10/8, 172.16-31/12, 192.168/16), and IPv6 unique-local addresses
  (fc00::/7).

- `is_external_service` property on the `LLMProvider` base class.
  Subclasses inherit; the URL determines (no provider-specific
  hardcoding). This means: Ollama on localhost = local. LM Studio on
  LAN = local. Anthropic with default `https://api.anthropic.com` =
  external. A user proxying Anthropic through localhost (advanced
  setup) = local, no false-positive warning.

- One-line warning print in `cmd_init` after successful provider
  acquisition, gated on `is_external_service`:

      ⚠ {provider_name} is an EXTERNAL API. Your folder content will be
      sent to the provider during init. MemPalace does not control how
      the provider logs, retains, or uses your data. Pass --no-llm to
      keep init fully local.

  The warning fires AFTER `LLM enabled: ...` so users see both that
  the LLM is engaged AND the privacy implications of where it lives,
  before Pass 0 / entity detection actually runs.

LOCAL providers (Ollama on localhost, LM Studio on localhost or LAN,
llama.cpp on localhost, vLLM on localhost) DO NOT trigger the warning —
nothing leaves the user's machine/network in those configurations.

TDD: 7 tests added across 2 files.

Unit tests in `tests/test_llm_client.py` (4 tests, all RED-first):

1. test_ollama_provider_default_endpoint_is_local — pins that the
   default `http://localhost:11434` is classified local.
2. test_openai_compat_provider_localhost_endpoint_is_local — covers
   the LM Studio / llama.cpp / vLLM common case (localhost,
   127.0.0.1, and 192.168.x LAN).
3. test_openai_compat_provider_cloud_endpoint_is_external — pins
   that pointing openai-compat at https://api.openai.com (or any
   non-local URL) classifies as external.
4. test_anthropic_provider_default_endpoint_is_external — pins that
   AnthropicProvider's default endpoint is external (the dominant
   user-facing case for `--llm-provider anthropic`).

Integration tests in `tests/test_corpus_origin_integration.py` (3 tests,
RED-first; 1 was the critical RED — the other 2 passed by accident
since nothing printed "EXTERNAL API" before this PR):

5. test_init_prints_privacy_warning_when_provider_is_external —
   captures stdout from cmd_init with a mocked external provider,
   asserts the warning text contains "EXTERNAL API" + "--no-llm" +
   language about MemPalace not controlling provider behavior.
6. test_init_no_privacy_warning_when_provider_is_local — same flow
   with a mocked local provider, asserts the warning text does NOT
   appear.
7. test_init_no_privacy_warning_with_no_llm_flag — pins the --no-llm
   path: no provider acquisition attempted, no warning fires.

Tests: 1382 total mempalace tests pass. 2 pre-existing environmental
failures unrelated to this change (chromadb optional dep). Ruff check +
format both clean.

Backwards compatible: `is_external_service` is a new property; existing
callers don't reference it. The warning is a new print statement that
fires only when an external endpoint is acquired. The `--no-llm` opt-out
existed before this PR and continues to work identically.

Out of scope for follow-up (deliberately not in this PR per Igor's
"small PR" guidance): Tailscale CGNAT (100.64.0.0/10) treatment,
pre-init confirmation prompt, persistent privacy-mode config flag,
explicit cloud-provider name detection. Tracked for future iteration.

2026-04-26 14:43:20 -07:00

Igor Lins e Silva

df6c7d0dc3

feat(llm): pluggable provider abstraction for entity refinement

Three providers cover the useful space while keeping the zero-API
default:

- `ollama` (default): local models via http://localhost:11434. Works
  fully offline. Tag-matching check accepts both `model` and
  `model:latest` forms.
- `openai-compat`: any /v1/chat/completions endpoint. Covers
  OpenRouter, LM Studio, llama.cpp server, vLLM, Groq, Together,
  Fireworks, and most self-hosted frameworks. API key falls back to
  $OPENAI_API_KEY. Endpoint normalization is forgiving about trailing
  `/v1`.
- `anthropic`: Messages API v2023-06-01. API key falls back to
  $ANTHROPIC_API_KEY. Concatenates multi-block text responses.

JSON mode is normalized across providers — Ollama uses
`format: "json"`, OpenAI-compat uses `response_format`, Anthropic uses
prompt-level instruction. Callers request JSON once; this module
handles the provider-specific plumbing.

No external SDK dependency; stdlib `urllib` throughout. HTTP errors
are wrapped into a single `LLMError` class so callers don't need to
distinguish transport, auth, and parse failures at the call site.

26 tests, all with mocked HTTP — suite runs offline with no real
provider required.

2026-04-24 00:46:43 -03:00

3 Commits