diff --git a/README.md b/README.md index b5ecf09..cee8b0f 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,12 @@ Webhooks are auto-registered when you add a controller from the UI. **Tenant filtering:** hide building tenants (or any non-staff actor) from the attendance table with one click — events are still recorded so unfiltering -restores their full history. (05/28/26) +restores their full history. + +**Identity merging:** the same person on two controllers gets two UniFi UUIDs. +Merge those identities into a single "person" so the dashboard computes ONE +first-badge time per human, not per badge UUID — no more false LATE warnings +for staff who badge into multiple buildings before the cutoff. (05/28/26) --- @@ -177,6 +182,7 @@ Per-controller actions in the modal: | **Refresh** | Reload the table | | **Sync Users** | Pull latest users from every enabled controller | | **🚫 Filtered** | Open the filtered-tenants modal to review and unhide | +| **👥 People** | Manage merged identities and review auto-suggested merges | | **⚙ Controllers** | Add / manage controllers | | **Reset Day** | Delete all badge records for the selected date (respects the Controller filter — testing only) | @@ -191,11 +197,42 @@ Per-controller actions in the modal: | **Latest Badge In** | Most recent entry — shows *"— same"* if only one badge event | | **Actor ID** | First 8 characters of the UniFi user UUID | | **Status** | ON TIME (green) or LATE (red) based on first badge vs cutoff | -| **Actions** | **Hide** filters this tenant out of future views; **Unhide** restores them | +| **Actions** | **Hide** filters this person out of future views; **Merge** joins two badge identities so they count as one human | -> The same physical person on two different controllers will appear as two rows -> (different controllers issue different user UUIDs). They're distinguishable -> by the Source column. +> Once two identities are merged, the Source column shows a chip for every +> controller the person badged into that day, plus a "MERGED" pill so it's +> clear the row represents N UniFi UUIDs. + +--- + +## Merging identities across controllers + +UniFi issues a new UUID per controller, so the same person on Main Office and +Warehouse shows up as two rows by default — and worst case, a badge at 8:45 on +one and 9:15 on the other produces one ON TIME row plus one LATE row. Merging +fixes this: + +- Click **Merge** on any row. A picker shows other actors ordered by best + name match — pick one, confirm a display name, done. +- Click **👥 People** in the header to open the people manager. The top of + that modal shows **Suggested merges** — pairs of `(controller, actor)` rows + with matching full names across different controllers. One click confirms + each suggestion (no auto-apply; you're always in the loop). +- The same modal lists every merged person below the suggestions, with + per-person actions: **Rename**, **Split off** (remove one identity from the + group), **Dissolve** (break the whole group up — past badge events are + preserved, the identities just become standalone rows again). + +Once merged, the attendance table: + +- Shows **one row** per person, with MIN/MAX badge times computed across all + their identities — first badge wins ON TIME / LATE. +- Renders **one source chip per controller** they badged into that day. +- **Hide** on a merged row filters the *person*, so all their identities go + with them. Splitting an identity off later returns it unfiltered. + +The merge data lives in two tables (`persons`, `person_members`); user-cache +syncs from UniFi never touch them. --- @@ -246,6 +283,13 @@ reverse proxy with auth in front of it. | `GET` | `/api/first-badge-status` | `date`, `cutoff`, `controller_id?`, `include_filtered?` | Returns first + latest badge per user (filtered tenants hidden unless `include_filtered=1`) | | `GET` | `/api/users` | `controller_id?`, `filtered?` | List cached actors with their filtered flag | | `PATCH` | `/api/users//` | `filtered` (bool) | Hide / unhide an actor from the attendance table | +| `GET` | `/api/persons` | — | List merged people with their members | +| `POST` | `/api/persons` | `display_name`, `members[]` | Create a merged person from 1+ `(controller_id, actor_id)` members | +| `PATCH` | `/api/persons/` | `display_name?`, `filtered?` | Rename or hide/unhide a merged person | +| `DELETE` | `/api/persons/` | — | Dissolve a merged person (members become standalone) | +| `POST` | `/api/persons//members` | `controller_id`, `actor_id` | Add another identity to an existing person | +| `DELETE` | `/api/persons//members//` | — | Split one identity off; dissolves the person if it was the last member | +| `GET` | `/api/persons/suggestions` | — | Exact full-name matches across controllers, excluding already-merged actors | | `GET` | `/api/controllers` | — | List configured controllers | | `POST` | `/api/controllers` | `name`, `host`, `port`, `api_token` | Add a controller (also registers webhook) | | `PATCH` | `/api/controllers/` | `name?`, `enabled?` | Rename or enable/disable a controller | @@ -269,8 +313,10 @@ reverse proxy with auth in front of it. | Webhook URL stored in controller points to the wrong address | Browser's origin isn't reachable from the controller | Set `DASHBOARD_BASE_URL` in `.env`, remove + re-add the controller | | `Port 12445 connection refused` | Firewall blocking port | Add LAN IN firewall rule in UniFi Network (Step 1) | | Dashboard shows stale names after a user rename | Cache not refreshed | Click **Sync Users** or wait for the hourly auto-sync | -| A tenant I hid is still showing up | Same person exists on a second controller | Hide them on each controller — the filter is per `(controller, actor)` | +| A tenant I hid is still showing up | Same person exists on a second controller | Hide them on each controller, or merge their identities under the **👥 People** modal so one Hide covers both | | Filtered tenant doesn't appear when I tick "Show filtered" | They've never badged in on the selected date | Open the **🚫 Filtered** modal to confirm they're filtered | +| Same person showing twice with ON TIME + LATE | They badged into two controllers and the identities aren't merged | Click **Merge** on either row (or confirm the auto-suggestion under **👥 People**) | +| "actor already belongs to another person" when merging | The actor is already part of an existing merged person | Open **👥 People**, find that person, and add this identity to it (or split it off first) | --- diff --git a/app.py b/app.py index f13532d..5972fb5 100644 --- a/app.py +++ b/app.py @@ -104,6 +104,31 @@ def init_db(): "ALTER TABLE user_cache ADD COLUMN filtered INTEGER NOT NULL DEFAULT 0" ) + db.execute( + """ + CREATE TABLE IF NOT EXISTS persons ( + id TEXT PRIMARY KEY, + display_name TEXT NOT NULL, + filtered INTEGER NOT NULL DEFAULT 0, + created_at TEXT NOT NULL + ) + """ + ) + db.execute( + """ + CREATE TABLE IF NOT EXISTS person_members ( + person_id TEXT NOT NULL, + controller_id TEXT NOT NULL, + actor_id TEXT NOT NULL, + PRIMARY KEY (controller_id, actor_id), + FOREIGN KEY (person_id) REFERENCES persons(id) ON DELETE CASCADE + ) + """ + ) + db.execute( + "CREATE INDEX IF NOT EXISTS idx_person_members_person ON person_members(person_id)" + ) + # Seed a Default controller from env vars when the table is empty. existing = db.execute("SELECT COUNT(*) AS n FROM controllers").fetchone()["n"] default_id = None @@ -313,17 +338,23 @@ def first_badge_status(): sql = """ SELECT - b.actor_id, - b.controller_id, - c.name AS source, - MIN(b.ts) AS first_ts, - MAX(b.ts) AS latest_ts, + COALESCE(pm.person_id, b.controller_id || '|' || b.actor_id) AS group_key, + pm.person_id AS person_id, COALESCE( + p.display_name, u.full_name, 'Unknown (' || SUBSTR(b.actor_id,1,8) || '...)' ) AS name, - COALESCE(u.filtered, 0) AS filtered + COALESCE(p.filtered, u.filtered, 0) AS filtered, + MIN(b.ts) AS first_ts, + MAX(b.ts) AS latest_ts, + GROUP_CONCAT(DISTINCT c.name) AS sources, + GROUP_CONCAT(DISTINCT b.controller_id) AS controller_ids, + GROUP_CONCAT(DISTINCT b.actor_id) AS actor_ids FROM badge_events b + LEFT JOIN person_members pm + ON pm.controller_id = b.controller_id AND pm.actor_id = b.actor_id + LEFT JOIN persons p ON p.id = pm.person_id LEFT JOIN user_cache u ON u.actor_id = b.actor_id AND u.controller_id = b.controller_id LEFT JOIN controllers c ON c.id = b.controller_id @@ -334,8 +365,8 @@ def first_badge_status(): sql += " AND b.controller_id = ?" params.append(controller_filter) if not include_filtered: - sql += " AND COALESCE(u.filtered, 0) = 0" - sql += " GROUP BY b.actor_id, b.controller_id ORDER BY first_ts ASC" + sql += " AND COALESCE(p.filtered, u.filtered, 0) = 0" + sql += " GROUP BY group_key ORDER BY first_ts ASC" with get_db() as db: rows = db.execute(sql, params).fetchall() @@ -344,11 +375,22 @@ def first_badge_status(): for r in rows: first = r["first_ts"] latest = r["latest_ts"] + sources_csv = r["sources"] or "" + ctrl_ids_csv = r["controller_ids"] or "" + actor_ids_csv = r["actor_ids"] or "" + sources_list = [s for s in sources_csv.split(",") if s] + ctrl_ids_list = [s for s in ctrl_ids_csv.split(",") if s] + actor_ids_list = [s for s in actor_ids_csv.split(",") if s] result.append({ - "actor_id": r["actor_id"], - "controller_id": r["controller_id"], + "person_id": r["person_id"], + "actor_id": actor_ids_list[0] if actor_ids_list else None, + "controller_id": ctrl_ids_list[0] if ctrl_ids_list else None, + "actor_ids": actor_ids_list, + "controller_ids": ctrl_ids_list, "name": r["name"], - "source": r["source"] or "—", + "source": sources_list[0] if sources_list else "—", + "sources": sources_list or ["—"], + "merged": bool(r["person_id"]), "first_ts": first, "latest_ts": latest if latest != first else None, "status": "ON TIME" if first <= cutoff_end else "LATE", @@ -678,6 +720,231 @@ def update_user(controller_id, actor_id): }) +# --------------------------------------------------------------------------- +# Persons (merge identities across controllers) +# --------------------------------------------------------------------------- +def _person_to_dict(db, person_row): + members = db.execute( + """ + SELECT pm.controller_id, pm.actor_id, + c.name AS controller_name, + u.full_name + FROM person_members pm + LEFT JOIN controllers c ON c.id = pm.controller_id + LEFT JOIN user_cache u + ON u.controller_id = pm.controller_id AND u.actor_id = pm.actor_id + WHERE pm.person_id = ? + ORDER BY c.name, u.full_name + """, + (person_row["id"],), + ).fetchall() + return { + "id": person_row["id"], + "display_name": person_row["display_name"], + "filtered": bool(person_row["filtered"]), + "created_at": person_row["created_at"], + "members": [ + { + "controller_id": m["controller_id"], + "controller_name": m["controller_name"] or "—", + "actor_id": m["actor_id"], + "full_name": m["full_name"] or f"User {m['actor_id'][:8]}", + } + for m in members + ], + } + + +@app.route("/api/persons", methods=["GET"]) +def list_persons(): + with get_db() as db: + rows = db.execute("SELECT * FROM persons ORDER BY display_name").fetchall() + return jsonify([_person_to_dict(db, r) for r in rows]) + + +@app.route("/api/persons", methods=["POST"]) +def create_person(): + body = request.get_json(silent=True) or {} + name = (body.get("display_name") or "").strip() + members = body.get("members") or [] + if not name: + return jsonify({"error": "display_name is required"}), 400 + if not isinstance(members, list) or len(members) < 1: + return jsonify({"error": "at least one member required"}), 400 + + cleaned = [] + for m in members: + cid = (m.get("controller_id") or "").strip() + aid = (m.get("actor_id") or "").strip() + if not cid or not aid: + return jsonify({"error": "each member needs controller_id and actor_id"}), 400 + cleaned.append((cid, aid)) + + person_id = str(uuid.uuid4()) + now_iso = datetime.now(timezone.utc).isoformat() + + with get_db() as db: + existing = db.execute( + f""" + SELECT controller_id, actor_id FROM person_members + WHERE (controller_id, actor_id) IN ({",".join(["(?,?)"] * len(cleaned))}) + """, + [v for pair in cleaned for v in pair], + ).fetchall() + if existing: + conflicts = [f"{r['controller_id']}/{r['actor_id'][:8]}" for r in existing] + return jsonify({ + "error": "some members already belong to another person", + "conflicts": conflicts, + }), 409 + + db.execute( + "INSERT INTO persons (id, display_name, filtered, created_at) VALUES (?, ?, 0, ?)", + (person_id, name, now_iso), + ) + db.executemany( + "INSERT INTO person_members (person_id, controller_id, actor_id) VALUES (?, ?, ?)", + [(person_id, cid, aid) for cid, aid in cleaned], + ) + db.commit() + row = db.execute("SELECT * FROM persons WHERE id = ?", (person_id,)).fetchone() + return jsonify(_person_to_dict(db, row)), 201 + + +@app.route("/api/persons/", methods=["PATCH"]) +def update_person(person_id): + body = request.get_json(silent=True) or {} + fields, values = [], [] + if "display_name" in body: + name = (body["display_name"] or "").strip() + if not name: + return jsonify({"error": "display_name cannot be empty"}), 400 + fields.append("display_name = ?"); values.append(name) + if "filtered" in body: + fields.append("filtered = ?"); values.append(1 if body["filtered"] else 0) + if not fields: + return jsonify({"error": "no updatable fields provided"}), 400 + values.append(person_id) + + with get_db() as db: + cur = db.execute( + f"UPDATE persons SET {', '.join(fields)} WHERE id = ?", values + ) + db.commit() + if cur.rowcount == 0: + return jsonify({"error": "not found"}), 404 + row = db.execute("SELECT * FROM persons WHERE id = ?", (person_id,)).fetchone() + return jsonify(_person_to_dict(db, row)) + + +@app.route("/api/persons/", methods=["DELETE"]) +def delete_person(person_id): + with get_db() as db: + cur = db.execute("DELETE FROM persons WHERE id = ?", (person_id,)) + db.commit() + if cur.rowcount == 0: + return jsonify({"error": "not found"}), 404 + return jsonify({"status": "ok"}) + + +@app.route("/api/persons//members", methods=["POST"]) +def add_person_member(person_id): + body = request.get_json(silent=True) or {} + cid = (body.get("controller_id") or "").strip() + aid = (body.get("actor_id") or "").strip() + if not cid or not aid: + return jsonify({"error": "controller_id and actor_id are required"}), 400 + + with get_db() as db: + if not db.execute("SELECT 1 FROM persons WHERE id = ?", (person_id,)).fetchone(): + return jsonify({"error": "person not found"}), 404 + existing = db.execute( + "SELECT person_id FROM person_members WHERE controller_id = ? AND actor_id = ?", + (cid, aid), + ).fetchone() + if existing and existing["person_id"] != person_id: + return jsonify({"error": "actor already belongs to another person"}), 409 + db.execute( + """ + INSERT OR IGNORE INTO person_members (person_id, controller_id, actor_id) + VALUES (?, ?, ?) + """, + (person_id, cid, aid), + ) + db.commit() + row = db.execute("SELECT * FROM persons WHERE id = ?", (person_id,)).fetchone() + return jsonify(_person_to_dict(db, row)) + + +@app.route("/api/persons//members//", methods=["DELETE"]) +def remove_person_member(person_id, controller_id, actor_id): + with get_db() as db: + cur = db.execute( + """ + DELETE FROM person_members + WHERE person_id = ? AND controller_id = ? AND actor_id = ? + """, + (person_id, controller_id, actor_id), + ) + if cur.rowcount == 0: + return jsonify({"error": "member not found"}), 404 + # If that was the last member, drop the person too. + remaining = db.execute( + "SELECT COUNT(*) AS n FROM person_members WHERE person_id = ?", (person_id,) + ).fetchone()["n"] + if remaining == 0: + db.execute("DELETE FROM persons WHERE id = ?", (person_id,)) + db.commit() + return jsonify({"status": "ok", "person_dissolved": True}) + db.commit() + row = db.execute("SELECT * FROM persons WHERE id = ?", (person_id,)).fetchone() + return jsonify(_person_to_dict(db, row)) + + +@app.route("/api/persons/suggestions", methods=["GET"]) +def person_suggestions(): + """Return groups of (controller, actor) rows that share an exact full_name + across different controllers and are NOT already part of a person.""" + with get_db() as db: + rows = db.execute( + """ + SELECT u.controller_id, u.actor_id, u.full_name, + c.name AS controller_name + FROM user_cache u + LEFT JOIN controllers c ON c.id = u.controller_id + LEFT JOIN person_members pm + ON pm.controller_id = u.controller_id AND pm.actor_id = u.actor_id + WHERE pm.person_id IS NULL + AND u.full_name IS NOT NULL + AND TRIM(u.full_name) <> '' + AND u.full_name NOT LIKE 'User %' + ORDER BY LOWER(u.full_name), c.name + """ + ).fetchall() + + groups = {} + for r in rows: + key = r["full_name"].strip().lower() + groups.setdefault(key, []).append({ + "controller_id": r["controller_id"], + "controller_name": r["controller_name"] or "—", + "actor_id": r["actor_id"], + "full_name": r["full_name"], + }) + + suggestions = [] + for key, members in groups.items(): + # Only suggest when the same name spans at least 2 distinct controllers + controllers = {m["controller_id"] for m in members} + if len(controllers) >= 2: + suggestions.append({ + "display_name": members[0]["full_name"], + "members": members, + }) + suggestions.sort(key=lambda s: s["display_name"].lower()) + return jsonify(suggestions) + + # --------------------------------------------------------------------------- # Misc admin # --------------------------------------------------------------------------- diff --git a/obsidian-rest-api-issue-draft.md b/obsidian-rest-api-issue-draft.md new file mode 100644 index 0000000..ea7a25c --- /dev/null +++ b/obsidian-rest-api-issue-draft.md @@ -0,0 +1,114 @@ +# Heading resolver broken: `Target-Type: heading` PATCH and `GET /heading/...` both fail on any valid heading (4.1.1) + +## Summary + +In Obsidian Local REST API **4.1.1**, two endpoints that both rely on heading resolution always fail, even on a freshly written file with trivially-valid markdown headings: + +- `PATCH /vault/` with `Target-Type: heading` → HTTP 400 `invalid-target` +- `GET /vault//heading/` → HTTP 404 `Not Found` + +`Target-Type: frontmatter` PATCH on the same file works fine, so the PATCH route itself is healthy. The bug is isolated to the heading-resolution code path shared by these two endpoints. + +Also reproduced in **3.6** before upgrading; the 3.6 → 4.1.1 upgrade did not change the behavior. + +## Minimal reproducer + +The setup writes a clean file with LF-only line endings, then exercises both heading endpoints and one frontmatter endpoint as a control. + +```bash +HOST="https://YOUR-HOST:27124" +KEY="YOUR_API_KEY" + +# 1. PUT a fresh, simple file (HTTP 204 expected) +cat > /tmp/repro.md <<'EOF' +--- +type: scratch +--- + +# Hello + +## Section A + +Content under A. + +## Section B + +Content under B. +EOF + +curl -sk -w "PUT: %{http_code}\n" -X PUT \ + -H "Authorization: Bearer $KEY" \ + -H "Content-Type: text/markdown" \ + --data-binary @/tmp/repro.md \ + "$HOST/vault/scratch/repro.md" + +# 2. PATCH append under heading "Section A" — expected 200, actual 400 invalid-target +echo "appended" > /tmp/p.md +curl -sk -w "\nPATCH heading: %{http_code}\n" -X PATCH \ + -H "Authorization: Bearer $KEY" \ + -H "Operation: append" \ + -H "Target-Type: heading" \ + -H "Target: Section A" \ + -H "Content-Type: text/markdown" \ + --data-binary @/tmp/p.md \ + "$HOST/vault/scratch/repro.md" + +# 3. GET the heading's content via URL path — expected 200, actual 404 +curl -sk -w "\nGET heading: %{http_code}\n" \ + -H "Authorization: Bearer $KEY" \ + "$HOST/vault/scratch/repro.md/heading/Section%20A" + +# 4. CONTROL: PATCH frontmatter — works (HTTP 200) +curl -sk -w "\nPATCH frontmatter: %{http_code}\n" -X PATCH \ + -H "Authorization: Bearer $KEY" \ + -H "Operation: replace" \ + -H "Target-Type: frontmatter" \ + -H "Target: type" \ + -H "Content-Type: application/json" \ + --data '"updated"' \ + "$HOST/vault/scratch/repro.md" +``` + +## Actual output + +``` +PUT: 204 + +{"message":"The patch you provided could not be applied to the target content.\ninvalid-target","errorCode":40080} +PATCH heading: 400 + +{"message":"Not Found","errorCode":40400} +GET heading: 404 + +PATCH frontmatter: 200 +``` + +## Variations tried (all fail identically with `invalid-target` / 404) + +- Single-word heading (`Target: Section`) and multi-word (`Target: Section A`) +- URL-encoded header value (`Target: Section%20A`) +- `Operation: append`, `prepend`, and `replace` +- Both freshly-PUT files (LF endings) and files originally written by Obsidian +- `Target-Type: block` also returns `invalid-target` — possibly the same resolver + +## What I ruled out + +- **CRLF / BOM in source file** — fresh file was written by `curl --data-binary` with LF-only content. +- **Spaces in heading text** — single-word headings fail the same way. +- **URL encoding of `Target:` header value** — raw and percent-encoded both fail. +- **PATCH route generally broken** — `Target-Type: frontmatter` succeeds on the same file. +- **Version regression in 4.1.1** — same behavior in 3.6 before upgrading. + +## Environment + +- Obsidian Local REST API: **4.1.1** (also reproduced in 3.6) +- Obsidian: +- OS hosting Obsidian: +- Client: `curl` over HTTPS to `:27124` with `-k` (self-signed cert) +- Vault is on local NTFS / ext4 / APFS (fill in) + +## Suspected cause + +The heading resolver appears to be returning "not found" for every valid heading. Since `GET /vault//heading/` and the `Target-Type: heading` PATCH variant both fail in lockstep (and frontmatter resolution works fine on the same request path), the most likely location is whichever helper enumerates headings within a parsed file's AST — perhaps it's getting an empty list, or matching on a wrong field. + +Happy to bisect or add logging if you can point me at the relevant file. diff --git a/static/index.html b/static/index.html index 017d43a..a1f07be 100644 --- a/static/index.html +++ b/static/index.html @@ -121,6 +121,43 @@ border-color: rgba(243,156,18,0.6); background: radial-gradient(circle at top, rgba(243,156,18,0.18), rgba(2,2,4,0.95)); } + .people-btn { + border-color: rgba(100,180,255,0.6); + background: radial-gradient(circle at top, rgba(100,180,255,0.18), rgba(2,2,4,0.95)); + } + + .source-list { display: inline-flex; flex-wrap: wrap; gap: 4px; } + .merged-pill { + display: inline-block; margin-left: 8px; padding: 1px 8px; + font-size: 0.65rem; letter-spacing: 0.1em; border-radius: 999px; + color: var(--blue); border: 1px solid rgba(100,180,255,0.5); + vertical-align: middle; + } + .row-action-btn.merge { border-color: rgba(100,180,255,0.55); color: var(--blue); } + .row-action-group { display: inline-flex; gap: 6px; justify-content: center; } + + .picker-list { display: flex; flex-direction: column; gap: 6px; max-height: 320px; overflow-y: auto; margin-bottom: 12px; } + .picker-row { + display: grid; grid-template-columns: 1fr auto; gap: 10px; + padding: 8px 12px; background: rgba(255,255,255,0.02); + border: 1px solid var(--border); border-radius: 8px; align-items: center; + } + .picker-row .ctrl-name { font-size: 0.88rem; } + .picker-row .ctrl-sub { font-size: 0.74rem; } + .picker-search { width: 100%; margin-bottom: 10px; } + + .suggestion-row { + padding: 10px 14px; background: rgba(100,180,255,0.05); + border: 1px solid rgba(100,180,255,0.25); border-radius: 10px; + display: grid; grid-template-columns: 1fr auto; gap: 10px; + align-items: center; margin-bottom: 8px; + } + .suggestion-row .ctrl-name { font-weight: 600; } + .suggestion-row .ctrl-sub { font-size: 0.76rem; color: var(--muted); } + .section-label { + font-size: 0.72rem; letter-spacing: 0.14em; text-transform: uppercase; + color: var(--muted); margin: 14px 0 8px; + } .modal-overlay { display: none; position: fixed; inset: 0; background: rgba(0,0,0,0.75); backdrop-filter: blur(4px); z-index: 100; align-items: center; justify-content: center; } .modal-overlay.open { display: flex; } @@ -211,6 +248,7 @@ + @@ -299,6 +337,56 @@ + + + + + +