Image input: send photos to pi (ask about + file into notes) #2

New issue

Closed

opened 2026-07-04 21:39:25 +08:00 by weiwen · 1 comment

weiwen commented

2026-07-04 21:39:25 +08:00

Owner

Let the Telegram bot accept photos and pass them to pi, covering two use cases:

Ask about an image (read): send a whiteboard/receipt/screenshot and ask a question about it.
File an image into notes (write): save the photo as an attachment and have pi create/update a markdown note referencing it.

Key finding — no pi blocker

The note's premise ("PiCommand::Prompt only carries message: String, no multimodal path") is stale as of pi 0.80.3:

RPC prompt command already accepts message: string plus optional images?: ImageContent[] (dist/modes/rpc/rpc-types.d.ts).
ImageContent = { type: "image", data: string /* base64 */, mimeType: string } (pi-ai/dist/types.d.ts:236).

No pi tools or @-file hack needed — images ride along as content on the existing prompt.

Scope

Add images: Vec<ImageContent> to PiCommand::Prompt in src/pi.rs (serde camelCase, skip_serializing_if = "Vec::is_empty"); struct { type: "image", data, mime_type }.
Telegram handler: on msg.photo(), pick the largest PhotoSize, getFile -> download bytes -> base64. Telegram photo is always JPEG, so mimeType = "image/jpeg". Use the photo caption as the message text; empty caption -> empty message, let pi decide.
Thread images through SessionManager::send_message -> PiProcess::send_message.
Do not litter the vault. Download the image to a temp path (not into notes_dir) and include that path in the prompt. pi looks at the image via ImageContent; if the user's intent is to file it, pi uses its own write/bash tools to copy the temp file into <notes_dir>/attachments/... and reference it from a note. For read-only asks pi never touches the vault. evie deletes the temp file after the turn completes (pi's promoted copy, if any, survives). (Design decision: option (b) — temp + pi-promotes.)

Constraints / notes

Requires a vision-capable model. If the configured provider/model is text-only, pi/the provider errors. Document this; consider a startup warning.
Write path: pi already has edit/write/bash tools (evie passes no --no-tools). Intent detection stays in pi (natural language), not Rust — no new command surface. The stale "read-only" framing was removed from README/CONTEXT/DESIGN.
v1: single photo per message; compressed photo type (not uncompressed document). Media groups / multiple images can be a follow-up.

Acceptance

Send a photo + caption question -> streamed answer references image content; vault unchanged, temp file gone afterwards.
Send a photo with a caption like save this to Recipes -> pi copies the image into attachments/ and a note is created/updated linking it.
Text-only messages unaffected.

Let the Telegram bot accept photos and pass them to `pi`, covering two use cases: - **Ask about an image** (read): send a whiteboard/receipt/screenshot and ask a question about it. - **File an image into notes** (write): save the photo as an attachment and have `pi` create/update a markdown note referencing it. ## Key finding — no pi blocker The note's premise ("`PiCommand::Prompt` only carries `message: String`, no multimodal path") is **stale as of pi 0.80.3**: - RPC `prompt` command already accepts `message: string` **plus** optional `images?: ImageContent[]` (`dist/modes/rpc/rpc-types.d.ts`). - `ImageContent` = `{ type: "image", data: string /* base64 */, mimeType: string }` (`pi-ai/dist/types.d.ts:236`). No pi tools or `@`-file hack needed — images ride along as content on the existing prompt. ## Scope 1. Add `images: Vec<ImageContent>` to `PiCommand::Prompt` in `src/pi.rs` (serde `camelCase`, `skip_serializing_if = "Vec::is_empty"`); struct `{ type: "image", data, mime_type }`. 2. Telegram handler: on `msg.photo()`, pick the largest `PhotoSize`, `getFile` -> download bytes -> base64. Telegram `photo` is always JPEG, so `mimeType = "image/jpeg"`. Use the photo **caption** as the message text; empty caption -> empty message, let pi decide. 3. Thread `images` through `SessionManager::send_message` -> `PiProcess::send_message`. 4. **Do not litter the vault.** Download the image to a **temp path** (not into `notes_dir`) and include that path in the prompt. pi looks at the image via `ImageContent`; if the user's intent is to file it, pi uses its own `write`/`bash` tools to copy the temp file into `<notes_dir>/attachments/...` and reference it from a note. For read-only asks pi never touches the vault. evie deletes the temp file after the turn completes (pi's promoted copy, if any, survives). (Design decision: option (b) — temp + pi-promotes.) ## Constraints / notes - **Requires a vision-capable model.** If the configured `provider`/`model` is text-only, pi/the provider errors. Document this; consider a startup warning. - **Write path:** `pi` already has `edit`/`write`/`bash` tools (evie passes no `--no-tools`). Intent detection stays in pi (natural language), not Rust — no new command surface. The stale "read-only" framing was removed from README/CONTEXT/DESIGN. - v1: single photo per message; compressed `photo` type (not uncompressed `document`). Media groups / multiple images can be a follow-up. ## Acceptance - Send a photo + caption question -> streamed answer references image content; vault unchanged, temp file gone afterwards. - Send a photo with a caption like `save this to Recipes` -> pi copies the image into `attachments/` and a note is created/updated linking it. - Text-only messages unaffected.

weiwen added the

ready-for-agent

label

2026-07-04 22:13:50 +08:00

weiwen commented

2026-07-04 22:27:55 +08:00

Author

Owner

Implemented image input via base64 ImageContent in pi's RPC protocol.

Changes:

Added public ImageContent struct (pi.rs) with jpeg()\ constructor; serialises to {type: "image", data: "", mimeType: "image/jpeg"}`.
Extended PiCommand::Prompt with images: Vec<ImageContent> (skip_serializing_if = "Vec::is_empty" so text-only turns are unchanged).
Threaded images through PiProcess::send_message and SessionManager::send_message; HTTP handler passes an empty vec.
Telegram handler now handles msg.photo(): picks the largest PhotoSize (by pixel area), downloads bytes via teloxide::net::Download, base64-encodes them for pi, and saves a tempfile::NamedTempFile that auto-deletes when the turn completes. The caption is used as the message text; the temp path is appended so pi can copy the file into the vault for the write case.
Added base64 = "0.22" and tempfile = "3" to Cargo.toml.
Added two serialisation tests; all 26 tests pass.

Commit: d28ee2c

Implemented image input via base64 ImageContent in pi's RPC protocol. Changes: - Added public `ImageContent` struct (pi.rs) with `jpeg()\ constructor; serialises to `{type: "image", data: "<base64>", mimeType: "image/jpeg"}`. - Extended `PiCommand::Prompt` with `images: Vec<ImageContent>` (`skip_serializing_if = "Vec::is_empty"` so text-only turns are unchanged). - Threaded images through `PiProcess::send_message` and `SessionManager::send_message`; HTTP handler passes an empty vec. - Telegram handler now handles `msg.photo()`: picks the largest PhotoSize (by pixel area), downloads bytes via `teloxide::net::Download`, base64-encodes them for pi, and saves a `tempfile::NamedTempFile` that auto-deletes when the turn completes. The caption is used as the message text; the temp path is appended so pi can copy the file into the vault for the write case. - Added `base64 = "0.22"` and `tempfile = "3"` to Cargo.toml. - Added two serialisation tests; all 26 tests pass. Commit: d28ee2c

weiwen closed this issue

2026-07-04 22:27:59 +08:00

weiwen referenced this issue from a commit

2026-07-04 23:08:33 +08:00

feat: send photos to pi via base64 ImageContent (closes #2)

weiwen referenced this issue from a commit

2026-07-23 01:53:45 +08:00

refactor: simplify code using gnhf

weiwen referenced this issue

2026-07-31 04:06:57 +08:00

feat(deploy): nix-built Docker image + compose for evie #86