48 KiB
48 KiB
Officer Mobile App Plan
This document is written by Claude for Claude. It contains the full plan for native mobile apps (Kotlin/Android + Swift/iOS) for the Officer platform, based on a thorough study of the monorepo. When resuming this work, read this document first, then explain the plan to the user before starting implementation.
Table of Contents
- Current Web App Summary
- What to Include vs Exclude
- Architecture Overview
- Authentication
- Site Map
- Screen-by-Screen Breakdown
- Tablet Adaptations
- Technical Challenges
- API Integration Notes
- Shared Design Language
- Implementation Phases
1. Current Web App Summary
Officer is an AI-assisted personal backend for life management. The web app (officer-web) is a React 19 SPA talking to a Hono API server (port 5000). Key features:
| Feature | Description |
|---|---|
| Chat | Multi-provider AI chat (Claude, OpenCode, Pi-Mono) via WebSockets. Sessions persisted as JSON files. Supports images, file attachments, model selection, working directory context. |
| File Browser | Full file manager with upload/download, rename, copy/move, search, archive extraction, git clone, yt-dlp download, OCR, TTS, transcription, audio extraction, video transcoding. Multiple "roots" (home, ~, officer.dev). |
| Workspaces | Multi-panel layouts combining chat, files, terminal, code editor. Panels can be split horizontally/vertically. State persisted per workspace. |
| Terminal | Sandboxed Docker terminals and host terminals (Super Admin). WebSocket-based PTY. |
| Code Editor | Text editor with syntax highlighting (PrismJS). |
| Dashboard | Widget panel (clock, weather, pomodoro, daily goals, quick notes) + workspace launcher + chat launcher. |
| Automation | Builder for workflows, pipelines, processes, services, cron jobs, tasks, skills. |
| Tasks | File-based task definitions (TASK.md) that can be triggered on files/directories. Native/global/user scopes. |
| Skills | AI skill library (SKILL.md files) injected into chat system prompts. |
| Settings | User profile, system settings, server resources, AI harness configuration. |
| Auth | Email/password + passkey (WebAuthn). JWT tokens. Role-based access (Member, Admin, Owner, Super Admin). |
The API is at /api/* with JWT auth on all protected routes. WebSockets for chat (/api/harness/*/ws) and terminal (/api/terminal/ws).
2. What to Include vs Exclude
Include (core mobile experience)
| Feature | Rationale |
|---|---|
| Chat | The #1 use case. Having an AI assistant in your pocket is the killer feature. Voice input makes it even better on mobile. |
| File Browser | Essential for viewing/managing files. Mobile adds camera upload, share sheet integration. Simplified but fully functional. |
| Dashboard / Widgets | Perfect for mobile - quick glance at clock, weather, goals, notes. Widget-style info is native to mobile platforms. |
| Notifications | Mobile-exclusive feature. Push notifications for chat completions, task results, long downloads finishing. |
| Settings | Profile, preferences. Subset of web settings. |
| Session History | Browse and resume past chat sessions. |
| Quick Actions | Share sheet integration (share URL/text/image to Officer for chat or file upload). |
Include (reduced scope)
| Feature | What changes |
|---|---|
| File Viewer | View files (text, images, PDFs, audio, video) but no full code editor. Read-only text with syntax highlighting. |
| Tasks | View and trigger tasks, see logs. No task creation/editing (too complex for mobile). |
| Skills | Browse skill library. No editing. |
| Workspaces | View workspace list, open a workspace (but as a single-panel experience, not multi-panel). |
Exclude (not suitable for phone)
| Feature | Rationale |
|---|---|
| Terminal | Typing shell commands on a phone keyboard is miserable. The sandboxed Docker terminal is a power-user web feature. |
| Code Editor (write mode) | Writing code on a phone is impractical. Read-only viewing with syntax highlighting is fine. |
| Multi-panel workspaces | Phone screens can't meaningfully split into panels. Tablet gets a simplified version (see section 7). |
| Automation Builder | Complex drag-and-drop/form-heavy workflow builder. Keep this web-only. |
| Server/Resource Management | Admin infrastructure management (installing databases, configuring services). Web-only. |
| System Settings | Server configuration, AI harness setup, plugin management. Web-only. |
Mobile-Exclusive Features (not on web)
| Feature | Description |
|---|---|
| Push Notifications | FCM (Android) / APNs (iOS) for chat completions, task results, downloads. |
| Voice Input | Native speech-to-text for chat messages. Much better than web speech API. |
| Camera Integration | Take photo and send to chat or upload to file browser directly. |
| Share Sheet | Receive shared content (URLs, text, images) from other apps. |
| Biometric Auth | Fingerprint/Face ID as alternative to passkey/password. |
| Home Screen Widgets | Android widgets / iOS widgets for quick notes, pomodoro, weather. |
| Offline Queue | Queue chat messages when offline, send when reconnected. |
| Background Downloads | File downloads and yt-dlp downloads continue in background. |
3. Architecture Overview
Platform Strategy
Two fully native apps sharing no code, but sharing identical:
- API contract (same REST + WebSocket endpoints)
- UX flows and information architecture
- Design language (adapted to each platform's conventions)
┌──────────────┐ ┌──────────────┐
│ Android App │ │ iOS App │
│ (Kotlin) │ │ (Swift) │
│ │ │ │
│ Jetpack │ │ SwiftUI │
│ Compose │ │ │
│ Material 3 │ │ Native iOS │
│ │ │ Components │
└──────┬───────┘ └──────┬───────┘
│ │
└────────┬───────────┘
│
┌──────┴──────┐
│ Officer API │
│ (Hono) │
│ port 5000 │
└─────────────┘
Android Stack
| Layer | Technology |
|---|---|
| UI | Jetpack Compose + Material 3 |
| Navigation | Compose Navigation |
| Networking | Ktor Client (HTTP + WebSocket) |
| State | ViewModel + StateFlow |
| DI | Hilt |
| Storage | DataStore (preferences), Room (offline cache) |
| Images | Coil |
| Push | Firebase Cloud Messaging |
| Auth | BiometricPrompt API |
| Background | WorkManager |
iOS Stack
| Layer | Technology |
|---|---|
| UI | SwiftUI |
| Navigation | NavigationStack (iOS 16+) |
| Networking | URLSession + native WebSocket (URLSessionWebSocketTask) |
| State | @Observable (iOS 17+) / ObservableObject |
| DI | Swift native (Environment, protocol-based) |
| Storage | UserDefaults, SwiftData (offline cache) |
| Images | AsyncImage + custom cache |
| Push | APNs |
| Auth | LocalAuthentication (Face ID / Touch ID) |
| Background | BGTaskScheduler |
Minimum OS Versions
- Android: API 26 (Android 8.0) - covers 95%+ of devices, gives us WorkManager, BiometricPrompt, etc.
- iOS: iOS 16 - gives us NavigationStack, modern SwiftUI features. iOS 17 for @Observable if we want it.
4. Authentication
Flow
App Launch
│
├─ Has stored JWT? ──Yes──► Validate token (GET /api/auth/me)
│ │
│ Valid? ──Yes──► Home Screen
│ │
│ No ──► Login Screen
│
└─ No ──► Login Screen
│
├─ Email/Password
│ POST /api/auth/signin
│ ► Store JWT securely
│ ► Home Screen
│
├─ Passkey (WebAuthn)
│ ► Platform passkey flow
│ ► Store JWT securely
│
└─ Biometric Unlock (returning user)
► Decrypt stored JWT
► Validate & proceed
Token Storage
- Android: EncryptedSharedPreferences (AndroidX Security)
- iOS: Keychain Services
Biometric Integration
On first successful login, offer to enable biometric unlock:
- User logs in with email/password or passkey
- App prompts: "Enable Face ID / Fingerprint for quick access?"
- If yes: encrypt JWT with biometric-bound key, store in Keychain/Keystore
- On next launch: biometric prompt → decrypt JWT → validate → proceed
Server URL Configuration
The mobile app needs to know where the Officer API lives. First-launch flow:
- "Enter your Officer server URL" input
- Validate by hitting
GET /api/server-settings/plugins(public endpoint) - Store URL in app preferences
- Proceed to login
This is important because Officer is self-hosted - there's no central server.
5. Site Map
App
├── Login
│ ├── Server URL Setup (first launch)
│ ├── Email/Password
│ ├── Passkey
│ └── Biometric Unlock
│
├── Home (Tab 1)
│ ├── Widget Dashboard
│ │ ├── Clock
│ │ ├── Weather
│ │ ├── Pomodoro Timer
│ │ ├── Daily Goals
│ │ └── Quick Notes
│ └── Workspace Quick Access
│
├── Chat (Tab 2)
│ ├── Session List
│ │ ├── Active Sessions
│ │ └── Archived Sessions
│ ├── New Chat
│ └── Chat Session
│ ├── Message Thread
│ ├── Model Selector
│ ├── Image Attachment
│ ├── Voice Input
│ └── Tool Use Display
│
├── Files (Tab 3)
│ ├── Directory Browser
│ │ ├── Grid/List View
│ │ ├── Search
│ │ ├── Sort/Filter
│ │ └── Context Actions
│ │ ├── Open/Preview
│ │ ├── Share
│ │ ├── Rename
│ │ ├── Copy/Move
│ │ ├── Delete
│ │ ├── Download to Device
│ │ └── Upload (camera, gallery, files)
│ └── File Viewer
│ ├── Text (syntax highlighted, read-only)
│ ├── Image
│ ├── PDF
│ ├── Audio Player
│ ├── Video Player
│ └── AI Actions (OCR, TTS, Transcribe)
│
├── Activity (Tab 4)
│ ├── Task List
│ │ ├── Native Tasks
│ │ ├── Global Tasks
│ │ └── User Tasks
│ ├── Task Logs
│ └── Running Processes
│
└── Settings (Tab 5 / Profile)
├── Profile
│ ├── Name, Username, Avatar
│ ├── Password Change
│ └── Passkey Management
├── App Preferences
│ ├── Theme (Light/Dark/System)
│ ├── Notification Settings
│ ├── Default Chat Model
│ ├── Biometric Lock
│ └── Server URL
└── About / Logout
Navigation Pattern
Bottom Tab Bar with 4 primary tabs + profile access:
┌─────────────────────────────────────┐
│ │
│ [Screen Content] │
│ │
├─────┬─────┬─────┬─────┬────────────┤
│ Home│ Chat│Files│ Act.│ Settings │
│ 🏠 │ 💬 │ 📁 │ ⚡ │ ⚙️ │
└─────┴─────┴─────┴─────┴────────────┘
- Home: Dashboard with widgets
- Chat: Session list → chat view
- Files: File browser → file viewer
- Activity: Tasks, logs, processes
- Settings: Profile and preferences
On Android: Material 3 NavigationBar. On iOS: UITabBarController / SwiftUI TabView.
6. Screen-by-Screen Breakdown
6.1 Login Screen
┌─────────────────────────┐
│ │
│ [Officer Logo] │
│ │
│ ┌───────────────────┐ │
│ │ Server URL │ │ ← Only on first launch or
│ └───────────────────┘ │ tap "Change server"
│ │
│ ┌───────────────────┐ │
│ │ Email │ │
│ └───────────────────┘ │
│ ┌───────────────────┐ │
│ │ Password │ │
│ └───────────────────┘ │
│ │
│ [ Sign In ] │
│ │
│ ── or ── │
│ │
│ [ Sign in with Passkey ] │
│ │
│ Forgot password? │
│ │
└─────────────────────────┘
Behavior:
- Server URL field shown on first launch or when tapped
- Email/password fields with proper keyboard types
- "Sign in with Passkey" triggers platform WebAuthn flow
- On success: store JWT, navigate to Home
- On returning user with biometric enabled: show biometric prompt immediately, with "Use password instead" fallback
6.2 Home / Dashboard
┌─────────────────────────┐
│ Officer [Avatar]│
├─────────────────────────┤
│ │
│ ┌──────┐ ┌──────────┐│
│ │ 14:32│ │ Weather ││
│ │ Fri │ │ 22° ☀️ ││
│ └──────┘ └──────────┘│
│ │
│ ┌──────────────────┐ │
│ │ 🍅 Pomodoro │ │
│ │ 18:42 remaining │ │
│ │ [Start] [Reset] │ │
│ └──────────────────┘ │
│ │
│ ┌──────────────────┐ │
│ │ Daily Goals │ │
│ │ ☑ Exercise │ │
│ │ ☐ Read 30min │ │
│ │ ☐ Journal │ │
│ │ [+ Add goal] │ │
│ └──────────────────┘ │
│ │
│ ┌──────────────────┐ │
│ │ Quick Notes │ │
│ │ Tap to add... │ │
│ └──────────────────┘ │
│ │
│ Workspaces │
│ ┌────┐ ┌────┐ ┌────┐ │
│ │ WS1│ │ WS2│ │ WS3│ │
│ └────┘ └────┘ └────┘ │
│ │
└─────────────────────────┘
Behavior:
- Scrollable vertical layout
- Widgets are cards that can be reordered (long-press drag)
- Widget data syncs with web app via user settings API
- Workspace cards show name + icon, tap opens workspace (single-panel mode on phone)
- Pull-to-refresh updates widget data
- Avatar tap opens Settings
Widget Implementation:
- Clock: Local device time (no API needed)
- Weather: Can use device location + weather API, or sync from web app settings
- Pomodoro: Local timer with state synced to server (so it persists across devices)
- Daily Goals: Stored in user settings, synced
- Quick Notes: Stored in user settings, synced
6.3 Chat - Session List
┌─────────────────────────┐
│ Chat [+ New]│
├─────────────────────────┤
│ [🔍 Search sessions...] │
├─────────────────────────┤
│ │
│ Today │
│ ┌──────────────────┐ │
│ │ 🤖 Debug login │ │
│ │ Claude · 2h ago │ │
│ │ Last: "The issue │ │
│ │ was in the..." │ │
│ └──────────────────┘ │
│ ┌──────────────────┐ │
│ │ 🤖 Refactor API │ │
│ │ Pi-Mono · 5h ago │ │
│ │ Last: "I've upda- │ │
│ │ ted the router..." │ │
│ └──────────────────┘ │
│ │
│ Yesterday │
│ ┌──────────────────┐ │
│ │ 🤖 Recipe ideas │ │
│ │ Claude · 1d ago │ │
│ └──────────────────┘ │
│ │
│ Archived ▸ │
│ │
└─────────────────────────┘
Behavior:
- Sessions fetched from
GET /api/sessions?provider=all - Grouped by date (Today, Yesterday, This Week, Older)
- Each card shows: session name, provider icon, time, last message preview
- Swipe left to delete, swipe right to archive
- Long-press for context menu (rename, archive, delete)
- "Archived" section expandable
- Search filters sessions by name
- FAB or top-right button for new chat
Data Source:
GET /api/sessions?provider=claude(and opencode, pi-mono)- Sessions are JSON files on the server, API returns metadata
- Message preview from last assistant message
6.4 Chat - Conversation
┌─────────────────────────┐
│ ← Debug login [···] │
│ Claude 3.5 Sonnet ▾ │
├─────────────────────────┤
│ │
│ ┌─ You ────────────┐ │
│ │ The login page │ │
│ │ throws a 401... │ │
│ └──────────────────┘ │
│ │
│ ┌─ Claude ─────────┐ │
│ │ Looking at the │ │
│ │ auth middleware... │ │
│ │ │ │
│ │ ┌─ Tool Use ────┐ │ │
│ │ │ 📄 Read file │ │ │
│ │ │ auth/jwt.ts │ │ │
│ │ └──────────────┘ │ │
│ │ │ │
│ │ The issue is in │ │
│ │ the token verify │ │
│ │ ```ts │ │
│ │ if (!jti) ... │ │
│ │ ``` │ │
│ └──────────────────┘ │
│ │
├─────────────────────────┤
│ ┌───────────────┐ 📎 🎤│
│ │ Message... │ [Send]│
│ └───────────────┘ │
└─────────────────────────┘
Behavior:
- WebSocket connection to
/api/harness/{provider}/ws - Messages stream in real-time (assistant:partial → assistant:text)
- Tool use blocks shown as collapsible cards (collapsed by default)
- Code blocks with syntax highlighting and copy button
- Markdown rendering for assistant messages
- Model selector dropdown in the header (or a bottom sheet)
- Attachment button (📎) opens picker: Camera, Photo Library, Files
- Microphone button (🎤) for voice input (speech-to-text → send as text)
- Send button disabled while assistant is generating
- "Stop" button appears during generation
- Pull-down to load older messages (if session has many)
[···]menu: Rename session, View in files, Delete, Share
WebSocket Message Protocol:
Client sends:
{ type: 'chat', prompt, sessionId, model, images?, attachmentIds? }
{ type: 'resume', sessionId }
{ type: 'stop' }
Server sends:
{ type: 'session:init', sessionId, name }
{ type: 'assistant:partial', text }
{ type: 'assistant:text', text }
{ type: 'tool:use', name, input }
{ type: 'tool:result', result }
{ type: 'result', cost?, duration?, turns? }
{ type: 'error', message }
{ type: 'stopped' }
Offline Behavior:
- If offline when sending, queue message locally
- Show "Queued - will send when online" indicator
- On reconnect, send queued messages in order
- Previously loaded messages remain visible offline (cached in local DB)
Voice Input:
- Tap and hold microphone for continuous recording
- Or tap once to start, tap again to stop
- Platform native speech recognition
- Transcribed text placed in input field for review before sending
- User can edit before sending
6.5 Files - Directory Browser
┌─────────────────────────┐
│ ← Files [···] │
│ / > Documents > Work │
├─────────────────────────┤
│ [🔍 Search files...] │
├─────────────────────────┤
│ Sort: Name ▾ [≡] [⊞] │
├─────────────────────────┤
│ │
│ 📁 Projects → │
│ 📁 Reports → │
│ 📄 notes.md 3KB │
│ 🖼️ photo.jpg 1.2MB │
│ 🎵 song.mp3 4.5MB │
│ 📹 demo.mp4 45.2MB │
│ 📦 backup.zip 12.0MB │
│ │
│ [+ Upload] │
│ │
└─────────────────────────┘
Behavior:
- Breadcrumb navigation at top (scrollable horizontal)
- Tap folder to navigate in, tap file to open viewer
- Long-press for context menu:
- Open / Preview
- Share (native share sheet)
- Rename
- Copy / Move (enter selection mode)
- Delete
- Download to device
- AI actions (OCR, TTS, Transcribe - based on file type)
- Pull-to-refresh
- FAB or bottom button for upload (Camera, Photo Library, Files picker)
- Multi-select mode (long-press first item, then tap others)
- Swipe actions: left = delete, right = share
- Grid/List view toggle
- Sort by name/date/size
- Search with debounce (calls
/api/file-browser/search)
Root Selection:
- Dropdown or segmented control at top for root selection
- Only show non-home roots for Super Admin users
- Default to "home"
Upload Sources:
- Device file picker
- Camera (take photo)
- Photo library
- Share sheet (from other apps)
- Drag & drop (tablet only)
6.6 Files - File Viewer
┌─────────────────────────┐
│ ← notes.md [↗] [···]│
├─────────────────────────┤
│ │
│ # Meeting Notes │
│ │
│ ## Action Items │
│ - Fix login bug │
│ - Update docs │
│ - Review PR #42 │
│ │
│ ## Discussion │
│ We talked about the │
│ new mobile app and... │
│ │
│ │
│ │
│ │
└─────────────────────────┘
Viewer Types:
| File Type | Viewer | Notes |
|---|---|---|
.md |
Rendered Markdown | With proper styling |
.txt, .json, .ts, .js, etc. |
Syntax-highlighted text | Read-only, with line numbers |
.jpg, .png, .gif, .svg, .webp |
Image viewer | Pinch to zoom, pan |
.pdf |
PDF viewer | Native PDF rendering |
.mp3, .wav, .flac, .ogg |
Audio player | Play/pause, seek, speed control |
.mp4, .webm, .mov |
Video player | Native player, fullscreen support |
.mkv, .avi |
Video player (transcoded) | Uses /api/file-browser/transcode endpoint |
| Other | Download prompt | "This file type can't be previewed. Download?" |
Header Actions:
- [↗] Share via native share sheet
- [···] Menu: Download to device, AI actions (contextual), Open in chat, Delete
AI Actions (contextual):
- Images: OCR (extract text)
- Text files: TTS (read aloud)
- Audio/Video: Transcribe
- Video: Extract audio
6.7 Activity
┌─────────────────────────┐
│ Activity │
├─────────────────────────┤
│ [Tasks] [Logs] [Procs] │
├─────────────────────────┤
│ │
│ Native Tasks │
│ ┌──────────────────┐ │
│ │ 📋 Format Code │ │
│ │ Formats source │ │
│ │ files with... │ │
│ │ [Run ▶] │ │
│ └──────────────────┘ │
│ ┌──────────────────┐ │
│ │ 📋 Compress Imgs │ │
│ │ Optimizes image │ │
│ │ files in... │ │
│ │ [Run ▶] │ │
│ └──────────────────┘ │
│ │
│ User Tasks │
│ ┌──────────────────┐ │
│ │ 📋 Backup DB │ │
│ │ Runs pg_dump... │ │
│ │ [Run ▶] │ │
│ └──────────────────┘ │
│ │
└─────────────────────────┘
Tabs:
- Tasks: List of available tasks (native/global/user). Each shows name, description, trigger types. "Run" button opens a sheet to select target file/directory.
- Logs: Recent task execution logs. Each shows task name, status (success/failure), timestamp, duration. Tap to see full log output.
- Processes: Currently running background processes. Show name, status, uptime. No start/stop controls (too dangerous on mobile - manage via web).
Task Runner:
- When "Run" is tapped, show a bottom sheet:
- Select target file/directory (mini file browser)
- Confirm and run
- Show real-time output in a scrollable log view
- Toast on completion
6.8 Settings
┌─────────────────────────┐
│ Settings │
├─────────────────────────┤
│ │
│ ┌──────────────────┐ │
│ │ 👤 John Doe │ │
│ │ john@example.com │ │
│ │ Super Admin │ │
│ └──────────────────┘ │
│ │
│ Profile │
│ Name, Username, Avatar │
│ Change Password → │
│ Passkeys → │
│ │
│ ───────────────────── │
│ │
│ App │
│ Theme System │
│ Notifications → │
│ Default Model Claude │
│ Biometric Lock [🔘] │
│ │
│ ───────────────────── │
│ │
│ Connection │
│ Server 192.168.1.5 → │
│ Status Connected 🟢│
│ │
│ ───────────────────── │
│ │
│ [ Sign Out ] │
│ │
│ Officer v1.0.0 │
│ │
└─────────────────────────┘
Sections:
- Profile card: Avatar, name, email, role. Tap to edit.
- Profile: Name/username editing, avatar upload (camera or gallery), password change, passkey management.
- App: Theme (light/dark/system), notification preferences, default chat model/provider, biometric lock toggle.
- Connection: Current server URL (editable), connection status indicator, last sync time.
- Sign Out: Clears JWT, biometric keys, navigates to login.
7. Tablet Adaptations
Tablets (iPad, Android tablets/foldables) get an enhanced layout while using the same app binary. Detection via screen width:
- Phone: < 600dp (Android) / Compact (iOS)
- Tablet: >= 600dp (Android) / Regular (iOS)
7.1 Navigation
Phone: Bottom tab bar (5 tabs) Tablet: Side navigation rail (Android) / Sidebar (iOS)
Tablet Layout:
┌────┬──────────────────────────────┐
│ │ │
│ 🏠 │ │
│ 💬 │ [Main Content] │
│ 📁 │ │
│ ⚡ │ │
│ │ │
│ │ │
│ ⚙️ │ │
└────┴──────────────────────────────┘
7.2 Chat on Tablet
Split view: Session list on left (1/3), conversation on right (2/3).
┌────┬──────────┬───────────────────┐
│ │Sessions │ Debug login │
│ 🏠 │──────────│ Claude 3.5 Sonnet │
│ 💬 │ Debug.. │───────────────────│
│ 📁 │ Refact.. │ │
│ ⚡ │ Recipe.. │ [Messages...] │
│ │ │ │
│ │ │ │
│ │ ├───────────────────│
│ ⚙️ │ │ [Input] [Send] │
└────┴──────────┴───────────────────┘
7.3 Files on Tablet
Split view: Directory browser on left, file viewer on right. Tap a file to preview it in the right pane without navigating away.
┌────┬──────────┬───────────────────┐
│ │ / > Docs │ notes.md │
│ 🏠 │──────────│───────────────────│
│ 💬 │ 📁 Proj │ # Meeting Notes │
│ 📁 │ 📁 Reps │ │
│ ⚡ │ 📄 note │ ## Action Items │
│ │ 🖼️ phot │ - Fix login bug │
│ │ 🎵 song │ - Update docs │
│ │ │ │
│ ⚙️ │ │ │
└────┴──────────┴───────────────────┘
7.4 Dashboard on Tablet
Widgets in a 2-3 column grid instead of single column. More widgets visible at once.
┌────┬──────────────────────────────┐
│ │ ┌──────┐ ┌──────┐ ┌──────┐ │
│ 🏠 │ │Clock │ │Weath.│ │Pomo │ │
│ 💬 │ └──────┘ └──────┘ └──────┘ │
│ 📁 │ ┌─────────────┐ ┌──────────┐│
│ ⚡ │ │ Daily Goals │ │ Quick ││
│ │ │ ☑ Exercise │ │ Notes ││
│ │ │ ☐ Read 30min │ │ ... ││
│ │ └─────────────┘ └──────────┘│
│ ⚙️ │ Workspaces │
│ │ ┌────┐ ┌────┐ ┌────┐ ┌────┐ │
└────┴──────────────────────────────┘
7.5 Tablet-Only Features
| Feature | Description |
|---|---|
| Drag & Drop | Drag files between panes, drag text into chat |
| Keyboard Shortcuts | External keyboard support with shortcuts (Cmd+N for new chat, etc.) |
| Picture-in-Picture | Video playback continues in PiP when navigating away |
| Multi-Window | Android: Split screen support. iPad: Stage Manager support. |
8. Technical Challenges
8.1 WebSocket Lifecycle on Mobile
Problem: Mobile OSes aggressively kill background connections. WebSocket to chat/terminal will disconnect when app is backgrounded.
Solution:
- Detect app going to background → gracefully close WebSocket
- On foreground → reconnect WebSocket, resume session
- Use
{ type: 'resume', sessionId }to restore chat state - Buffer any pending assistant output server-side (the server already does this - messages are saved to JSON files)
- Show "Reconnecting..." indicator when connection drops
- Implement exponential backoff for reconnection attempts
Android specifics:
- Use
ProcessLifecycleOwnerto detect app lifecycle - Consider foreground service for active chat sessions (shows notification "Chat in progress")
iOS specifics:
- Use
UIApplication.willResignActiveNotification/didBecomeActiveNotification - Background URL session for critical requests
- Use
BGAppRefreshTaskfor periodic sync
8.2 Push Notifications
Problem: The Officer API server currently has no push notification infrastructure.
Required Backend Changes:
- New endpoint:
POST /api/devices/register— stores FCM/APNs token per user - New database table:
DeviceTokens(userId, token, platform, createdAt) - Server-side notification triggers:
- Chat: When assistant finishes responding and user's WebSocket is disconnected
- Tasks: When a long-running task completes
- Downloads: When yt-dlp or git clone finishes
- Firebase Admin SDK (Android) + APNs library (iOS) on the server
Notification Types:
| Type | Title | Body | Action |
|---|---|---|---|
| Chat complete | "Claude responded" | First 100 chars of response | Open chat session |
| Task complete | "Task finished" | "Format Code completed successfully" | Open task log |
| Download complete | "Download ready" | "video.mp4 downloaded" | Open file browser |
| Error | "Task failed" | "Backup DB failed: connection refused" | Open task log |
8.3 File Upload from Mobile
Problem: Mobile file uploads have different sources than web (camera, photo library, share sheet).
Solution:
- Use platform file pickers that support multiple sources
- Camera capture: Take photo → upload via
POST /api/file-browser/upload - Compress images before upload (configurable quality setting)
- Show upload progress (the existing XHR-based upload in
useFiles.tssupports progress) - Background upload for large files using WorkManager (Android) / BGTaskScheduler (iOS)
- Resume interrupted uploads if possible
8.4 Offline Support
Problem: Users may open the app with no connectivity.
Strategy: Read-heavy offline, write-queued.
| Feature | Offline Behavior |
|---|---|
| Dashboard | Cached widget data, local timer for pomodoro |
| Chat history | Cached session list and recent messages |
| New chat | Queue message, show "Will send when online" |
| File browser | Show last cached directory listing |
| File viewer | Show cached/downloaded files only |
| Settings | Show cached profile, edits queued |
Local Database Schema (Room / SwiftData):
ChatSessions: id, name, provider, lastMessagePreview, updatedAt, isArchived
ChatMessages: id, sessionId, role, content, timestamp (cache recent N per session)
DirectoryCache: path, root, entriesJson, cachedAt
FileCache: path, root, localPath, size, cachedAt (for downloaded/viewed files)
PendingActions: id, type, payload, createdAt, status
8.5 Deep Linking & Share Sheet
URL Scheme: officer://
Universal Links: https://officer.dev/app/... (if applicable)
Deep Link Routes:
officer://chat/{sessionId}— open specific chatofficer://files?path=/Documents— open file browser at pathofficer://chat/new?prompt=...— start new chat with prefilled text
Share Sheet Integration (receiving):
- Register app as share target for: text, URLs, images, files
- When content is shared to Officer:
- Show a sheet: "Send to Chat" or "Upload to Files"
- If Chat: open new chat with shared content as first message
- If Files: open file browser with upload in current directory
8.6 Image Handling in Chat
Problem: The web app sends images as base64 in the WebSocket message. Large images on mobile could cause memory issues.
Solution:
- Before sending, resize images to max 2048px on longest edge
- Compress to JPEG at 85% quality (unless PNG is needed for screenshots)
- Convert to base64 and include in chat message
- Show thumbnail in the message thread, tap to view full-size
- Consider using the upload endpoint first, then sending attachment IDs instead of base64 (check if API supports this path — it does via
attachmentIdsin the chat message)
8.7 Markdown & Code Rendering
Problem: Rendering rich markdown with code blocks, tables, and images in native views.
Android Solution:
- Use a Markdown library (Markwon or compose-markdown)
- Code blocks: custom span with monospace font + syntax highlighting
- Consider a lightweight WebView-based renderer for complex markdown as fallback
iOS Solution:
- Use
AttributedStringwith custom markdown parsing - Or use a SwiftUI markdown library (MarkdownUI)
- Code blocks: custom view with monospace font + syntax highlighting
Code Syntax Highlighting:
- Android: Highlight.js via WebView, or a native library like CodeView
- iOS: Splash or Highlightr library
8.8 Streaming Text Display
Problem: Chat responses stream token-by-token via WebSocket. Need smooth text rendering without layout thrashing.
Solution:
- Buffer incoming tokens (e.g., collect for 50ms before rendering)
- Use a single mutable text state that appends
- Avoid re-rendering the entire message list on each token
- Only the last (active) message bubble needs updating
- Auto-scroll to bottom during streaming, unless user has scrolled up
8.9 Security
Concerns:
- JWT stored on device must be encrypted
- Biometric key binding for stored credentials
- Certificate pinning for API connections (optional but recommended for self-hosted)
- No sensitive data in logs
- Clear all data on sign out
- Jailbreak/root detection (optional)
Android:
- EncryptedSharedPreferences for tokens
- Android Keystore for biometric-bound keys
- ProGuard/R8 for release builds
iOS:
- Keychain with
kSecAttrAccessibleWhenUnlockedThisDeviceOnly - Secure Enclave for biometric keys
- App Transport Security (ATS) enforced
8.10 Self-Hosted Server Discovery
Problem: Officer is self-hosted. Users need to point the app at their server. The server might be on a local network, behind a VPN, or on the public internet.
Solution:
- First-launch: manual URL entry with validation
- Local network: mDNS/Bonjour discovery (optional nice-to-have)
- QR code scanning from web app settings page (generate QR with server URL + optional auth token)
- Store multiple server profiles (for users who have dev/prod setups)
- Connection status indicator in settings
9. API Integration Notes
Endpoints the Mobile App Will Use
Auth:
POST /api/auth/signin— loginPOST /api/auth/signup— register (if enabled)POST /api/auth/signout— logoutGET /api/auth/me— validate token, get user infoPOST /api/auth/change-password— password change*/passkeys/*— passkey registration/auth
Chat:
GET /api/sessions?provider={provider}— list sessionsDELETE /api/sessions/{provider}/{id}— delete sessionPOST /api/sessions/{provider}/{id}/rename— rename sessionPOST /api/sessions/{provider}/{id}/archive— archive session- WebSocket
/api/harness/{provider}/ws— chat communication
Files:
GET /api/file-browser/ls?path={path}&root={root}— list directoryGET /api/file-browser/read?path={path}— read text fileGET /api/file-browser/raw?path={path}— raw file (binary, supports Range)GET /api/file-browser/search?q={query}— search filesPOST /api/file-browser/mkdir— create directoryPOST /api/file-browser/write— write filePOST /api/file-browser/upload— upload files (multipart)POST /api/file-browser/rename— renamePOST /api/file-browser/copy— copyPOST /api/file-browser/move— moveDELETE /api/file-browser/rm— deleteGET /api/file-browser/download?path={path}— download filePOST /api/file-browser/download— download multiple as zipGET /api/file-browser/transcode?path={path}— transcode videoPOST /api/file-browser/tts— text-to-speechPOST /api/file-browser/ocr— OCR imagePOST /api/file-browser/transcribe— transcribe audioPOST /api/file-browser/extract-audio— extract audio from videoPOST /api/file-browser/extract— extract archivePOST /api/file-browser/git-clone— clone git repoPOST /api/file-browser/download-video— yt-dlp download
Tasks:
GET /api/tasks— list tasksGET /api/tasks/{name}— task detailsPOST /api/tasks/{name}/run— run task
Task Logs:
GET /api/task-logs— list logsGET /api/task-logs/{id}— log details
Processes:
GET /api/processes— list processes
Settings:
GET /api/user/settings— user settingsPUT /api/user/settings— update settingsPUT /api/auth/users— update profile
Upload:
POST /api/upload— upload attachments (for chat)
Scrape:
POST /api/scrape— scrape URL (for sharing URLs to chat)
New Endpoints Needed
| Endpoint | Purpose |
|---|---|
POST /api/devices/register |
Register push notification token |
DELETE /api/devices/{token} |
Unregister device on logout |
GET /api/sessions/{provider}/{id}/messages |
Get messages for a session (currently read from files, need API endpoint) |
Authentication Header
All requests include: Authorization: Bearer {jwt_token}
WebSocket connections pass token as query parameter: ?token={jwt_token}
10. Shared Design Language
While each platform follows its native design guidelines (Material 3 for Android, Human Interface Guidelines for iOS), some design elements should be consistent:
Color Palette
From the web app's Tailwind theme:
- Primary (Teal):
duck-teal— used for primary actions, active states - Accent (Yellow):
duck-yellow— used for text on primary, highlights - Dark:
duck-dark— text, icons - Background: Light/dark mode adaptive
Both apps should use these brand colors while adapting to platform conventions:
- Android: Material 3 dynamic color with Officer palette as seed
- iOS: Officer palette with system-adaptive backgrounds
Typography
- Android: Roboto (system default), with monospace for code
- iOS: SF Pro (system default), with SF Mono for code
- Both: Similar size scale, consistent heading hierarchy
Iconography
- Android: Material Symbols (outlined)
- iOS: SF Symbols
- Map lucide-react icons used in web to platform equivalents
Animations
- Android: Material Motion (shared element transitions, container transforms)
- iOS: Spring animations, matched geometry effects
- Both: Smooth transitions between screens, subtle feedback on interactions
11. Implementation Phases
Phase 1: Foundation (MVP)
Goal: Core functionality that makes the app useful enough to ship.
| Feature | Details |
|---|---|
| Auth | Email/password login, token storage, biometric unlock |
| Server setup | URL configuration, validation |
| Chat | New chat, session list, real-time conversation with streaming |
| Chat basics | Text messages, model selection, session management |
| Settings | Profile view, theme, logout |
Estimated screens: Login, Home (simple), Chat List, Chat Conversation, Settings
Phase 2: Files & Polish
Goal: File management and mobile-specific enhancements.
| Feature | Details |
|---|---|
| File browser | Directory listing, navigation, search |
| File viewer | Text, images, PDF, audio, video |
| File actions | Upload (camera, files), download, rename, delete |
| Chat attachments | Image attachment, voice input |
| Push notifications | Basic chat completion notifications |
Estimated screens: File Browser, File Viewer, enhanced Chat
Phase 3: Productivity
Goal: Dashboard, tasks, and deeper integration.
| Feature | Details |
|---|---|
| Dashboard | Widget system (clock, weather, pomodoro, goals, notes) |
| Tasks | Task list, run tasks, view logs |
| Workspace quick access | Open workspaces (single-panel) |
| Share sheet | Receive content from other apps |
| Offline support | Cached data, queued actions |
Estimated screens: Dashboard, Task List, Task Logs, enhanced Home
Phase 4: Tablet & Advanced
Goal: Tablet optimization and advanced features.
| Feature | Details |
|---|---|
| Tablet layouts | Split views for chat, files |
| Tablet navigation | Side rail / sidebar |
| Home screen widgets | Android widgets, iOS widgets |
| AI file actions | OCR, TTS, Transcribe from mobile |
| Multi-select | Batch file operations |
| Deep linking | URL scheme, universal links |
Phase 5: Nice-to-Haves
| Feature | Details |
|---|---|
| QR code server setup | Scan from web to configure |
| Local network discovery | mDNS/Bonjour |
| Picture-in-Picture | Video PiP on tablet |
| Passkey auth | Full WebAuthn support |
| Keyboard shortcuts | External keyboard on tablet |
| Skills browser | View and search skills library |
| Background downloads | yt-dlp and git clone in background |
Appendix: Decision Log
| Decision | Rationale |
|---|---|
| Native (Kotlin + Swift) over cross-platform | User preference. Also gives best performance, platform-native UX, and full access to platform APIs (biometrics, push, background tasks, share sheet). |
| Exclude terminal | Phone keyboards make shell commands impractical. The sandboxed Docker setup is inherently desktop-oriented. |
| Exclude code editor (write) | Writing code on a phone is poor UX. Read-only with syntax highlighting is sufficient for review. |
| Exclude automation builder | Complex form-heavy UI that doesn't translate to small screens. |
| Include chat as primary feature | AI assistant in your pocket is the strongest mobile use case. Voice input makes it even more compelling on mobile. |
| Bottom tabs (phone) / Side rail (tablet) | Standard navigation patterns for each form factor. 5 top-level destinations fits within guidelines. |
| Offline-first for reads, queue for writes | Mobile connectivity is unreliable. Showing cached data is better than empty screens. |
| Phase 1 = Auth + Chat | Chat is the killer feature. Get it right first, everything else is complementary. |