apify tool, tools API + automation UI, integrations config, super admin restrictions
- apify tool: TOOL.md definition, index.ts implementation with auto-auth via OFFICER_APIFY_TOKEN, output_path for large datasets - tools API: /tools routes (list, detail, chat, create, delete) mirroring tasks pattern - automation UI: tools tab in sidebar, NewTool component, tool detail view - apify integration: settings page for enterprise API key config, pi-bridge passes env var to containers - tiktok-trends task: rewritten as agent instructions using apify tool with output_path, scripted report generation for 50KB read limit - restrict edit/delete of native/global capabilities to Super Admin only (backend + frontend) - tools authoring guide: TOOLS.md with full spec for TOOL.md frontmatter, index.ts execute signature, patterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
+142
-129
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: TikTok Trends
|
||||
description: Fetch top trending TikTok videos for a given country and generate an engagement report with optional video downloads.
|
||||
version: 2
|
||||
version: 3
|
||||
author: pastilhas
|
||||
tags:
|
||||
- social-media
|
||||
@@ -9,17 +9,9 @@ tags:
|
||||
- trends
|
||||
- apify
|
||||
- content-analysis
|
||||
skills:
|
||||
- Apify
|
||||
tools:
|
||||
- apify
|
||||
dependencies:
|
||||
- name: tiktok-trends-script
|
||||
description: The main script at $OFFICER_USER_ROOT/../resources/scripts/apify/tiktok-trends.ts
|
||||
check_command: test -f "$OFFICER_USER_ROOT/../resources/scripts/apify/tiktok-trends.ts"
|
||||
optional: false
|
||||
- name: bun
|
||||
description: Required to run the TypeScript script
|
||||
check_command: bun --version
|
||||
optional: false
|
||||
- name: yt-dlp
|
||||
description: Required only when download option is enabled. Downloads TikTok videos.
|
||||
check_command: yt-dlp --version
|
||||
@@ -103,26 +95,15 @@ inputs:
|
||||
type: boolean
|
||||
default: false
|
||||
required: false
|
||||
- name: api_token
|
||||
description: Apify API token with access to actors. Get yours at https://console.apify.com/account/integrations
|
||||
type: string
|
||||
required: true
|
||||
sensitive: true
|
||||
outputs:
|
||||
- name: engagement_report
|
||||
description: Markdown report with trending videos analysis, engagement metrics, top hashtags, sounds, and creators.
|
||||
path: tiktok_trends_<country>_<timestamp>/report-YYYY-MM-DD.md
|
||||
path: tiktok_trends_<country>_<timestamp>/report.md
|
||||
- name: raw_data
|
||||
description: Raw JSON data from Apify actor containing all video metadata.
|
||||
path: tiktok_trends_<country>_<timestamp>/raw-YYYY-MM-DD.json
|
||||
- name: execution_log
|
||||
description: Complete script execution log with stdout and stderr for debugging.
|
||||
path: tiktok_trends_<country>_<timestamp>/run.log
|
||||
- name: summary
|
||||
description: Human-readable summary of the run including cost analysis and file listing.
|
||||
path: tiktok_trends_<country>_<timestamp>/report.md
|
||||
path: tiktok_trends_<country>_<timestamp>/raw.json
|
||||
- name: videos
|
||||
description: Downloaded video files with metadata (only present if download option was enabled).
|
||||
description: Downloaded video files (only present if download option was enabled).
|
||||
path: tiktok_trends_<country>_<timestamp>/videos/
|
||||
optional: true
|
||||
config:
|
||||
@@ -134,125 +115,157 @@ config:
|
||||
|
||||
Fetch top trending TikTok videos for a given country and generate a comprehensive engagement report.
|
||||
|
||||
## Overview
|
||||
## Important
|
||||
|
||||
This task uses the Apify `novi~fast-tiktok-scraper` actor to fetch trending videos from TikTok for a specified country. It generates:
|
||||
- **Engagement metrics** (total/average views, likes, shares, comments)
|
||||
- **Top hashtags** found in trending content
|
||||
- **Popular sounds** being used
|
||||
- **Creator analysis** (who appears multiple times in trending)
|
||||
- **Complete video list** with links and stats
|
||||
- Use the `apify` tool to fetch data. Do NOT call the Apify REST API directly via curl or fetch.
|
||||
- Do NOT explore or list files before starting. Create the output directory, call the tool, process results.
|
||||
- If `download` is false (default), do NOT attempt to download any videos.
|
||||
- Large data files (like `raw.json`) exceed the 50KB read limit. Never try to read them directly. Instead, write a Node.js script to process and transform the data, execute it, then delete the script.
|
||||
|
||||
Optionally downloads videos using yt-dlp for offline analysis.
|
||||
## Steps
|
||||
|
||||
## Pre-execution Checks
|
||||
### 1. Setup output directory
|
||||
|
||||
1. **Validate inputs**:
|
||||
- `api_token` must be provided (required)
|
||||
- `country` must be a valid 2-letter country code
|
||||
- `limit` must be between 1-100
|
||||
|
||||
2. **Check dependencies**:
|
||||
- Verify script exists at `$OFFICER_USER_ROOT/../resources/scripts/apify/tiktok-trends.ts`
|
||||
- Verify bun is installed
|
||||
- If `download=true`, verify yt-dlp is installed
|
||||
|
||||
3. **Test Apify token**:
|
||||
- Make a test call to `GET https://api.apify.com/v2/users/me` to validate the token
|
||||
- Abort with clear error if token is invalid or account has no credits
|
||||
|
||||
## Execution
|
||||
|
||||
Run the script with the following command:
|
||||
|
||||
```bash
|
||||
bun "$OFFICER_USER_ROOT/../resources/scripts/apify/tiktok-trends.ts" \
|
||||
--apikey "<api_token>" \
|
||||
--country "<country>" \
|
||||
--limit <limit> \
|
||||
--output "<output_dir>" \
|
||||
<download_flag>
|
||||
Create a timestamped output directory:
|
||||
```
|
||||
$HOME/tiktok-trends/tiktok_trends_<country>_<YYYYMMDD_HHMMSS>/
|
||||
```
|
||||
|
||||
Where:
|
||||
- `<download_flag>` is `--download` if `download=true`, otherwise omitted
|
||||
- `<output_dir>` is a timestamped directory created inside `$HOME/tiktok-trends/` (the logged-in user's home directory)
|
||||
### 2. Fetch trending videos
|
||||
|
||||
**Pipe output to log**:
|
||||
```bash
|
||||
<command_above> 2>&1 | tee "<output_dir>/run.log"
|
||||
Call the `apify` tool with `output_path` pointing to `raw.json` in the output directory:
|
||||
```
|
||||
apify(
|
||||
actor_id: "novi~fast-tiktok-scraper",
|
||||
input: { "type": "TREND", "region": "<country>", "maxItems": <limit> },
|
||||
output_path: "<output_dir>/raw.json"
|
||||
)
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
The tool saves the full dataset to `raw.json` and returns a summary (item count). If it returns an error or 0 items, report the error and stop.
|
||||
|
||||
The task is considered successful when:
|
||||
- Script exits with code 0
|
||||
- `report-YYYY-MM-DD.md` exists and is non-empty
|
||||
- `raw-YYYY-MM-DD.json` exists and contains valid JSON array
|
||||
- (If download enabled) `videos/` directory exists with at least one file
|
||||
### 3. Generate engagement report
|
||||
|
||||
**Important:** The raw JSON file is too large to read directly (exceeds the 50KB read limit). Instead, write a Node.js script (e.g. `generate-report.js`) in the output directory that reads `raw.json`, processes the data, and writes `report.md`. Then execute it with `node generate-report.js`. Delete the script after it runs successfully.
|
||||
|
||||
The script should read `raw.json`, parse it as a JSON array of video items, and write `report.md` with the following sections:
|
||||
|
||||
#### Header
|
||||
```markdown
|
||||
# TikTok Trending Report — <COUNTRY> — YYYY-MM-DD
|
||||
|
||||
Total videos analyzed: <count>
|
||||
```
|
||||
|
||||
#### Engagement Summary
|
||||
|
||||
Build a table from each video's `statistics` object (`play_count`, `digg_count`, `share_count`, `comment_count`):
|
||||
|
||||
| Metric | Total | Avg per video |
|
||||
|--------|------:|-------------:|
|
||||
| Views | ... | ... |
|
||||
| Likes | ... | ... |
|
||||
| Shares | ... | ... |
|
||||
| Comments | ... | ... |
|
||||
|
||||
#### Top Hashtags (up to 20)
|
||||
|
||||
Extract hashtags from each video's `text_extra` array (entries where `hashtag_name` is set). If `text_extra` is empty, fall back to parsing `#tags` from the `desc` field. Count occurrences, sort descending.
|
||||
|
||||
| Hashtag | Count |
|
||||
|---------|------:|
|
||||
|
||||
#### Top Sounds (up to 10)
|
||||
|
||||
From each video's `music` object, format as `title — author`. Count occurrences, sort descending.
|
||||
|
||||
| Sound | Count |
|
||||
|-------|------:|
|
||||
|
||||
#### Creators Appearing in Trending (up to 10)
|
||||
|
||||
From each video's `author.unique_id`. Count occurrences, sort descending.
|
||||
|
||||
| Creator | Videos |
|
||||
|---------|-------:|
|
||||
|
||||
#### Video List
|
||||
|
||||
Full table of all videos, sorted by position:
|
||||
|
||||
| # | Creator | Description | Views | Likes | URL |
|
||||
|--:|---------|-------------|------:|------:|-----|
|
||||
|
||||
- Creator: `@author.unique_id`
|
||||
- Description: first 60 chars of `desc`, pipe and newline characters replaced, with `...` if truncated
|
||||
- URL: `share_url`
|
||||
- Format numbers with locale separators (e.g. `1,234,567`)
|
||||
|
||||
### 4. Download videos (only if `download` is true)
|
||||
|
||||
If `download` is false, skip this step entirely.
|
||||
|
||||
If `download` is true:
|
||||
1. Check that `yt-dlp` is installed
|
||||
2. Create a `videos/` subdirectory in the output directory
|
||||
3. Write all `share_url` values to a `urls.txt` file
|
||||
4. Run: `yt-dlp -a urls.txt -o "videos/%(id)s.%(ext)s" --write-info-json --no-overwrites`
|
||||
5. Report how many videos were downloaded. Partial failures are acceptable — do not fail the task if some downloads fail.
|
||||
|
||||
### 5. Report results
|
||||
|
||||
Print a summary to the user:
|
||||
- Country and date
|
||||
- Number of videos fetched
|
||||
- Top video: `@creator` — views count — URL
|
||||
- Top hashtag and count
|
||||
- Output directory path
|
||||
- Files created and their sizes
|
||||
|
||||
## Data Shape Reference
|
||||
|
||||
Each video item from the actor has this structure:
|
||||
```json
|
||||
{
|
||||
"aweme_id": "string",
|
||||
"desc": "video description with #hashtags",
|
||||
"create_time": 1234567890,
|
||||
"share_url": "https://www.tiktok.com/@user/video/123",
|
||||
"author": {
|
||||
"unique_id": "username",
|
||||
"nickname": "Display Name",
|
||||
"uid": "123"
|
||||
},
|
||||
"statistics": {
|
||||
"play_count": 1000000,
|
||||
"digg_count": 50000,
|
||||
"share_count": 5000,
|
||||
"comment_count": 2000,
|
||||
"collect_count": 1000,
|
||||
"download_count": 500
|
||||
},
|
||||
"music": {
|
||||
"title": "Sound Name",
|
||||
"author": "Sound Author"
|
||||
},
|
||||
"text_extra": [
|
||||
{ "hashtag_name": "trending", "type": 1 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Scenario | Action |
|
||||
|----------|--------|
|
||||
| Script exits non-zero | Check `run.log` for Apify/API errors. Common causes: invalid token, no credits, rate limits |
|
||||
| Empty results | Report "No trending videos found" but mark as success if files were created |
|
||||
| Partial results | Report success if at least 1 video returned, note the discrepancy |
|
||||
| Timeout (>10 min) | Kill process, report timeout. Check Apify actor status in console |
|
||||
| Download failures | Report which videos failed but mark task as success if main report generated |
|
||||
|
||||
## Post-execution
|
||||
|
||||
After successful execution:
|
||||
|
||||
1. **Verify outputs**:
|
||||
- Check all expected files exist
|
||||
- Validate JSON is parseable
|
||||
- If download enabled, count video files
|
||||
|
||||
2. **Generate summary** (`report.md`):
|
||||
```markdown
|
||||
# TikTok Trends Run Summary
|
||||
|
||||
## Run Details
|
||||
- **Date**: YYYY-MM-DD
|
||||
- **Timestamp**: HH:MM:SS
|
||||
- **Apify Actor**: novi~fast-tiktok-scraper
|
||||
- **Country**: <country_name> (<country_code>)
|
||||
- **Requested**: <limit> videos
|
||||
- **Received**: <actual_count> videos
|
||||
- **Duration**: <X> minutes <Y> seconds
|
||||
|
||||
## Cost Analysis
|
||||
- **Apify Credits Used**: <cost> USD
|
||||
- **Cost per Video**: $<cost/actual_count>
|
||||
|
||||
## Key Findings
|
||||
- **Top Video**: @<creator> - <views> views
|
||||
<description_preview>
|
||||
URL: <url>
|
||||
|
||||
- **Top Hashtag**: #<hashtag> (<count> occurrences)
|
||||
- **Top Sound**: <sound_name> (<count> uses)
|
||||
|
||||
## Output Files
|
||||
- `report-YYYY-MM-DD.md` (<size>) - Engagement report
|
||||
- `raw-YYYY-MM-DD.json` (<size>) - Raw API data
|
||||
- `run.log` (<size>) - Execution log
|
||||
- `videos/` (<count> files, <total_size>) - Downloaded videos (if enabled)
|
||||
|
||||
## Notes
|
||||
- <any_warnings_or_errors_from_log>
|
||||
```
|
||||
|
||||
3. **Cleanup**:
|
||||
- Remove temporary files if any
|
||||
- Report final status to user
|
||||
| Apify tool returns error | Report the error message. Common causes: invalid token, no credits, rate limits |
|
||||
| Empty results | Report "No trending videos found for <country>" and stop |
|
||||
| Partial results (fewer than requested) | Proceed normally, note the discrepancy in the summary |
|
||||
| yt-dlp not installed when download=true | Report that yt-dlp is required and skip downloads |
|
||||
| Download failures | Report which videos failed but do not fail the task |
|
||||
|
||||
## Notes
|
||||
|
||||
- **Rate limiting**: The Apify actor may take 1-3 minutes depending on the limit
|
||||
- **Costs**: Each run uses Apify compute units. Monitor usage at https://console.apify.com/billing
|
||||
- **Data freshness**: Trends data is near real-time but may have slight delays
|
||||
- **Video downloads**: Downloading many videos can take significant time and storage
|
||||
- **Country availability**: Not all countries have sufficient trending data; some may return fewer results than requested
|
||||
- The Apify actor may take 1-3 minutes depending on the limit
|
||||
- Each run uses Apify compute units — monitor at https://console.apify.com/billing
|
||||
- Not all countries have sufficient trending data; some may return fewer results than requested
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
# Tools
|
||||
|
||||
A tool is a callable capability that agents can use during task execution. Each tool lives in its own directory under `tools/` and is defined by a `TOOL.md` file and an `index.ts` (or `index.js`) entry point.
|
||||
|
||||
Tools are loaded by the `tool-loader` extension at startup and registered as callable functions. They appear in the agent's system prompt and can be invoked by name.
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
tools/
|
||||
<tool-name>/
|
||||
TOOL.md # Metadata and documentation
|
||||
index.ts # Implementation (required)
|
||||
```
|
||||
|
||||
Both files are required. The loader skips directories missing either `TOOL.md` or an entry file.
|
||||
|
||||
## TOOL.md Format
|
||||
|
||||
A tool file has two parts: **frontmatter** (YAML metadata) and **body** (Markdown documentation).
|
||||
|
||||
### Frontmatter
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: tool_name
|
||||
label: Tool Name
|
||||
description: What the tool does and when the agent should use it.
|
||||
version: 1
|
||||
language: typescript
|
||||
inputs:
|
||||
param_name:
|
||||
type: string
|
||||
description: What this parameter is for.
|
||||
optional_param:
|
||||
type: number
|
||||
description: An optional parameter.
|
||||
optional: true
|
||||
secret_param:
|
||||
type: string
|
||||
description: A sensitive value (e.g., API token).
|
||||
optional: true
|
||||
sensitive: true
|
||||
mode:
|
||||
type: enum
|
||||
values: single,batch
|
||||
description: Choose between modes.
|
||||
---
|
||||
```
|
||||
|
||||
#### Fields
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `name` | string | yes | Identifier for the tool (used in tool calls). Use `snake_case`. |
|
||||
| `label` | string | no | Human-readable display name. Defaults to `name` if omitted. |
|
||||
| `description` | string | yes | What the tool does. This appears in the agent's system prompt — make it specific enough for the agent to know when to use it. |
|
||||
| `version` | integer | no | Version number. Used by `sync-tools` to detect updates — bump when changing the tool. |
|
||||
| `language` | string | no | Implementation language: `typescript`, `bash`, or `python`. Defaults to `typescript`. |
|
||||
| `inputs` | object | no | Input parameters the tool accepts. Keys are parameter names. |
|
||||
|
||||
#### Input Fields
|
||||
|
||||
Each input is a key under `inputs:` with these properties:
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `type` | string | yes | Parameter type: `string`, `number`, `boolean`, or `enum`. |
|
||||
| `description` | string | yes | What this parameter is for. Shown to the agent. |
|
||||
| `optional` | boolean | no | Whether the parameter is optional. Defaults to required. |
|
||||
| `sensitive` | boolean | no | Mark sensitive values (tokens, passwords). Prevents logging. |
|
||||
| `values` | string | no | Comma-separated allowed values when `type` is `enum`. |
|
||||
| `default` | any | no | Default value if not provided. |
|
||||
|
||||
### Body
|
||||
|
||||
The body is Markdown documentation that the agent sees when the tool is loaded. Include:
|
||||
|
||||
- **Title** — `# Tool Name`
|
||||
- **Authentication** — How credentials are resolved (env vars, integrations, etc.)
|
||||
- **Usage** — How to call the tool and what parameters to pass.
|
||||
- **Examples** — Common usage patterns.
|
||||
- **Error Handling** — What errors can occur and what they mean.
|
||||
- **Notes** — Limits, billing, external links.
|
||||
|
||||
## index.ts Format
|
||||
|
||||
The entry file must export an `execute` function:
|
||||
|
||||
```typescript
|
||||
type ToolResult = {
|
||||
content: Array<{ type: string; text: string }>;
|
||||
isError?: boolean;
|
||||
};
|
||||
|
||||
type OnUpdate = (partial: { content: Array<{ type: string; text: string }> }) => void;
|
||||
|
||||
export async function execute(
|
||||
toolCallId: string,
|
||||
params: Record<string, unknown>,
|
||||
signal: AbortSignal | undefined,
|
||||
onUpdate?: OnUpdate,
|
||||
): Promise<ToolResult> {
|
||||
// Implementation here
|
||||
}
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
| Parameter | Description |
|
||||
|-----------|-------------|
|
||||
| `toolCallId` | Unique ID for this tool call. |
|
||||
| `params` | Input values from the agent, matching the `inputs` defined in TOOL.md. |
|
||||
| `signal` | AbortSignal for cancellation. |
|
||||
| `onUpdate` | Callback for streaming progress updates to the agent during long operations. |
|
||||
|
||||
### Return Value
|
||||
|
||||
Return a `ToolResult` object:
|
||||
- `content` — Array of content blocks. Usually one `{ type: 'text', text: '...' }`.
|
||||
- `isError` — Set `true` to indicate failure. The agent sees the error and can react.
|
||||
|
||||
### Progress Updates
|
||||
|
||||
Use `onUpdate` to stream status during long-running operations:
|
||||
|
||||
```typescript
|
||||
onUpdate?.({ content: [{ type: 'text', text: 'Processing step 2 of 5...' }] });
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
Tools should resolve credentials internally, not require the agent to pass them. Pattern:
|
||||
|
||||
1. Check for an explicit parameter override (e.g., `params.api_token`)
|
||||
2. Fall back to an environment variable (e.g., `process.env.OFFICER_APIFY_TOKEN`)
|
||||
3. Return a helpful error if neither is available
|
||||
|
||||
Environment variables are set by `pi-bridge.ts` from the integration config stored in the database (Settings → Integrations).
|
||||
|
||||
### Runtime Environment
|
||||
|
||||
Tools run inside sandboxed containers using **Node.js** (not Bun). Do not use Bun-specific APIs like `Bun.sleep`, `Bun.file`, etc. Use Node.js equivalents:
|
||||
|
||||
- `setTimeout` / `setInterval` for delays
|
||||
- `fs.readFileSync` / `fs.writeFileSync` for file I/O
|
||||
- `fetch` (available in Node 18+) for HTTP requests
|
||||
|
||||
### Large Output
|
||||
|
||||
If a tool may return large data (e.g., API responses with many items), provide an `output_path` parameter. When set, save the data to the file and return a summary instead:
|
||||
|
||||
```typescript
|
||||
if (params.output_path) {
|
||||
writeFileSync(params.output_path, JSON.stringify(items, null, 2));
|
||||
return { content: [{ type: 'text', text: `${items.length} items saved to ${params.output_path}` }] };
|
||||
}
|
||||
```
|
||||
|
||||
This prevents flooding the agent's context window with raw data.
|
||||
|
||||
## Sync and Discovery
|
||||
|
||||
Tools are synced from `seed/tools/` to `DATA_PATH/tools/` at server startup by `sync-tools.ts`. The sync is version-based — it only overwrites when the seed version is higher than the target version. Always bump `version` in the frontmatter when updating a tool.
|
||||
|
||||
The `tool-loader` extension discovers tools from directories listed in the `PI_TOOLS_DIRS` environment variable (colon-separated). Only `DATA_PATH/tools/` is mounted into containers — `seed/tools/` is not directly accessible at runtime.
|
||||
|
||||
## Existing Tools
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `gmail` | Read Gmail messages, threads, labels via Google API |
|
||||
| `web_search` | Search the web via SearXNG |
|
||||
| `web_fetch` | Fetch and extract content from URLs |
|
||||
| `browser` | Control a Chrome browser via Browser Relay |
|
||||
| `apify` | Run any Apify actor (web scraping, social media data) |
|
||||
| `convert_audio_to_mp3` | Convert audio files to MP3 via ffmpeg |
|
||||
| `ocr` | Optical character recognition on images |
|
||||
| `email_db` | Query the synced email database |
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
name: apify
|
||||
label: Apify
|
||||
version: 4
|
||||
description: Run any Apify actor and return its dataset results. Use for web scraping, data extraction, and automation — TikTok, Twitter, Facebook, Instagram, YouTube, Google, and hundreds more. Authentication is handled automatically when configured in Settings → Integrations.
|
||||
language: typescript
|
||||
inputs:
|
||||
actor_id:
|
||||
type: string
|
||||
description: "Actor ID to run (format: owner~actor-name or owner/actor-name, e.g. 'novi~fast-tiktok-scraper', 'apify/twitter-scraper')"
|
||||
input:
|
||||
type: object
|
||||
description: Actor-specific input parameters as a JSON object (varies per actor)
|
||||
optional: true
|
||||
output_path:
|
||||
type: string
|
||||
description: File path to save JSON results to. When provided, the tool writes the dataset to this file and returns a summary instead of the raw JSON. Recommended for large datasets to avoid flooding the context.
|
||||
optional: true
|
||||
api_token:
|
||||
type: string
|
||||
description: Override the configured API token. Usually not needed — the token is provided automatically from Settings → Integrations → Apify.
|
||||
optional: true
|
||||
sensitive: true
|
||||
timeout_ms:
|
||||
type: number
|
||||
description: Max time to wait for actor completion in milliseconds (default 300000 = 5 min)
|
||||
optional: true
|
||||
poll_interval_ms:
|
||||
type: number
|
||||
description: How often to check run status in milliseconds (default 3000)
|
||||
optional: true
|
||||
---
|
||||
|
||||
# Apify
|
||||
|
||||
Run any actor from the Apify Store, wait for completion, and return the dataset items as JSON.
|
||||
|
||||
## Authentication
|
||||
|
||||
The API token is resolved automatically:
|
||||
1. `api_token` input parameter (explicit override)
|
||||
2. `OFFICER_APIFY_TOKEN` environment variable (set automatically when configured in Settings → Integrations → Apify)
|
||||
|
||||
If neither is available, the tool returns an error prompting the user to configure the integration.
|
||||
|
||||
## Usage
|
||||
|
||||
Just provide the `actor_id` and optional `input`:
|
||||
```
|
||||
apify(actor_id: "novi~fast-tiktok-scraper", input: { type: "TREND", region: "PT", maxItems: 20 })
|
||||
```
|
||||
|
||||
The tool starts the actor, polls until completion, fetches the dataset, and returns all items as JSON.
|
||||
|
||||
## Common Actors
|
||||
|
||||
| Actor | ID | Input example |
|
||||
|-------|----|---------------|
|
||||
| TikTok Scraper | `novi~fast-tiktok-scraper` | `{ type: 'TREND', region: 'US', maxItems: 20 }` |
|
||||
| Twitter Scraper | `apify/twitter-scraper` | `{ searchTerms: ['#ai'], tweetsCount: 100 }` |
|
||||
| Twitter User | `jupri/twitter-user-scraper` | `{ twitterUser: 'username', maxPosts: 50 }` |
|
||||
| Facebook Scraper | `apify/facebook-scraper` | `{ startUrls: ['https://facebook.com/Page'], maxPostsPerPage: 50 }` |
|
||||
| Facebook Search | `jupri/facebook-search-scraper` | `{ searchTerm: 'keyword', maxPosts: 30 }` |
|
||||
| Instagram Hashtag | `apify/instagram-hashtag-scraper` | `{ hashtags: ['travel'], resultsLimit: 50 }` |
|
||||
| Instagram User | `apify/instagram-user-scraper` | `{ usernames: ['natgeo'], resultsLimit: 50 }` |
|
||||
| YouTube Scraper | `apify/youtube-scraper` | `{ searchTerms: ['tutorial'], maxResults: 20 }` |
|
||||
| Google Search | `apify/google-search-scraper` | `{ queries: ['best restaurants lisbon'] }` |
|
||||
|
||||
## Error Handling
|
||||
|
||||
The tool returns clear error messages for:
|
||||
- Missing API token → "Configure it in Settings → Integrations → Apify"
|
||||
- API errors (401, 403, etc.) → includes the HTTP status and response body
|
||||
- Actor failures → includes the actor's `statusMessage`
|
||||
- Timeouts → reports the run ID and last known status
|
||||
|
||||
## Notes
|
||||
|
||||
- Actor IDs use `~` (Apify URL format) or `/` — both work
|
||||
- Browse actors at https://apify.com/store
|
||||
- Monitor usage and credits at https://console.apify.com/billing
|
||||
@@ -0,0 +1,167 @@
|
||||
import { mkdirSync, writeFileSync } from 'node:fs';
|
||||
import { dirname } from 'node:path';
|
||||
|
||||
const BASE = 'https://api.apify.com/v2';
|
||||
|
||||
type RunStatus = 'READY' | 'RUNNING' | 'SUCCEEDED' | 'FAILED' | 'ABORTING' | 'ABORTED' | 'TIMING-OUT' | 'TIMED-OUT';
|
||||
|
||||
type RunData = {
|
||||
id: string;
|
||||
actId: string;
|
||||
status: RunStatus;
|
||||
statusMessage?: string;
|
||||
defaultDatasetId: string;
|
||||
defaultKeyValueStoreId: string;
|
||||
startedAt?: string;
|
||||
finishedAt?: string;
|
||||
};
|
||||
|
||||
type ToolResult = {
|
||||
content: Array<{ type: string; text: string }>;
|
||||
isError?: boolean;
|
||||
};
|
||||
|
||||
type OnUpdate = (partial: { content: Array<{ type: string; text: string }> }) => void;
|
||||
|
||||
type Params = {
|
||||
actor_id: string;
|
||||
input?: Record<string, unknown> | string;
|
||||
api_token?: string;
|
||||
output_path?: string;
|
||||
timeout_ms?: number;
|
||||
poll_interval_ms?: number;
|
||||
};
|
||||
|
||||
function update(onUpdate: OnUpdate | undefined, text: string): void {
|
||||
onUpdate?.({ content: [{ type: 'text', text }] });
|
||||
}
|
||||
|
||||
function resolveToken(params: Params): string | null {
|
||||
if (params.api_token) return params.api_token;
|
||||
return process.env.OFFICER_APIFY_TOKEN ?? null;
|
||||
}
|
||||
|
||||
function apiUrl(path: string, token: string, extra?: Record<string, string>): string {
|
||||
const params = new URLSearchParams({ token, ...extra });
|
||||
return `${BASE}${path}?${params}`;
|
||||
}
|
||||
|
||||
async function apiRequest<T>(url: string, init?: RequestInit): Promise<T> {
|
||||
const res = await fetch(url, init);
|
||||
if (!res.ok) {
|
||||
const body = await res.text();
|
||||
throw new Error(`Apify API error ${res.status}: ${body}`);
|
||||
}
|
||||
return res.json() as Promise<T>;
|
||||
}
|
||||
|
||||
async function startRun(token: string, actorId: string, input: Record<string, unknown>): Promise<RunData> {
|
||||
const { data } = await apiRequest<{ data: RunData }>(apiUrl(`/acts/${actorId}/runs`, token), {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(input),
|
||||
});
|
||||
return data;
|
||||
}
|
||||
|
||||
async function getRun(token: string, runId: string): Promise<RunData> {
|
||||
const { data } = await apiRequest<{ data: RunData }>(apiUrl(`/actor-runs/${runId}`, token));
|
||||
return data;
|
||||
}
|
||||
|
||||
async function getDatasetItems<T>(token: string, datasetId: string): Promise<T[]> {
|
||||
return apiRequest<T[]>(apiUrl(`/datasets/${datasetId}/items`, token, { format: 'json' }));
|
||||
}
|
||||
|
||||
const TERMINAL_STATUSES = new Set<RunStatus>(['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT']);
|
||||
|
||||
export async function execute(
|
||||
_toolCallId: string,
|
||||
params: Params,
|
||||
_signal: AbortSignal | undefined,
|
||||
onUpdate?: OnUpdate,
|
||||
): Promise<ToolResult> {
|
||||
const token = resolveToken(params);
|
||||
if (!token) {
|
||||
return {
|
||||
content: [{ type: 'text', text: 'Apify API token not available. Configure it in Settings → Integrations → Apify, or pass api_token explicitly.' }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
|
||||
const { actor_id } = params;
|
||||
if (!actor_id) {
|
||||
return {
|
||||
content: [{ type: 'text', text: 'actor_id is required.' }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
|
||||
let input: Record<string, unknown> = {};
|
||||
if (params.input) {
|
||||
if (typeof params.input === 'string') {
|
||||
try {
|
||||
input = JSON.parse(params.input);
|
||||
} catch {
|
||||
return {
|
||||
content: [{ type: 'text', text: 'Invalid JSON in input parameter.' }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
} else {
|
||||
input = params.input;
|
||||
}
|
||||
}
|
||||
|
||||
const timeoutMs = params.timeout_ms ?? 300_000;
|
||||
const pollIntervalMs = params.poll_interval_ms ?? 3_000;
|
||||
|
||||
try {
|
||||
update(onUpdate, `Starting actor ${actor_id}...`);
|
||||
const run = await startRun(token, actor_id, input);
|
||||
update(onUpdate, `Run ${run.id} started. Waiting for completion...`);
|
||||
|
||||
const start = Date.now();
|
||||
let finished = run;
|
||||
|
||||
while (!TERMINAL_STATUSES.has(finished.status)) {
|
||||
if (Date.now() - start > timeoutMs) {
|
||||
return {
|
||||
content: [{ type: 'text', text: `Timeout after ${timeoutMs}ms waiting for run ${run.id}. Status: ${finished.status}` }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
await new Promise((r) => setTimeout(r, pollIntervalMs));
|
||||
finished = await getRun(token, run.id);
|
||||
update(onUpdate, `Status: ${finished.status}...`);
|
||||
}
|
||||
|
||||
if (finished.status !== 'SUCCEEDED') {
|
||||
return {
|
||||
content: [{ type: 'text', text: `Actor run ${finished.status}: ${finished.statusMessage ?? 'unknown error'}` }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
|
||||
update(onUpdate, `Run succeeded. Fetching dataset items...`);
|
||||
const items = await getDatasetItems(token, finished.defaultDatasetId);
|
||||
|
||||
if (params.output_path) {
|
||||
mkdirSync(dirname(params.output_path), { recursive: true });
|
||||
writeFileSync(params.output_path, JSON.stringify(items, null, 2));
|
||||
return {
|
||||
content: [{ type: 'text', text: `${items.length} items saved to ${params.output_path}` }],
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
content: [{ type: 'text', text: JSON.stringify(items) }],
|
||||
};
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
return {
|
||||
content: [{ type: 'text', text: `Apify error: ${message}` }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import { NewPipeline } from './NewPipeline';
|
||||
import { NewCron } from './NewCron';
|
||||
import { NewService } from './NewService';
|
||||
import { NewWorkflow } from './NewWorkflow';
|
||||
import { NewTool } from './NewTool';
|
||||
|
||||
export type AutomationSelection = {
|
||||
kind: string;
|
||||
@@ -28,6 +29,7 @@ const newComponentMap: Record<string, React.ComponentType<{ selection: NonNullab
|
||||
Cron: NewCron,
|
||||
Service: NewService,
|
||||
Workflow: NewWorkflow,
|
||||
Tool: NewTool,
|
||||
};
|
||||
|
||||
export const AutomationRightPanel = () => {
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { Box, Clock, Cpu, GitBranch, ListTodo, Server, Sparkles, Workflow } from 'lucide-react';
|
||||
import { Box, Clock, Cpu, GitBranch, ListTodo, Server, Sparkles, Workflow, Wrench } from 'lucide-react';
|
||||
import { usePanelChannel } from 'hooks/usePanelChannel';
|
||||
import type { AutomationSelection } from './AutomationRightPanel';
|
||||
|
||||
@@ -6,6 +6,7 @@ const capabilityItems = [
|
||||
{ id: 'processes', label: 'Processes', icon: Cpu, kind: 'Process', endpoint: '/processes', queryKey: 'processes' },
|
||||
{ id: 'tasks', label: 'Tasks', icon: ListTodo, kind: 'Task', endpoint: '/tasks', queryKey: 'tasks' },
|
||||
{ id: 'skills', label: 'Skills', icon: Sparkles, kind: 'Skill', endpoint: '/skills', queryKey: 'skills' },
|
||||
{ id: 'tools', label: 'Tools', icon: Wrench, kind: 'Tool', endpoint: '/tools', queryKey: 'tools' },
|
||||
{ id: 'pipelines', label: 'Pipelines', icon: Workflow, kind: 'Pipeline', endpoint: '/pipelines', queryKey: 'pipelines' },
|
||||
{ id: 'crons', label: 'Crons', icon: Clock, kind: 'Cron', endpoint: '/crons', queryKey: 'crons' },
|
||||
{ id: 'services', label: 'Services', icon: Server, kind: 'Service', endpoint: '/services', queryKey: 'services' },
|
||||
|
||||
@@ -8,6 +8,7 @@ import { ArrowLeft, Pencil, Play, Trash2 } from 'lucide-react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Dialog, DialogContent, DialogHeader, DialogTitle, DialogDescription } from '@/components/ui/dialog';
|
||||
import { useClient } from 'hooks/useClient';
|
||||
import { useAuth } from 'hooks/useAuth';
|
||||
import { usePanelChannel } from 'hooks/usePanelChannel';
|
||||
import { Card } from '@/components/Card';
|
||||
import { FrontmatterBlock } from '../CapabilityPage';
|
||||
@@ -73,6 +74,7 @@ type CapabilityDetailViewProps = {
|
||||
export const CapabilityDetailView = ({ kind, endpoint, queryKey, dirName, editing }: CapabilityDetailViewProps) => {
|
||||
const client = useClient();
|
||||
const qc = useQueryClient();
|
||||
const { user } = useAuth();
|
||||
const [selection, setSelection] = usePanelChannel<AutomationSelection>('automation:selected-capability', null);
|
||||
const [deleteConfirm, setDeleteConfirm] = useState(false);
|
||||
const [showInputForm, setShowInputForm] = useState(false);
|
||||
@@ -149,6 +151,8 @@ export const CapabilityDetailView = ({ kind, endpoint, queryKey, dirName, editin
|
||||
<Play className="h-3.5 w-3.5 text-duck-teal" />
|
||||
</button>
|
||||
)}
|
||||
{(detail.scope === 'user' || user?.role === 'Super Admin') && (
|
||||
<>
|
||||
<button
|
||||
onClick={toggleEditing}
|
||||
className={`p-1 rounded hover:bg-duck-dark/10 dark:hover:bg-foreground/10 cursor-pointer transition-colors ${editing ? 'bg-duck-teal/10' : ''}`}
|
||||
@@ -163,6 +167,8 @@ export const CapabilityDetailView = ({ kind, endpoint, queryKey, dirName, editin
|
||||
</button>
|
||||
</>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
<div className="overflow-y-auto flex-1 p-6">
|
||||
{detail?.rawFrontmatter && <FrontmatterBlock yaml={detail.rawFrontmatter} />}
|
||||
@@ -275,7 +281,7 @@ export const CapabilityDetailView = ({ kind, endpoint, queryKey, dirName, editin
|
||||
onOpenChange={(open) => {
|
||||
if (!open) setRunPrompt(null);
|
||||
}}
|
||||
task={{ dirName, name: detail.name, description: detail.description ?? '', scope: detail.scope ?? 'user', triggers: [], filePath: detail.filePath }}
|
||||
task={{ dirName, name: detail.name, description: detail.description ?? '', scope: detail.scope === 'user' ? 'user' : 'global', triggers: [], filePath: detail.filePath }}
|
||||
promptOverride={runPrompt}
|
||||
/>
|
||||
)}
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
import { useState } from 'react';
|
||||
import { useQueryClient } from '@tanstack/react-query';
|
||||
import { X } from 'lucide-react';
|
||||
import { toast } from 'sonner';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { useClient } from 'hooks/useClient';
|
||||
import { usePanelChannel } from 'hooks/usePanelChannel';
|
||||
import type { AutomationSelection } from './AutomationRightPanel';
|
||||
|
||||
type NewToolProps = {
|
||||
selection: NonNullable<AutomationSelection>;
|
||||
};
|
||||
|
||||
export const NewTool = ({ selection }: NewToolProps) => {
|
||||
const client = useClient();
|
||||
const qc = useQueryClient();
|
||||
const [, setSelection] = usePanelChannel<AutomationSelection>('automation:selected-capability', null);
|
||||
const [name, setName] = useState('');
|
||||
const [description, setDescription] = useState('');
|
||||
const [submitting, setSubmitting] = useState(false);
|
||||
|
||||
const handleCreate = async () => {
|
||||
const trimmed = name.trim();
|
||||
if (!trimmed) return;
|
||||
setSubmitting(true);
|
||||
try {
|
||||
const res = await client.post<{ name: string; dirName: string }>(selection.endpoint, { name: trimmed });
|
||||
await qc.invalidateQueries({ queryKey: [selection.queryKey] });
|
||||
setSelection({
|
||||
kind: selection.kind,
|
||||
endpoint: selection.endpoint,
|
||||
queryKey: selection.queryKey,
|
||||
dirName: res.dirName,
|
||||
isNew: true,
|
||||
editing: true,
|
||||
description: description.trim() || undefined,
|
||||
});
|
||||
} catch {
|
||||
toast.error('Failed to create tool');
|
||||
setSubmitting(false);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="h-full flex flex-col">
|
||||
<div className="shrink-0 px-4 py-2 border-b border-duck-dark/10 dark:border-foreground/10 bg-background/60 flex items-center gap-2">
|
||||
<span className="text-sm font-medium text-duck-dark/70 dark:text-foreground/70 flex-1">New Tool</span>
|
||||
<button
|
||||
onClick={() => setSelection(null)}
|
||||
className="p-1 rounded hover:bg-duck-dark/10 dark:hover:bg-foreground/10 cursor-pointer transition-colors"
|
||||
>
|
||||
<X className="h-3.5 w-3.5 text-duck-dark/50 dark:text-foreground/50" />
|
||||
</button>
|
||||
</div>
|
||||
<div className="flex-1 overflow-y-auto p-6 flex flex-col gap-4">
|
||||
<div className="flex flex-col gap-1.5">
|
||||
<label className="text-sm font-medium text-duck-dark/70 dark:text-foreground/70">Name</label>
|
||||
<input
|
||||
value={name}
|
||||
onChange={(ev) => setName(ev.target.value)}
|
||||
onKeyDown={(ev) => {
|
||||
if (ev.key === 'Enter') {
|
||||
ev.preventDefault();
|
||||
handleCreate();
|
||||
}
|
||||
}}
|
||||
placeholder="Tool name..."
|
||||
autoFocus
|
||||
className="rounded border border-duck-dark/20 dark:border-foreground/20 bg-background px-3 py-2 text-sm text-duck-dark dark:text-foreground placeholder:text-duck-dark/30 dark:placeholder:text-foreground/30 focus:outline-none focus:ring-1 focus:ring-duck-teal/30"
|
||||
/>
|
||||
</div>
|
||||
<div className="flex flex-col gap-1.5">
|
||||
<label className="text-sm font-medium text-duck-dark/70 dark:text-foreground/70">Description</label>
|
||||
<textarea
|
||||
value={description}
|
||||
onChange={(ev) => setDescription(ev.target.value)}
|
||||
placeholder="Describe what this tool does..."
|
||||
rows={4}
|
||||
className="rounded border border-duck-dark/20 dark:border-foreground/20 bg-background px-3 py-2 text-sm text-duck-dark dark:text-foreground placeholder:text-duck-dark/30 dark:placeholder:text-foreground/30 focus:outline-none focus:ring-1 focus:ring-duck-teal/30 resize-none"
|
||||
/>
|
||||
</div>
|
||||
<div className="flex justify-end">
|
||||
<Button
|
||||
onClick={handleCreate}
|
||||
disabled={!name.trim() || submitting}
|
||||
className="bg-duck-teal text-duck-yellow hover:bg-duck-teal/90 cursor-pointer"
|
||||
>
|
||||
{submitting ? 'Creating...' : 'Create'}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
@@ -8,6 +8,7 @@ import { ArrowLeft, Pencil, Plus, Check, X, Trash2, Search, ChevronRight } from
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Dialog, DialogContent, DialogHeader, DialogTitle, DialogDescription } from '@/components/ui/dialog';
|
||||
import { useClient } from 'hooks/useClient';
|
||||
import { useAuth } from 'hooks/useAuth';
|
||||
import { Card } from '@/components/Card';
|
||||
import { usePiChat, EmbeddableChat } from 'officerdev';
|
||||
type CapabilitySummary = {
|
||||
@@ -249,6 +250,7 @@ export const CapabilityList = ({ kind, endpoint, queryKey, selected, onSelect, o
|
||||
export const CapabilityPage = ({ kind, endpoint, queryKey }: CapabilityPageProps) => {
|
||||
const client = useClient();
|
||||
const qc = useQueryClient();
|
||||
const { user } = useAuth();
|
||||
const [selected, setSelected] = useState<string | null>(null);
|
||||
const [editing, setEditing] = useState(false);
|
||||
const [isNew, setIsNew] = useState(false);
|
||||
@@ -330,7 +332,7 @@ export const CapabilityPage = ({ kind, endpoint, queryKey }: CapabilityPageProps
|
||||
<span className="text-sm font-medium text-duck-dark/70 flex-1">
|
||||
{detail?.name ?? `Select a ${kind.toLowerCase()}`}
|
||||
</span>
|
||||
{detail && (
|
||||
{detail && (detail.scope === 'user' || user?.role === 'Super Admin') && (
|
||||
<>
|
||||
<button
|
||||
onClick={() => setEditing((e) => !e)}
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
import { useState, useEffect } from 'react';
|
||||
import { toast } from 'sonner';
|
||||
import { Input } from '@/components/ui/input';
|
||||
import { Label } from '@/components/ui/label';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { useClient } from 'hooks/useClient';
|
||||
|
||||
type ApifyConfigData = {
|
||||
apiToken: string;
|
||||
};
|
||||
|
||||
type ApifyStatus = {
|
||||
configured: boolean;
|
||||
};
|
||||
|
||||
export const ApifyConfig = () => {
|
||||
const client = useClient();
|
||||
const [isLoading, setIsLoading] = useState(true);
|
||||
const [isSaving, setIsSaving] = useState(false);
|
||||
const [apiToken, setApiToken] = useState('');
|
||||
const [status, setStatus] = useState<ApifyStatus | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
Promise.all([
|
||||
client.get<ApifyConfigData>('/integrations/apify/config').then((data) => {
|
||||
if (data?.apiToken) setApiToken(data.apiToken);
|
||||
}),
|
||||
client.get<ApifyStatus>('/integrations/apify/status').then(setStatus),
|
||||
])
|
||||
.catch(() => {})
|
||||
.finally(() => setIsLoading(false));
|
||||
}, []);
|
||||
|
||||
const handleSave = async () => {
|
||||
if (isSaving) return;
|
||||
setIsSaving(true);
|
||||
try {
|
||||
await client.put('/integrations/apify/config', { apiToken: apiToken.trim() });
|
||||
toast.success('Apify API token saved');
|
||||
setStatus({ configured: !!apiToken.trim() });
|
||||
} catch {
|
||||
toast.error('Failed to save Apify API token');
|
||||
} finally {
|
||||
setIsSaving(false);
|
||||
}
|
||||
};
|
||||
|
||||
if (isLoading) return null;
|
||||
|
||||
return (
|
||||
<div className="grid gap-5">
|
||||
{status && (
|
||||
<div className="flex items-center gap-2 rounded-lg border border-duck-dark/10 dark:border-foreground/10 p-3">
|
||||
<div className={`h-2.5 w-2.5 rounded-full shrink-0 ${status.configured ? 'bg-green-500' : 'bg-duck-dark/20 dark:bg-foreground/20'}`} />
|
||||
<span className="text-sm text-duck-dark dark:text-foreground">
|
||||
{status.configured ? 'API token configured' : 'Not configured'}
|
||||
</span>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className="grid gap-5">
|
||||
<Label className="grid gap-2">
|
||||
<span className="text-duck-dark/70 dark:text-foreground/70">API Token</span>
|
||||
<Input
|
||||
className="h-11 bg-background/60 border-duck-dark/20 text-duck-dark placeholder:text-duck-dark/40"
|
||||
type="password"
|
||||
value={apiToken}
|
||||
onChange={(ev) => setApiToken(ev.target.value)}
|
||||
placeholder="apify_api_..."
|
||||
/>
|
||||
<span className="text-xs text-duck-dark/40 dark:text-foreground/40">
|
||||
Get your token at{' '}
|
||||
<a href="https://console.apify.com/account/integrations" target="_blank" rel="noopener noreferrer" className="text-duck-teal underline">
|
||||
console.apify.com/account/integrations
|
||||
</a>
|
||||
</span>
|
||||
</Label>
|
||||
|
||||
<Button
|
||||
type="button"
|
||||
onClick={handleSave}
|
||||
disabled={isSaving || !apiToken.trim()}
|
||||
className="w-full h-11 bg-duck-yellow hover:bg-duck-yellow/90 text-duck-teal font-bold transition-all duration-200 hover:scale-105 cursor-pointer disabled:opacity-50 disabled:hover:scale-100"
|
||||
>
|
||||
{isSaving ? 'Saving...' : 'Save'}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
@@ -1,5 +1,5 @@
|
||||
import { useMemo } from 'react';
|
||||
import { Puzzle, KeyRound, UserCircle, MessageCircle, Globe } from 'lucide-react';
|
||||
import { Puzzle, KeyRound, UserCircle, MessageCircle, Globe, Wrench } from 'lucide-react';
|
||||
import type { LayoutNode, PanelComponents } from 'officerdev';
|
||||
import { WorkspaceLayout } from 'officerdev';
|
||||
import { useAuth } from 'hooks/useAuth';
|
||||
@@ -16,6 +16,7 @@ import { TelegramAccount } from './TelegramAccount';
|
||||
import { WhatsAppBotConfig } from './WhatsAppBotConfig';
|
||||
import { WhatsAppAccount } from './WhatsAppAccount';
|
||||
import { BrowserRelay } from './BrowserRelay';
|
||||
import { ApifyConfig } from './ApifyConfig';
|
||||
|
||||
const GLOBAL_KEY = 'INTEGRATIONS_SETTINGS_SELECTED';
|
||||
const TAB_KEY = 'INTEGRATIONS_SETTINGS_TAB';
|
||||
@@ -49,6 +50,13 @@ const enterpriseSections: SettingsSection[] = [
|
||||
description: 'WhatsApp Web connection',
|
||||
content: <WhatsAppBotConfig />,
|
||||
},
|
||||
{
|
||||
key: 'apify',
|
||||
icon: Wrench,
|
||||
title: 'Apify',
|
||||
description: 'API token for web scraping actors',
|
||||
content: <ApifyConfig />,
|
||||
},
|
||||
];
|
||||
|
||||
const personalSections: SettingsSection[] = [
|
||||
|
||||
@@ -33,6 +33,40 @@ integrationsRouter.get('/', async (ctx) => {
|
||||
return ctx.json([]);
|
||||
});
|
||||
|
||||
// --- Enterprise: Apify config (Super Admin only) ---
|
||||
|
||||
type ApifyConfig = { apiToken: string };
|
||||
|
||||
export const readApifyConfig = async (): Promise<ApifyConfig | null> => {
|
||||
const integration = await getServerIntegration('apify');
|
||||
if (!integration) return null;
|
||||
const config = integration.config as Record<string, unknown>;
|
||||
if (!config.apiToken) return null;
|
||||
return config as unknown as ApifyConfig;
|
||||
};
|
||||
|
||||
integrationsRouter.get('/apify/config', async (ctx) => {
|
||||
const user = ctx.get('user');
|
||||
if (user.role !== 'Super Admin') throw FORBIDDEN();
|
||||
return ctx.json(await readApifyConfig());
|
||||
});
|
||||
|
||||
integrationsRouter.put('/apify/config', async (ctx) => {
|
||||
const user = ctx.get('user');
|
||||
if (user.role !== 'Super Admin') throw FORBIDDEN();
|
||||
|
||||
const body = ctx.get('body') as { apiToken?: string };
|
||||
const config = { apiToken: body.apiToken ?? '' };
|
||||
|
||||
await upsertServerIntegration('apify', config);
|
||||
return ctx.json(config);
|
||||
});
|
||||
|
||||
integrationsRouter.get('/apify/status', async (ctx) => {
|
||||
const config = await readApifyConfig();
|
||||
return ctx.json({ configured: !!config?.apiToken });
|
||||
});
|
||||
|
||||
// --- Enterprise: Google OAuth config (Super Admin only) ---
|
||||
|
||||
integrationsRouter.get('/google/config', async (ctx) => {
|
||||
|
||||
@@ -190,6 +190,16 @@ async function ensureGoogleTokenFile(userId: number, email: string): Promise<str
|
||||
return filePath;
|
||||
}
|
||||
|
||||
async function getApifyToken(): Promise<string> {
|
||||
try {
|
||||
const integration = await getServerIntegration('apify');
|
||||
const config = integration?.config as Record<string, string> | undefined;
|
||||
return config?.apiToken ?? '';
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
|
||||
async function getBrowserRelayEnv(userId: number): Promise<Record<string, string>> {
|
||||
const port = getRelayPort();
|
||||
if (!port) return {};
|
||||
@@ -265,6 +275,7 @@ export async function spawnPi(
|
||||
const googleConfigHost = await ensureGoogleConfigFile();
|
||||
await ensureGoogleTokenFile(sandbox.userId, sandbox.email);
|
||||
const browserRelayEnv = await getBrowserRelayEnv(sandbox.userId);
|
||||
const apifyToken = await getApifyToken();
|
||||
|
||||
const envFlags = [
|
||||
'-e', `HOME=${containerHome}`,
|
||||
@@ -276,6 +287,7 @@ export async function spawnPi(
|
||||
'-e', `OFFICER_GOOGLE_CONFIG_PATH=/officer/google-oauth.json`,
|
||||
'-e', `OFFICER_GOOGLE_TOKEN_PATH=/officer/user/integrations/google.json`,
|
||||
'-e', `OFFICER_EMAIL_DB=/officer/emails.db`,
|
||||
...(apifyToken ? ['-e', `OFFICER_APIFY_TOKEN=${apifyToken}`] : []),
|
||||
...Object.entries(browserRelayEnv).flatMap(([k, v]) => ['-e', `${k}=${v}`]),
|
||||
];
|
||||
|
||||
@@ -321,6 +333,7 @@ export async function spawnPi(
|
||||
const googleConfigPath = await ensureGoogleConfigFile();
|
||||
const googleTokenPath = await ensureGoogleTokenFile(userId, email);
|
||||
const browserRelayEnv = await getBrowserRelayEnv(userId);
|
||||
const apifyTokenLocal = await getApifyToken();
|
||||
|
||||
proc = Bun.spawn(args, {
|
||||
cwd,
|
||||
@@ -339,6 +352,7 @@ export async function spawnPi(
|
||||
OFFICER_GOOGLE_CONFIG_PATH: googleConfigPath,
|
||||
OFFICER_GOOGLE_TOKEN_PATH: googleTokenPath,
|
||||
OFFICER_EMAIL_DB: join(DATA_PATH, email, 'emails.db'),
|
||||
...(apifyTokenLocal ? { OFFICER_APIFY_TOKEN: apifyTokenLocal } : {}),
|
||||
...browserRelayEnv,
|
||||
},
|
||||
});
|
||||
|
||||
@@ -54,7 +54,7 @@ function resolveFile(name: string, native: Map<string, string>, global: Map<stri
|
||||
}
|
||||
|
||||
function isPrivileged(role: string) {
|
||||
return role === 'Admin' || role === 'Owner' || role === 'Super Admin';
|
||||
return role === 'Super Admin';
|
||||
}
|
||||
|
||||
export const processesRouter = createRouter();
|
||||
|
||||
@@ -41,7 +41,7 @@ function mergeConfig(native: Record<string, string>, global: Record<string, stri
|
||||
}
|
||||
|
||||
function isPrivileged(role: string) {
|
||||
return role === 'Admin' || role === 'Owner' || role === 'Super Admin';
|
||||
return role === 'Super Admin';
|
||||
}
|
||||
|
||||
export async function readResourceConfig(name: string): Promise<Record<string, string>> {
|
||||
|
||||
@@ -41,7 +41,7 @@ export async function readSkillDirs(dir: string): Promise<Map<string, string>> {
|
||||
type Scope = 'native' | 'global' | 'user';
|
||||
|
||||
function isPrivileged(role: string) {
|
||||
return role === 'Admin' || role === 'Owner' || role === 'Super Admin';
|
||||
return role === 'Super Admin';
|
||||
}
|
||||
|
||||
function resolveScope(dirName: string, native: Map<string, string>, global: Map<string, string>, user: Map<string, string>): Scope {
|
||||
|
||||
@@ -73,7 +73,7 @@ function resolveFile(name: string, native: Map<string, string>, global: Map<stri
|
||||
}
|
||||
|
||||
function isPrivileged(role: string) {
|
||||
return role === 'Admin' || role === 'Owner' || role === 'Super Admin';
|
||||
return role === 'Super Admin';
|
||||
}
|
||||
|
||||
export const tasksRouter = createRouter();
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
import { createRouter } from '../../create-router';
|
||||
import { readdir, mkdir, rm } from 'node:fs/promises';
|
||||
import { join, dirname } from 'node:path';
|
||||
import { getNativeToolsDir, getGlobalToolsDir, getUserToolsDir } from '../../data-path';
|
||||
|
||||
type Frontmatter = {
|
||||
name: string;
|
||||
description: string;
|
||||
};
|
||||
|
||||
export function parseFrontmatter(raw: string): { frontmatter: Frontmatter; body: string; rawYaml: string } {
|
||||
const match = raw.match(/^---\n([\s\S]*?)\n---\n?([\s\S]*)$/);
|
||||
if (!match) return { frontmatter: { name: '', description: '' }, body: raw, rawYaml: '' };
|
||||
|
||||
const yaml = match[1]!;
|
||||
const body = match[2]!;
|
||||
|
||||
const name = yaml.match(/^name:\s*(.+)$/m)?.[1]?.trim() ?? '';
|
||||
const description = yaml.match(/^description:\s*(.+)$/m)?.[1]?.trim() ?? '';
|
||||
|
||||
return { frontmatter: { name, description }, body, rawYaml: yaml };
|
||||
}
|
||||
|
||||
export async function readToolDirs(dir: string): Promise<Map<string, string>> {
|
||||
const result = new Map<string, string>();
|
||||
try {
|
||||
const entries = await readdir(dir, { withFileTypes: true });
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory()) continue;
|
||||
const toolFile = join(dir, entry.name, 'TOOL.md');
|
||||
if (await Bun.file(toolFile).exists()) {
|
||||
result.set(entry.name, toolFile);
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// directory doesn't exist yet
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
type Scope = 'native' | 'global' | 'user';
|
||||
|
||||
function resolveScope(dirName: string, native: Map<string, string>, global: Map<string, string>, user: Map<string, string>): Scope {
|
||||
if (user.has(dirName)) return 'user';
|
||||
if (global.has(dirName)) return 'global';
|
||||
return 'native';
|
||||
}
|
||||
|
||||
function resolveFile(name: string, native: Map<string, string>, global: Map<string, string>, user: Map<string, string>): { filePath: string; scope: Scope } | null {
|
||||
if (user.has(name)) return { filePath: user.get(name)!, scope: 'user' };
|
||||
if (global.has(name)) return { filePath: global.get(name)!, scope: 'global' };
|
||||
if (native.has(name)) return { filePath: native.get(name)!, scope: 'native' };
|
||||
return null;
|
||||
}
|
||||
|
||||
function isPrivileged(role: string) {
|
||||
return role === 'Super Admin';
|
||||
}
|
||||
|
||||
export const toolsRouter = createRouter();
|
||||
|
||||
toolsRouter.get('/', async (ctx) => {
|
||||
const user = ctx.get('user');
|
||||
const nativeTools = await readToolDirs(getNativeToolsDir());
|
||||
const globalTools = await readToolDirs(getGlobalToolsDir());
|
||||
const userTools = await readToolDirs(getUserToolsDir(user.email));
|
||||
|
||||
const merged = new Map(nativeTools);
|
||||
for (const [name, path] of globalTools) merged.set(name, path);
|
||||
for (const [name, path] of userTools) merged.set(name, path);
|
||||
|
||||
const tools = await Promise.all(
|
||||
Array.from(merged.entries()).map(async ([dirName, filePath]) => {
|
||||
const raw = await Bun.file(filePath).text();
|
||||
const { frontmatter } = parseFrontmatter(raw);
|
||||
const scope = resolveScope(dirName, nativeTools, globalTools, userTools);
|
||||
return {
|
||||
dirName,
|
||||
name: frontmatter.name || dirName,
|
||||
description: frontmatter.description,
|
||||
scope,
|
||||
filePath,
|
||||
};
|
||||
}),
|
||||
);
|
||||
|
||||
return ctx.json(tools);
|
||||
});
|
||||
|
||||
toolsRouter.get('/:name', async (ctx) => {
|
||||
const user = ctx.get('user');
|
||||
const name = ctx.req.param('name');
|
||||
|
||||
const nativeTools = await readToolDirs(getNativeToolsDir());
|
||||
const globalTools = await readToolDirs(getGlobalToolsDir());
|
||||
const userTools = await readToolDirs(getUserToolsDir(user.email));
|
||||
|
||||
const resolved = resolveFile(name, nativeTools, globalTools, userTools);
|
||||
if (!resolved) return ctx.text('Not found', 404);
|
||||
|
||||
const raw = await Bun.file(resolved.filePath).text();
|
||||
const { frontmatter, body, rawYaml } = parseFrontmatter(raw);
|
||||
|
||||
const chatMeta = join(dirname(resolved.filePath), 'chat', 'meta.json');
|
||||
const chatSessionId = await Bun.file(chatMeta).json().then((m: { id: string }) => m.id).catch(() => null);
|
||||
|
||||
return ctx.json({
|
||||
name: frontmatter.name || name,
|
||||
description: frontmatter.description,
|
||||
scope: resolved.scope,
|
||||
body,
|
||||
rawFrontmatter: rawYaml,
|
||||
filePath: resolved.filePath,
|
||||
chatSessionId,
|
||||
});
|
||||
});
|
||||
|
||||
toolsRouter.get('/:name/chat', async (ctx) => {
|
||||
const user = ctx.get('user');
|
||||
const name = ctx.req.param('name');
|
||||
|
||||
const nativeTools = await readToolDirs(getNativeToolsDir());
|
||||
const globalTools = await readToolDirs(getGlobalToolsDir());
|
||||
const userTools = await readToolDirs(getUserToolsDir(user.email));
|
||||
|
||||
const resolved = resolveFile(name, nativeTools, globalTools, userTools);
|
||||
if (!resolved) return ctx.text('Not found', 404);
|
||||
|
||||
const chatDir = join(dirname(resolved.filePath), 'chat');
|
||||
const sessionId = await Bun.file(join(chatDir, 'meta.json')).json().then((m: { id: string }) => m.id).catch(() => null);
|
||||
const messages = await Bun.file(join(chatDir, 'messages.json')).json().catch(() => []);
|
||||
|
||||
return ctx.json({ sessionId, messages });
|
||||
});
|
||||
|
||||
toolsRouter.put('/:name/chat', async (ctx) => {
|
||||
const user = ctx.get('user');
|
||||
const name = ctx.req.param('name');
|
||||
|
||||
const nativeTools = await readToolDirs(getNativeToolsDir());
|
||||
const globalTools = await readToolDirs(getGlobalToolsDir());
|
||||
const userTools = await readToolDirs(getUserToolsDir(user.email));
|
||||
|
||||
const resolved = resolveFile(name, nativeTools, globalTools, userTools);
|
||||
if (!resolved) return ctx.text('Not found', 404);
|
||||
|
||||
if (resolved.scope !== 'user' && !isPrivileged(user.role)) return ctx.text('Forbidden', 403);
|
||||
|
||||
const chatDir = join(dirname(resolved.filePath), 'chat');
|
||||
const { sessionId, messages } = await ctx.req.json<{ sessionId: string; messages: unknown[] }>();
|
||||
|
||||
await mkdir(chatDir, { recursive: true });
|
||||
await Bun.write(join(chatDir, 'messages.json'), JSON.stringify(messages));
|
||||
if (sessionId) await Bun.write(join(chatDir, 'meta.json'), JSON.stringify({ id: sessionId }));
|
||||
|
||||
return ctx.json({ ok: true });
|
||||
});
|
||||
|
||||
toolsRouter.delete('/:name/chat', async (ctx) => {
|
||||
const user = ctx.get('user');
|
||||
const name = ctx.req.param('name');
|
||||
|
||||
const nativeTools = await readToolDirs(getNativeToolsDir());
|
||||
const globalTools = await readToolDirs(getGlobalToolsDir());
|
||||
const userTools = await readToolDirs(getUserToolsDir(user.email));
|
||||
|
||||
const resolved = resolveFile(name, nativeTools, globalTools, userTools);
|
||||
if (!resolved) return ctx.text('Not found', 404);
|
||||
|
||||
if (resolved.scope !== 'user' && !isPrivileged(user.role)) return ctx.text('Forbidden', 403);
|
||||
|
||||
const chatDir = join(dirname(resolved.filePath), 'chat');
|
||||
await rm(chatDir, { recursive: true, force: true });
|
||||
|
||||
return ctx.json({ ok: true });
|
||||
});
|
||||
|
||||
toolsRouter.post('/', async (ctx) => {
|
||||
const user = ctx.get('user');
|
||||
const { name } = await ctx.req.json<{ name: string }>();
|
||||
if (!name?.trim()) return ctx.text('Name is required', 400);
|
||||
|
||||
const dirName = name.trim().toLowerCase().replace(/\s+/g, '-').replace(/[^a-z0-9-]/g, '');
|
||||
if (!dirName) return ctx.text('Invalid name', 400);
|
||||
|
||||
const targetDir = isPrivileged(user.role) ? getGlobalToolsDir() : getUserToolsDir(user.email);
|
||||
const scope: Scope = isPrivileged(user.role) ? 'global' : 'user';
|
||||
|
||||
const dir = join(targetDir, dirName);
|
||||
const filePath = join(dir, 'TOOL.md');
|
||||
|
||||
if (await Bun.file(filePath).exists()) {
|
||||
return ctx.text('Tool already exists', 409);
|
||||
}
|
||||
|
||||
await mkdir(dir, { recursive: true });
|
||||
await Bun.write(filePath, `---\nname: ${name.trim()}\ndescription: \n---\n`);
|
||||
|
||||
return ctx.json({ name: name.trim(), dirName, filePath, scope });
|
||||
});
|
||||
|
||||
toolsRouter.delete('/:name', async (ctx) => {
|
||||
const user = ctx.get('user');
|
||||
const name = ctx.req.param('name');
|
||||
|
||||
const nativeTools = await readToolDirs(getNativeToolsDir());
|
||||
const globalTools = await readToolDirs(getGlobalToolsDir());
|
||||
const userTools = await readToolDirs(getUserToolsDir(user.email));
|
||||
|
||||
const resolved = resolveFile(name, nativeTools, globalTools, userTools);
|
||||
if (!resolved) return ctx.text('Not found', 404);
|
||||
|
||||
if (resolved.scope !== 'user' && !isPrivileged(user.role)) return ctx.text('Forbidden', 403);
|
||||
|
||||
await rm(dirname(resolved.filePath), { recursive: true });
|
||||
return ctx.json({ ok: true });
|
||||
});
|
||||
@@ -10,6 +10,7 @@ import { usersRouter } from './api/users/users-router';
|
||||
import { plansRouter } from './api/plans/plans';
|
||||
import { skillsRouter } from './api/skills/skills';
|
||||
import { tasksRouter } from './api/tasks/tasks';
|
||||
import { toolsRouter } from './api/tools/tools';
|
||||
import { processesRouter } from './api/processes/processes';
|
||||
import { sessionsRouter } from './api/sessions/sessions';
|
||||
import { scrapeRouter } from './api/scrape/scrape';
|
||||
@@ -71,6 +72,7 @@ protectedRouter.route('/users', usersRouter);
|
||||
protectedRouter.route('/plans', plansRouter);
|
||||
protectedRouter.route('/skills', skillsRouter);
|
||||
protectedRouter.route('/tasks', tasksRouter);
|
||||
protectedRouter.route('/tools', toolsRouter);
|
||||
protectedRouter.route('/processes', processesRouter);
|
||||
protectedRouter.route('/', sessionsRouter);
|
||||
protectedRouter.route('/scrape', scrapeRouter);
|
||||
|
||||
Reference in New Issue
Block a user