apify tool, tools API + automation UI, integrations config, super admin restrictions
- apify tool: TOOL.md definition, index.ts implementation with auto-auth via OFFICER_APIFY_TOKEN, output_path for large datasets - tools API: /tools routes (list, detail, chat, create, delete) mirroring tasks pattern - automation UI: tools tab in sidebar, NewTool component, tool detail view - apify integration: settings page for enterprise API key config, pi-bridge passes env var to containers - tiktok-trends task: rewritten as agent instructions using apify tool with output_path, scripted report generation for 50KB read limit - restrict edit/delete of native/global capabilities to Super Admin only (backend + frontend) - tools authoring guide: TOOLS.md with full spec for TOOL.md frontmatter, index.ts execute signature, patterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.6
+142
-129
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: TikTok Trends
|
||||
description: Fetch top trending TikTok videos for a given country and generate an engagement report with optional video downloads.
|
||||
version: 2
|
||||
version: 3
|
||||
author: pastilhas
|
||||
tags:
|
||||
- social-media
|
||||
@@ -9,17 +9,9 @@ tags:
|
||||
- trends
|
||||
- apify
|
||||
- content-analysis
|
||||
skills:
|
||||
- Apify
|
||||
tools:
|
||||
- apify
|
||||
dependencies:
|
||||
- name: tiktok-trends-script
|
||||
description: The main script at $OFFICER_USER_ROOT/../resources/scripts/apify/tiktok-trends.ts
|
||||
check_command: test -f "$OFFICER_USER_ROOT/../resources/scripts/apify/tiktok-trends.ts"
|
||||
optional: false
|
||||
- name: bun
|
||||
description: Required to run the TypeScript script
|
||||
check_command: bun --version
|
||||
optional: false
|
||||
- name: yt-dlp
|
||||
description: Required only when download option is enabled. Downloads TikTok videos.
|
||||
check_command: yt-dlp --version
|
||||
@@ -103,26 +95,15 @@ inputs:
|
||||
type: boolean
|
||||
default: false
|
||||
required: false
|
||||
- name: api_token
|
||||
description: Apify API token with access to actors. Get yours at https://console.apify.com/account/integrations
|
||||
type: string
|
||||
required: true
|
||||
sensitive: true
|
||||
outputs:
|
||||
- name: engagement_report
|
||||
description: Markdown report with trending videos analysis, engagement metrics, top hashtags, sounds, and creators.
|
||||
path: tiktok_trends_<country>_<timestamp>/report-YYYY-MM-DD.md
|
||||
path: tiktok_trends_<country>_<timestamp>/report.md
|
||||
- name: raw_data
|
||||
description: Raw JSON data from Apify actor containing all video metadata.
|
||||
path: tiktok_trends_<country>_<timestamp>/raw-YYYY-MM-DD.json
|
||||
- name: execution_log
|
||||
description: Complete script execution log with stdout and stderr for debugging.
|
||||
path: tiktok_trends_<country>_<timestamp>/run.log
|
||||
- name: summary
|
||||
description: Human-readable summary of the run including cost analysis and file listing.
|
||||
path: tiktok_trends_<country>_<timestamp>/report.md
|
||||
path: tiktok_trends_<country>_<timestamp>/raw.json
|
||||
- name: videos
|
||||
description: Downloaded video files with metadata (only present if download option was enabled).
|
||||
description: Downloaded video files (only present if download option was enabled).
|
||||
path: tiktok_trends_<country>_<timestamp>/videos/
|
||||
optional: true
|
||||
config:
|
||||
@@ -134,125 +115,157 @@ config:
|
||||
|
||||
Fetch top trending TikTok videos for a given country and generate a comprehensive engagement report.
|
||||
|
||||
## Overview
|
||||
## Important
|
||||
|
||||
This task uses the Apify `novi~fast-tiktok-scraper` actor to fetch trending videos from TikTok for a specified country. It generates:
|
||||
- **Engagement metrics** (total/average views, likes, shares, comments)
|
||||
- **Top hashtags** found in trending content
|
||||
- **Popular sounds** being used
|
||||
- **Creator analysis** (who appears multiple times in trending)
|
||||
- **Complete video list** with links and stats
|
||||
- Use the `apify` tool to fetch data. Do NOT call the Apify REST API directly via curl or fetch.
|
||||
- Do NOT explore or list files before starting. Create the output directory, call the tool, process results.
|
||||
- If `download` is false (default), do NOT attempt to download any videos.
|
||||
- Large data files (like `raw.json`) exceed the 50KB read limit. Never try to read them directly. Instead, write a Node.js script to process and transform the data, execute it, then delete the script.
|
||||
|
||||
Optionally downloads videos using yt-dlp for offline analysis.
|
||||
## Steps
|
||||
|
||||
## Pre-execution Checks
|
||||
### 1. Setup output directory
|
||||
|
||||
1. **Validate inputs**:
|
||||
- `api_token` must be provided (required)
|
||||
- `country` must be a valid 2-letter country code
|
||||
- `limit` must be between 1-100
|
||||
|
||||
2. **Check dependencies**:
|
||||
- Verify script exists at `$OFFICER_USER_ROOT/../resources/scripts/apify/tiktok-trends.ts`
|
||||
- Verify bun is installed
|
||||
- If `download=true`, verify yt-dlp is installed
|
||||
|
||||
3. **Test Apify token**:
|
||||
- Make a test call to `GET https://api.apify.com/v2/users/me` to validate the token
|
||||
- Abort with clear error if token is invalid or account has no credits
|
||||
|
||||
## Execution
|
||||
|
||||
Run the script with the following command:
|
||||
|
||||
```bash
|
||||
bun "$OFFICER_USER_ROOT/../resources/scripts/apify/tiktok-trends.ts" \
|
||||
--apikey "<api_token>" \
|
||||
--country "<country>" \
|
||||
--limit <limit> \
|
||||
--output "<output_dir>" \
|
||||
<download_flag>
|
||||
Create a timestamped output directory:
|
||||
```
|
||||
$HOME/tiktok-trends/tiktok_trends_<country>_<YYYYMMDD_HHMMSS>/
|
||||
```
|
||||
|
||||
Where:
|
||||
- `<download_flag>` is `--download` if `download=true`, otherwise omitted
|
||||
- `<output_dir>` is a timestamped directory created inside `$HOME/tiktok-trends/` (the logged-in user's home directory)
|
||||
### 2. Fetch trending videos
|
||||
|
||||
**Pipe output to log**:
|
||||
```bash
|
||||
<command_above> 2>&1 | tee "<output_dir>/run.log"
|
||||
Call the `apify` tool with `output_path` pointing to `raw.json` in the output directory:
|
||||
```
|
||||
apify(
|
||||
actor_id: "novi~fast-tiktok-scraper",
|
||||
input: { "type": "TREND", "region": "<country>", "maxItems": <limit> },
|
||||
output_path: "<output_dir>/raw.json"
|
||||
)
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
The tool saves the full dataset to `raw.json` and returns a summary (item count). If it returns an error or 0 items, report the error and stop.
|
||||
|
||||
The task is considered successful when:
|
||||
- Script exits with code 0
|
||||
- `report-YYYY-MM-DD.md` exists and is non-empty
|
||||
- `raw-YYYY-MM-DD.json` exists and contains valid JSON array
|
||||
- (If download enabled) `videos/` directory exists with at least one file
|
||||
### 3. Generate engagement report
|
||||
|
||||
**Important:** The raw JSON file is too large to read directly (exceeds the 50KB read limit). Instead, write a Node.js script (e.g. `generate-report.js`) in the output directory that reads `raw.json`, processes the data, and writes `report.md`. Then execute it with `node generate-report.js`. Delete the script after it runs successfully.
|
||||
|
||||
The script should read `raw.json`, parse it as a JSON array of video items, and write `report.md` with the following sections:
|
||||
|
||||
#### Header
|
||||
```markdown
|
||||
# TikTok Trending Report — <COUNTRY> — YYYY-MM-DD
|
||||
|
||||
Total videos analyzed: <count>
|
||||
```
|
||||
|
||||
#### Engagement Summary
|
||||
|
||||
Build a table from each video's `statistics` object (`play_count`, `digg_count`, `share_count`, `comment_count`):
|
||||
|
||||
| Metric | Total | Avg per video |
|
||||
|--------|------:|-------------:|
|
||||
| Views | ... | ... |
|
||||
| Likes | ... | ... |
|
||||
| Shares | ... | ... |
|
||||
| Comments | ... | ... |
|
||||
|
||||
#### Top Hashtags (up to 20)
|
||||
|
||||
Extract hashtags from each video's `text_extra` array (entries where `hashtag_name` is set). If `text_extra` is empty, fall back to parsing `#tags` from the `desc` field. Count occurrences, sort descending.
|
||||
|
||||
| Hashtag | Count |
|
||||
|---------|------:|
|
||||
|
||||
#### Top Sounds (up to 10)
|
||||
|
||||
From each video's `music` object, format as `title — author`. Count occurrences, sort descending.
|
||||
|
||||
| Sound | Count |
|
||||
|-------|------:|
|
||||
|
||||
#### Creators Appearing in Trending (up to 10)
|
||||
|
||||
From each video's `author.unique_id`. Count occurrences, sort descending.
|
||||
|
||||
| Creator | Videos |
|
||||
|---------|-------:|
|
||||
|
||||
#### Video List
|
||||
|
||||
Full table of all videos, sorted by position:
|
||||
|
||||
| # | Creator | Description | Views | Likes | URL |
|
||||
|--:|---------|-------------|------:|------:|-----|
|
||||
|
||||
- Creator: `@author.unique_id`
|
||||
- Description: first 60 chars of `desc`, pipe and newline characters replaced, with `...` if truncated
|
||||
- URL: `share_url`
|
||||
- Format numbers with locale separators (e.g. `1,234,567`)
|
||||
|
||||
### 4. Download videos (only if `download` is true)
|
||||
|
||||
If `download` is false, skip this step entirely.
|
||||
|
||||
If `download` is true:
|
||||
1. Check that `yt-dlp` is installed
|
||||
2. Create a `videos/` subdirectory in the output directory
|
||||
3. Write all `share_url` values to a `urls.txt` file
|
||||
4. Run: `yt-dlp -a urls.txt -o "videos/%(id)s.%(ext)s" --write-info-json --no-overwrites`
|
||||
5. Report how many videos were downloaded. Partial failures are acceptable — do not fail the task if some downloads fail.
|
||||
|
||||
### 5. Report results
|
||||
|
||||
Print a summary to the user:
|
||||
- Country and date
|
||||
- Number of videos fetched
|
||||
- Top video: `@creator` — views count — URL
|
||||
- Top hashtag and count
|
||||
- Output directory path
|
||||
- Files created and their sizes
|
||||
|
||||
## Data Shape Reference
|
||||
|
||||
Each video item from the actor has this structure:
|
||||
```json
|
||||
{
|
||||
"aweme_id": "string",
|
||||
"desc": "video description with #hashtags",
|
||||
"create_time": 1234567890,
|
||||
"share_url": "https://www.tiktok.com/@user/video/123",
|
||||
"author": {
|
||||
"unique_id": "username",
|
||||
"nickname": "Display Name",
|
||||
"uid": "123"
|
||||
},
|
||||
"statistics": {
|
||||
"play_count": 1000000,
|
||||
"digg_count": 50000,
|
||||
"share_count": 5000,
|
||||
"comment_count": 2000,
|
||||
"collect_count": 1000,
|
||||
"download_count": 500
|
||||
},
|
||||
"music": {
|
||||
"title": "Sound Name",
|
||||
"author": "Sound Author"
|
||||
},
|
||||
"text_extra": [
|
||||
{ "hashtag_name": "trending", "type": 1 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Scenario | Action |
|
||||
|----------|--------|
|
||||
| Script exits non-zero | Check `run.log` for Apify/API errors. Common causes: invalid token, no credits, rate limits |
|
||||
| Empty results | Report "No trending videos found" but mark as success if files were created |
|
||||
| Partial results | Report success if at least 1 video returned, note the discrepancy |
|
||||
| Timeout (>10 min) | Kill process, report timeout. Check Apify actor status in console |
|
||||
| Download failures | Report which videos failed but mark task as success if main report generated |
|
||||
|
||||
## Post-execution
|
||||
|
||||
After successful execution:
|
||||
|
||||
1. **Verify outputs**:
|
||||
- Check all expected files exist
|
||||
- Validate JSON is parseable
|
||||
- If download enabled, count video files
|
||||
|
||||
2. **Generate summary** (`report.md`):
|
||||
```markdown
|
||||
# TikTok Trends Run Summary
|
||||
|
||||
## Run Details
|
||||
- **Date**: YYYY-MM-DD
|
||||
- **Timestamp**: HH:MM:SS
|
||||
- **Apify Actor**: novi~fast-tiktok-scraper
|
||||
- **Country**: <country_name> (<country_code>)
|
||||
- **Requested**: <limit> videos
|
||||
- **Received**: <actual_count> videos
|
||||
- **Duration**: <X> minutes <Y> seconds
|
||||
|
||||
## Cost Analysis
|
||||
- **Apify Credits Used**: <cost> USD
|
||||
- **Cost per Video**: $<cost/actual_count>
|
||||
|
||||
## Key Findings
|
||||
- **Top Video**: @<creator> - <views> views
|
||||
<description_preview>
|
||||
URL: <url>
|
||||
|
||||
- **Top Hashtag**: #<hashtag> (<count> occurrences)
|
||||
- **Top Sound**: <sound_name> (<count> uses)
|
||||
|
||||
## Output Files
|
||||
- `report-YYYY-MM-DD.md` (<size>) - Engagement report
|
||||
- `raw-YYYY-MM-DD.json` (<size>) - Raw API data
|
||||
- `run.log` (<size>) - Execution log
|
||||
- `videos/` (<count> files, <total_size>) - Downloaded videos (if enabled)
|
||||
|
||||
## Notes
|
||||
- <any_warnings_or_errors_from_log>
|
||||
```
|
||||
|
||||
3. **Cleanup**:
|
||||
- Remove temporary files if any
|
||||
- Report final status to user
|
||||
| Apify tool returns error | Report the error message. Common causes: invalid token, no credits, rate limits |
|
||||
| Empty results | Report "No trending videos found for <country>" and stop |
|
||||
| Partial results (fewer than requested) | Proceed normally, note the discrepancy in the summary |
|
||||
| yt-dlp not installed when download=true | Report that yt-dlp is required and skip downloads |
|
||||
| Download failures | Report which videos failed but do not fail the task |
|
||||
|
||||
## Notes
|
||||
|
||||
- **Rate limiting**: The Apify actor may take 1-3 minutes depending on the limit
|
||||
- **Costs**: Each run uses Apify compute units. Monitor usage at https://console.apify.com/billing
|
||||
- **Data freshness**: Trends data is near real-time but may have slight delays
|
||||
- **Video downloads**: Downloading many videos can take significant time and storage
|
||||
- **Country availability**: Not all countries have sufficient trending data; some may return fewer results than requested
|
||||
- The Apify actor may take 1-3 minutes depending on the limit
|
||||
- Each run uses Apify compute units — monitor at https://console.apify.com/billing
|
||||
- Not all countries have sufficient trending data; some may return fewer results than requested
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
# Tools
|
||||
|
||||
A tool is a callable capability that agents can use during task execution. Each tool lives in its own directory under `tools/` and is defined by a `TOOL.md` file and an `index.ts` (or `index.js`) entry point.
|
||||
|
||||
Tools are loaded by the `tool-loader` extension at startup and registered as callable functions. They appear in the agent's system prompt and can be invoked by name.
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
tools/
|
||||
<tool-name>/
|
||||
TOOL.md # Metadata and documentation
|
||||
index.ts # Implementation (required)
|
||||
```
|
||||
|
||||
Both files are required. The loader skips directories missing either `TOOL.md` or an entry file.
|
||||
|
||||
## TOOL.md Format
|
||||
|
||||
A tool file has two parts: **frontmatter** (YAML metadata) and **body** (Markdown documentation).
|
||||
|
||||
### Frontmatter
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: tool_name
|
||||
label: Tool Name
|
||||
description: What the tool does and when the agent should use it.
|
||||
version: 1
|
||||
language: typescript
|
||||
inputs:
|
||||
param_name:
|
||||
type: string
|
||||
description: What this parameter is for.
|
||||
optional_param:
|
||||
type: number
|
||||
description: An optional parameter.
|
||||
optional: true
|
||||
secret_param:
|
||||
type: string
|
||||
description: A sensitive value (e.g., API token).
|
||||
optional: true
|
||||
sensitive: true
|
||||
mode:
|
||||
type: enum
|
||||
values: single,batch
|
||||
description: Choose between modes.
|
||||
---
|
||||
```
|
||||
|
||||
#### Fields
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `name` | string | yes | Identifier for the tool (used in tool calls). Use `snake_case`. |
|
||||
| `label` | string | no | Human-readable display name. Defaults to `name` if omitted. |
|
||||
| `description` | string | yes | What the tool does. This appears in the agent's system prompt — make it specific enough for the agent to know when to use it. |
|
||||
| `version` | integer | no | Version number. Used by `sync-tools` to detect updates — bump when changing the tool. |
|
||||
| `language` | string | no | Implementation language: `typescript`, `bash`, or `python`. Defaults to `typescript`. |
|
||||
| `inputs` | object | no | Input parameters the tool accepts. Keys are parameter names. |
|
||||
|
||||
#### Input Fields
|
||||
|
||||
Each input is a key under `inputs:` with these properties:
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `type` | string | yes | Parameter type: `string`, `number`, `boolean`, or `enum`. |
|
||||
| `description` | string | yes | What this parameter is for. Shown to the agent. |
|
||||
| `optional` | boolean | no | Whether the parameter is optional. Defaults to required. |
|
||||
| `sensitive` | boolean | no | Mark sensitive values (tokens, passwords). Prevents logging. |
|
||||
| `values` | string | no | Comma-separated allowed values when `type` is `enum`. |
|
||||
| `default` | any | no | Default value if not provided. |
|
||||
|
||||
### Body
|
||||
|
||||
The body is Markdown documentation that the agent sees when the tool is loaded. Include:
|
||||
|
||||
- **Title** — `# Tool Name`
|
||||
- **Authentication** — How credentials are resolved (env vars, integrations, etc.)
|
||||
- **Usage** — How to call the tool and what parameters to pass.
|
||||
- **Examples** — Common usage patterns.
|
||||
- **Error Handling** — What errors can occur and what they mean.
|
||||
- **Notes** — Limits, billing, external links.
|
||||
|
||||
## index.ts Format
|
||||
|
||||
The entry file must export an `execute` function:
|
||||
|
||||
```typescript
|
||||
type ToolResult = {
|
||||
content: Array<{ type: string; text: string }>;
|
||||
isError?: boolean;
|
||||
};
|
||||
|
||||
type OnUpdate = (partial: { content: Array<{ type: string; text: string }> }) => void;
|
||||
|
||||
export async function execute(
|
||||
toolCallId: string,
|
||||
params: Record<string, unknown>,
|
||||
signal: AbortSignal | undefined,
|
||||
onUpdate?: OnUpdate,
|
||||
): Promise<ToolResult> {
|
||||
// Implementation here
|
||||
}
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
| Parameter | Description |
|
||||
|-----------|-------------|
|
||||
| `toolCallId` | Unique ID for this tool call. |
|
||||
| `params` | Input values from the agent, matching the `inputs` defined in TOOL.md. |
|
||||
| `signal` | AbortSignal for cancellation. |
|
||||
| `onUpdate` | Callback for streaming progress updates to the agent during long operations. |
|
||||
|
||||
### Return Value
|
||||
|
||||
Return a `ToolResult` object:
|
||||
- `content` — Array of content blocks. Usually one `{ type: 'text', text: '...' }`.
|
||||
- `isError` — Set `true` to indicate failure. The agent sees the error and can react.
|
||||
|
||||
### Progress Updates
|
||||
|
||||
Use `onUpdate` to stream status during long-running operations:
|
||||
|
||||
```typescript
|
||||
onUpdate?.({ content: [{ type: 'text', text: 'Processing step 2 of 5...' }] });
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
Tools should resolve credentials internally, not require the agent to pass them. Pattern:
|
||||
|
||||
1. Check for an explicit parameter override (e.g., `params.api_token`)
|
||||
2. Fall back to an environment variable (e.g., `process.env.OFFICER_APIFY_TOKEN`)
|
||||
3. Return a helpful error if neither is available
|
||||
|
||||
Environment variables are set by `pi-bridge.ts` from the integration config stored in the database (Settings → Integrations).
|
||||
|
||||
### Runtime Environment
|
||||
|
||||
Tools run inside sandboxed containers using **Node.js** (not Bun). Do not use Bun-specific APIs like `Bun.sleep`, `Bun.file`, etc. Use Node.js equivalents:
|
||||
|
||||
- `setTimeout` / `setInterval` for delays
|
||||
- `fs.readFileSync` / `fs.writeFileSync` for file I/O
|
||||
- `fetch` (available in Node 18+) for HTTP requests
|
||||
|
||||
### Large Output
|
||||
|
||||
If a tool may return large data (e.g., API responses with many items), provide an `output_path` parameter. When set, save the data to the file and return a summary instead:
|
||||
|
||||
```typescript
|
||||
if (params.output_path) {
|
||||
writeFileSync(params.output_path, JSON.stringify(items, null, 2));
|
||||
return { content: [{ type: 'text', text: `${items.length} items saved to ${params.output_path}` }] };
|
||||
}
|
||||
```
|
||||
|
||||
This prevents flooding the agent's context window with raw data.
|
||||
|
||||
## Sync and Discovery
|
||||
|
||||
Tools are synced from `seed/tools/` to `DATA_PATH/tools/` at server startup by `sync-tools.ts`. The sync is version-based — it only overwrites when the seed version is higher than the target version. Always bump `version` in the frontmatter when updating a tool.
|
||||
|
||||
The `tool-loader` extension discovers tools from directories listed in the `PI_TOOLS_DIRS` environment variable (colon-separated). Only `DATA_PATH/tools/` is mounted into containers — `seed/tools/` is not directly accessible at runtime.
|
||||
|
||||
## Existing Tools
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `gmail` | Read Gmail messages, threads, labels via Google API |
|
||||
| `web_search` | Search the web via SearXNG |
|
||||
| `web_fetch` | Fetch and extract content from URLs |
|
||||
| `browser` | Control a Chrome browser via Browser Relay |
|
||||
| `apify` | Run any Apify actor (web scraping, social media data) |
|
||||
| `convert_audio_to_mp3` | Convert audio files to MP3 via ffmpeg |
|
||||
| `ocr` | Optical character recognition on images |
|
||||
| `email_db` | Query the synced email database |
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
name: apify
|
||||
label: Apify
|
||||
version: 4
|
||||
description: Run any Apify actor and return its dataset results. Use for web scraping, data extraction, and automation — TikTok, Twitter, Facebook, Instagram, YouTube, Google, and hundreds more. Authentication is handled automatically when configured in Settings → Integrations.
|
||||
language: typescript
|
||||
inputs:
|
||||
actor_id:
|
||||
type: string
|
||||
description: "Actor ID to run (format: owner~actor-name or owner/actor-name, e.g. 'novi~fast-tiktok-scraper', 'apify/twitter-scraper')"
|
||||
input:
|
||||
type: object
|
||||
description: Actor-specific input parameters as a JSON object (varies per actor)
|
||||
optional: true
|
||||
output_path:
|
||||
type: string
|
||||
description: File path to save JSON results to. When provided, the tool writes the dataset to this file and returns a summary instead of the raw JSON. Recommended for large datasets to avoid flooding the context.
|
||||
optional: true
|
||||
api_token:
|
||||
type: string
|
||||
description: Override the configured API token. Usually not needed — the token is provided automatically from Settings → Integrations → Apify.
|
||||
optional: true
|
||||
sensitive: true
|
||||
timeout_ms:
|
||||
type: number
|
||||
description: Max time to wait for actor completion in milliseconds (default 300000 = 5 min)
|
||||
optional: true
|
||||
poll_interval_ms:
|
||||
type: number
|
||||
description: How often to check run status in milliseconds (default 3000)
|
||||
optional: true
|
||||
---
|
||||
|
||||
# Apify
|
||||
|
||||
Run any actor from the Apify Store, wait for completion, and return the dataset items as JSON.
|
||||
|
||||
## Authentication
|
||||
|
||||
The API token is resolved automatically:
|
||||
1. `api_token` input parameter (explicit override)
|
||||
2. `OFFICER_APIFY_TOKEN` environment variable (set automatically when configured in Settings → Integrations → Apify)
|
||||
|
||||
If neither is available, the tool returns an error prompting the user to configure the integration.
|
||||
|
||||
## Usage
|
||||
|
||||
Just provide the `actor_id` and optional `input`:
|
||||
```
|
||||
apify(actor_id: "novi~fast-tiktok-scraper", input: { type: "TREND", region: "PT", maxItems: 20 })
|
||||
```
|
||||
|
||||
The tool starts the actor, polls until completion, fetches the dataset, and returns all items as JSON.
|
||||
|
||||
## Common Actors
|
||||
|
||||
| Actor | ID | Input example |
|
||||
|-------|----|---------------|
|
||||
| TikTok Scraper | `novi~fast-tiktok-scraper` | `{ type: 'TREND', region: 'US', maxItems: 20 }` |
|
||||
| Twitter Scraper | `apify/twitter-scraper` | `{ searchTerms: ['#ai'], tweetsCount: 100 }` |
|
||||
| Twitter User | `jupri/twitter-user-scraper` | `{ twitterUser: 'username', maxPosts: 50 }` |
|
||||
| Facebook Scraper | `apify/facebook-scraper` | `{ startUrls: ['https://facebook.com/Page'], maxPostsPerPage: 50 }` |
|
||||
| Facebook Search | `jupri/facebook-search-scraper` | `{ searchTerm: 'keyword', maxPosts: 30 }` |
|
||||
| Instagram Hashtag | `apify/instagram-hashtag-scraper` | `{ hashtags: ['travel'], resultsLimit: 50 }` |
|
||||
| Instagram User | `apify/instagram-user-scraper` | `{ usernames: ['natgeo'], resultsLimit: 50 }` |
|
||||
| YouTube Scraper | `apify/youtube-scraper` | `{ searchTerms: ['tutorial'], maxResults: 20 }` |
|
||||
| Google Search | `apify/google-search-scraper` | `{ queries: ['best restaurants lisbon'] }` |
|
||||
|
||||
## Error Handling
|
||||
|
||||
The tool returns clear error messages for:
|
||||
- Missing API token → "Configure it in Settings → Integrations → Apify"
|
||||
- API errors (401, 403, etc.) → includes the HTTP status and response body
|
||||
- Actor failures → includes the actor's `statusMessage`
|
||||
- Timeouts → reports the run ID and last known status
|
||||
|
||||
## Notes
|
||||
|
||||
- Actor IDs use `~` (Apify URL format) or `/` — both work
|
||||
- Browse actors at https://apify.com/store
|
||||
- Monitor usage and credits at https://console.apify.com/billing
|
||||
@@ -0,0 +1,167 @@
|
||||
import { mkdirSync, writeFileSync } from 'node:fs';
|
||||
import { dirname } from 'node:path';
|
||||
|
||||
const BASE = 'https://api.apify.com/v2';
|
||||
|
||||
type RunStatus = 'READY' | 'RUNNING' | 'SUCCEEDED' | 'FAILED' | 'ABORTING' | 'ABORTED' | 'TIMING-OUT' | 'TIMED-OUT';
|
||||
|
||||
type RunData = {
|
||||
id: string;
|
||||
actId: string;
|
||||
status: RunStatus;
|
||||
statusMessage?: string;
|
||||
defaultDatasetId: string;
|
||||
defaultKeyValueStoreId: string;
|
||||
startedAt?: string;
|
||||
finishedAt?: string;
|
||||
};
|
||||
|
||||
type ToolResult = {
|
||||
content: Array<{ type: string; text: string }>;
|
||||
isError?: boolean;
|
||||
};
|
||||
|
||||
type OnUpdate = (partial: { content: Array<{ type: string; text: string }> }) => void;
|
||||
|
||||
type Params = {
|
||||
actor_id: string;
|
||||
input?: Record<string, unknown> | string;
|
||||
api_token?: string;
|
||||
output_path?: string;
|
||||
timeout_ms?: number;
|
||||
poll_interval_ms?: number;
|
||||
};
|
||||
|
||||
function update(onUpdate: OnUpdate | undefined, text: string): void {
|
||||
onUpdate?.({ content: [{ type: 'text', text }] });
|
||||
}
|
||||
|
||||
function resolveToken(params: Params): string | null {
|
||||
if (params.api_token) return params.api_token;
|
||||
return process.env.OFFICER_APIFY_TOKEN ?? null;
|
||||
}
|
||||
|
||||
function apiUrl(path: string, token: string, extra?: Record<string, string>): string {
|
||||
const params = new URLSearchParams({ token, ...extra });
|
||||
return `${BASE}${path}?${params}`;
|
||||
}
|
||||
|
||||
async function apiRequest<T>(url: string, init?: RequestInit): Promise<T> {
|
||||
const res = await fetch(url, init);
|
||||
if (!res.ok) {
|
||||
const body = await res.text();
|
||||
throw new Error(`Apify API error ${res.status}: ${body}`);
|
||||
}
|
||||
return res.json() as Promise<T>;
|
||||
}
|
||||
|
||||
async function startRun(token: string, actorId: string, input: Record<string, unknown>): Promise<RunData> {
|
||||
const { data } = await apiRequest<{ data: RunData }>(apiUrl(`/acts/${actorId}/runs`, token), {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(input),
|
||||
});
|
||||
return data;
|
||||
}
|
||||
|
||||
async function getRun(token: string, runId: string): Promise<RunData> {
|
||||
const { data } = await apiRequest<{ data: RunData }>(apiUrl(`/actor-runs/${runId}`, token));
|
||||
return data;
|
||||
}
|
||||
|
||||
async function getDatasetItems<T>(token: string, datasetId: string): Promise<T[]> {
|
||||
return apiRequest<T[]>(apiUrl(`/datasets/${datasetId}/items`, token, { format: 'json' }));
|
||||
}
|
||||
|
||||
const TERMINAL_STATUSES = new Set<RunStatus>(['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT']);
|
||||
|
||||
export async function execute(
|
||||
_toolCallId: string,
|
||||
params: Params,
|
||||
_signal: AbortSignal | undefined,
|
||||
onUpdate?: OnUpdate,
|
||||
): Promise<ToolResult> {
|
||||
const token = resolveToken(params);
|
||||
if (!token) {
|
||||
return {
|
||||
content: [{ type: 'text', text: 'Apify API token not available. Configure it in Settings → Integrations → Apify, or pass api_token explicitly.' }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
|
||||
const { actor_id } = params;
|
||||
if (!actor_id) {
|
||||
return {
|
||||
content: [{ type: 'text', text: 'actor_id is required.' }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
|
||||
let input: Record<string, unknown> = {};
|
||||
if (params.input) {
|
||||
if (typeof params.input === 'string') {
|
||||
try {
|
||||
input = JSON.parse(params.input);
|
||||
} catch {
|
||||
return {
|
||||
content: [{ type: 'text', text: 'Invalid JSON in input parameter.' }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
} else {
|
||||
input = params.input;
|
||||
}
|
||||
}
|
||||
|
||||
const timeoutMs = params.timeout_ms ?? 300_000;
|
||||
const pollIntervalMs = params.poll_interval_ms ?? 3_000;
|
||||
|
||||
try {
|
||||
update(onUpdate, `Starting actor ${actor_id}...`);
|
||||
const run = await startRun(token, actor_id, input);
|
||||
update(onUpdate, `Run ${run.id} started. Waiting for completion...`);
|
||||
|
||||
const start = Date.now();
|
||||
let finished = run;
|
||||
|
||||
while (!TERMINAL_STATUSES.has(finished.status)) {
|
||||
if (Date.now() - start > timeoutMs) {
|
||||
return {
|
||||
content: [{ type: 'text', text: `Timeout after ${timeoutMs}ms waiting for run ${run.id}. Status: ${finished.status}` }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
await new Promise((r) => setTimeout(r, pollIntervalMs));
|
||||
finished = await getRun(token, run.id);
|
||||
update(onUpdate, `Status: ${finished.status}...`);
|
||||
}
|
||||
|
||||
if (finished.status !== 'SUCCEEDED') {
|
||||
return {
|
||||
content: [{ type: 'text', text: `Actor run ${finished.status}: ${finished.statusMessage ?? 'unknown error'}` }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
|
||||
update(onUpdate, `Run succeeded. Fetching dataset items...`);
|
||||
const items = await getDatasetItems(token, finished.defaultDatasetId);
|
||||
|
||||
if (params.output_path) {
|
||||
mkdirSync(dirname(params.output_path), { recursive: true });
|
||||
writeFileSync(params.output_path, JSON.stringify(items, null, 2));
|
||||
return {
|
||||
content: [{ type: 'text', text: `${items.length} items saved to ${params.output_path}` }],
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
content: [{ type: 'text', text: JSON.stringify(items) }],
|
||||
};
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
return {
|
||||
content: [{ type: 'text', text: `Apify error: ${message}` }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user