skills/docs-manage/SKILL.md (6273 bytes)
1 --- 2 name: docs-manage 3 description: >- 4 Manage the Grounded Docs MCP Server documentation index. Covers scraping 5 and indexing documentation from URLs or local files, refreshing existing 6 indexes with changed content, and removing libraries from the index. 7 Use when you need to add, update, or delete indexed documentation. 8 compatibility: Requires the local `docs-mcp` wrapper on PATH 9 metadata: 10 author: grounded.tools 11 --- 12 13 # Docs Manage 14 15 Index, refresh, and remove library documentation in the local Grounded Docs 16 store. These commands modify the index and produce plain-text status messages 17 on stdout. 18 19 ## When to use 20 21 - A library is not yet indexed and you need its docs available for search. 22 - Documentation may be stale and you want to pull in updated pages. 23 - You want to remove a library or version from the index to free space. 24 25 ## Commands 26 27 ### scrape 28 29 Download and index documentation from a URL or local directory. 30 31 ```bash 32 docs-mcp scrape <library> <url> [options] 33 ``` 34 35 | Flag | Alias | Default | Description | 36 |------|-------|---------|-------------| 37 | `--version <ver>` | `-v` | | Library version label | 38 | `--max-pages <n>` | `-p` | config default | Maximum pages to scrape | 39 | `--max-depth <n>` | `-d` | config default | Maximum navigation depth | 40 | `--max-concurrency <n>` | `-c` | config default | Concurrent page requests | 41 | `--ignore-errors` | | `true` | Continue on individual page errors | 42 | `--scope subpages\|hostname\|domain` | | `subpages` | Crawling boundary | 43 | `--follow-redirects` | | `true` | Follow HTTP redirects | 44 | `--no-follow-redirects` | | | Disable following redirects | 45 | `--scrape-mode auto\|fetch\|playwright` | | `auto` | HTML processing strategy | 46 | `--include-pattern <glob>` | | | URL include pattern (repeatable) | 47 | `--exclude-pattern <glob>` | | | URL exclude pattern (repeatable, takes precedence) | 48 | `--header "Name: Value"` | | | Custom HTTP header (repeatable) | 49 | `--embedding-model <model>` | | | Embedding model configuration | 50 | `--server-url <url>` | | | Remote pipeline worker URL | 51 | `--clean` | | `true` | Clear existing documents before scraping | 52 | `--quiet` | | | Suppress non-error diagnostics | 53 | `--verbose` | | | Enable debug logging | 54 55 Examples: 56 57 ```bash 58 # Scrape React docs, version-tagged 59 docs-mcp scrape react https://react.dev/reference/react --version 19.0.0 60 61 # Scrape local files 62 docs-mcp scrape mylib file:///Users/me/docs/my-library 63 64 # Scrape with depth and page limits 65 docs-mcp scrape nextjs https://nextjs.org/docs --max-pages 200 --max-depth 3 66 67 # Scrape with custom headers (e.g. authentication) 68 docs-mcp scrape internal-api https://docs.internal.com \ 69 --header "Authorization: Bearer tok_xxx" 70 71 # Exclude changelog pages 72 docs-mcp scrape react https://react.dev/reference/react \ 73 --exclude-pattern "**/changelog*" 74 ``` 75 76 Output is a plain-text status line, e.g. `Successfully scraped 42 pages`. 77 Progress updates appear on stderr during the run. 78 79 ### refresh 80 81 Re-scrape an existing library version, skipping unchanged pages via HTTP ETags. 82 83 ```bash 84 docs-mcp refresh <library> [options] 85 ``` 86 87 | Flag | Alias | Description | 88 |------|-------|-------------| 89 | `--version <ver>` | `-v` | Version to refresh (omit for latest) | 90 | `--embedding-model <model>` | | Embedding model configuration | 91 | `--server-url <url>` | | Remote pipeline worker URL | 92 | `--quiet` | | Suppress non-error diagnostics | 93 | `--verbose` | | Enable debug logging | 94 95 Example: 96 97 ```bash 98 docs-mcp refresh react --version 19.0.0 99 ``` 100 101 The library and version must already be indexed. Use `scrape` for first-time 102 indexing. 103 104 ### remove 105 106 Delete a library (or a specific version) from the index. 107 108 ```bash 109 docs-mcp remove <library> [options] 110 ``` 111 112 | Flag | Alias | Description | 113 |------|-------|-------------| 114 | `--version <ver>` | `-v` | Specific version to remove (omit to remove latest) | 115 | `--server-url <url>` | | Remote pipeline worker URL | 116 | `--quiet` | | Suppress non-error diagnostics | 117 | `--verbose` | | Enable debug logging | 118 119 Example: 120 121 ```bash 122 docs-mcp remove react --version 18.3.1 123 ``` 124 125 This is destructive and cannot be undone. Re-run `scrape` to re-index. 126 127 ### export 128 129 Write indexed pages to Markdown files. Each page becomes 130 `<out>/<library>/<version>/<url-path>.md` with front matter (source URL, title). 131 132 ```bash 133 docs-mcp export [library] [options] 134 ``` 135 136 | Flag | Alias | Description | 137 |------|-------|-------------| 138 | `--version <ver>` | `-v` | Version to export (requires a library; omit for all versions) | 139 | `--out <dir>` | | Output directory (defaults to `app.exportDir` config) | 140 | `--server-url <url>` | | Remote pipeline worker URL | 141 | `--quiet` | | Suppress non-error diagnostics | 142 | `--verbose` | | Enable debug logging | 143 144 Example: 145 146 ```bash 147 docs-mcp export react --out ~/docs 148 ``` 149 150 The version directory is replaced on each export. Do not scrape the export 151 directory back into the index. 152 153 ## Output behaviour 154 155 The `scrape`, `refresh`, and `remove` commands write plain-text status messages to **stdout** and 156 diagnostics to **stderr**. The global `--output` flag is accepted but has no 157 effect because the output is plain text, not structured data. 158 159 In non-interactive sessions, diagnostics are suppressed by default. Use 160 `--verbose` (or set `LOG_LEVEL=INFO`) to re-enable them. Use `--quiet` to 161 suppress all non-error diagnostics regardless of session type. 162 163 ## Typical workflow 164 165 ```bash 166 # 1. Index documentation for the first time 167 docs-mcp scrape react https://react.dev/reference/react --version 19.0.0 168 169 # 2. Later, refresh to pick up any changes 170 docs-mcp refresh react --version 19.0.0 171 172 # 3. Clean up old versions 173 docs-mcp remove react --version 18.3.1 174 ``` 175 176 ## Important notes 177 178 - **Scraping can take time.** Large documentation sites with hundreds of pages 179 may run for several minutes. Use `--max-pages` and `--max-depth` to limit 180 scope when you only need a subset. 181 - **Local files** must use the `file://` URL scheme 182 (e.g. `file:///absolute/path/to/docs`). 183 - **`--clean` is on by default** for `scrape`, meaning existing documents for 184 the same library+version are removed before re-indexing. Pass `--no-clean` to 185 append instead. 186 - **`refresh`** only works on previously indexed content. It uses HTTP ETags to 187 skip pages that have not changed, making it much faster than a full re-scrape.