中文

mcp-server-requests

An MCP server that provides HTTP request capabilities, enabling LLMs to fetch and process web content.

Features

• fetch: Web content fetching tool
  • Supports converting web content to Markdown format
  • Supports filtering web content (non-visual elements like script, style, meta)
  • Feeds into LLM context, reducing token consumption through Markdown conversion or filtering non-visual elements
• fetch_to_file: Web content saving tool
  • Web content is not fed into LLM context, avoiding excessive token consumption
• http_request: Generic HTTP request tool
  • Full HTTP method support (GET, POST, PUT, PATCH, DELETE)
  • Supports custom request headers
  • Supports text and JSON request bodies
  • Returns complete HTTP response information (status, headers, body) for LLMs
• Supports custom User-Agent
• Supports random User-Agent generation
• Supports using MCP Root capability, controlled by `--use-root`, usable with MCP Clients that have `root` capability

Installation

git clone https://github.com/coucya/mcp-server-requests.git
cd mcp-server-requests
pip install .

Usage

MCP Server Configuration

{
    "mcpServers": {
        "mcp-server-requests": {
            "command": "python",
            "args": [
                "-m",
                "mcp_server_requests"
            ]
        }
    }
}

Command Line

0. Start MCP Server

Start the MCP server directly:

python -m mcp_server_requests

Options

• `--user-agent TEXT`: Specify custom User-Agent string
• `--random-user-agent [browser=xxx;os=xxx]`: Use randomly generated User-Agent
• `--force-user-agent`: Force using command line specified User-Agent, ignoring LLM provided UA
• `--list-os-and-browser`: List available browsers and OS for random User-Agent generation
• `--use-root`: Enable workspace root support for file operations (requires MCP Client with `root` capability support)
• `--allow-external-file-access`: Allow file operations outside workspace (only with --use-root)

Option Details

• `--user-agent` and `--random-user-agent` are mutually exclusive and cannot be used together
• User-Agent setup methods:
  • Custom string: `--user-agent "Mozilla/5.0 (...)"`
  • Fully random: `--random-user-agent`
  • Conditional random generation:
    • Specify browser type: `--random-user-agent browser=chrome`
    • Specify OS: `--random-user-agent os=windows`
    • Both browser and OS: `--random-user-agent browser=chrome;os=windows`
    • Note: Browser and OS parameters are case insensitive

• Use `--list-os-and-browser` to view available browsers and OS for `--random-user-agent`.

• `--force-user-agent` option controls User-Agent priority:
  • When `--force-user-agent` is enabled: Force using command-line specified User-Agent (via `--user-agent` or `--random-user-agent`)
  • When `--force-user-agent` is disabled:
    • If LLM provides User-Agent in `headers`, use the LLM-provided one
    • Otherwise use command-line specified User-Agent
  • If none specified, use default User-Agent: `Mozilla/5.0 (compatible; mcp-server-requests/{version})`

• `--use-root` enables file operations relative to workspace root:
  • Prerequisite: Requires MCP Client with `root` capability support
  • When enabled: `fetch_to_file` allows absolute paths or relative paths, relative paths are relative to workspace root
  • When disabled: `fetch_to_file` must use absolute paths, files are saved to specified absolute path locations
• `--allow-external-file-access`: Controls whether file operations outside workspace are allowed (only usable with --use-root)
  • Prerequisite: Requires `--use-root` to be enabled, this option has no effect if `--use-root` is not enabled
  • When enabled: `fetch_to_file` can operate on files outside the workspace directory
  • When disabled: `fetch_to_file` can only operate on files within the workspace directory, even if absolute paths are provided, files outside the workspace cannot be operated on

1. fetch - Fetch Web Content

The fetch subcommand is equivalent to the MCP `fetch` tool functionality, used to demonstrate fetch capabilities.

python -m mcp_server_requests fetch  [--return-content {raw,basic_clean,strict_clean,markdown}]

Options:

• `--return-content`: Return content type (default: markdown)
  • raw: Return raw unprocessed HTML content (includes response headers)
  • basic_clean: Basic cleanup, removing non-display tags (script, style, meta, etc.) while preserving structure
  • strict_clean: Strict cleanup, removing non-display tags and most HTML attributes, keeping only essential structure
  • markdown: Convert HTML to clean Markdown format

Functionality

MCP Tool Details

1. fetch - Fetch Web Content

Fetch content from the specified URL and return it in the specified format.

Parameters:

• url (required, string): Target URL, supports HTTP/HTTPS protocols
• return_content (optional, enum): Content processing format, defaults to "markdown"
  • `"raw"`: Return raw HTML content with complete HTTP response headers
  • `"basic_clean"`: Basic cleanup, removing non-display tags (script, style, meta, etc.) while preserving HTML structure
  • `"strict_clean"`: Strict cleanup, removing non-display tags and most HTML attributes, keeping only essential structure
  • `"markdown"`: Convert HTML content to clean Markdown format

Returns: Content processed according to the specified format

2. fetch_to_file - Fetch Web Content and Save to File

Fetch content from the specified URL and save it to a file.

Parameters:

• url (required, string): Target URL
• file_path (required, string): File save path
  • When `--use-root` is enabled: Must be a relative path (relative to workspace root)
  • When `--use-root` is disabled: Must be an absolute path
• return_content (optional, enum): Content processing format, defaults to "markdown", same options as fetch tool

Features:

• Automatically creates parent directories for nested paths
• All files saved with UTF-8 encoding
• When `--use-root` is enabled, can work with `--allow-external-file-access` to control file access scope

Returns: Operation result or error message

3. http_request - Generic HTTP Request Tool

Send HTTP requests using any method and get complete responses.

Parameters:

• url (required, string): Target URL
• method (optional, enum): HTTP method, defaults to "GET"
  • `"GET"`, `"POST"`, `"PUT"`, `"PATCH"`, `"DELETE"`
• query (optional, object): URL query parameters as key-value pairs, automatically URL-encoded
• headers (optional, object): Custom HTTP request headers
• data (optional, string): Text format request body data, mutually exclusive with json parameter
• json (optional, any type): JSON format request body data, mutually exclusive with data parameter

Parameter Constraints:

• `data` and `json` parameters cannot be used together
• When using `json` parameter, `Content-Type: application/json` is automatically set
• When using `data` parameter, it's recommended to manually set appropriate `Content-Type`

Returns: Complete HTTP response (status line, response headers, and response body)

Note: Individual HTTP method tools (http_get, http_post, http_put, http_patch, http_delete) have been removed and replaced by the unified http_request tool.

License

MIT