Web fetch

web_fetch

Web fetch tool — anti-bot web page fetching via Scrapling.

Provides three fetching modes: - basic: Fast HTTP requests with TLS fingerprinting. - dynamic: Full browser automation via Playwright/Chromium. - stealth: Anti-bot bypass with modified Firefox (Camoufox).

Scrapling is an optional dependency. If not installed, the tool returns a clear error message.

WebFetchTool

AXMTool-compatible wrapper around :func:fetch_page.

The MCP wrapping layer (register_one) invokes tools through a synchronous execute(**kwargs) seam, while the central fetching logic in :func:fetch_page is async. This wrapper bridges the two via :func:asyncio.run without duplicating any business logic — all fetching, mode dispatch, truncation and error handling stay in :func:fetch_page.

Source code in packages/axm-mcp/src/axm_mcp/web_fetch.py

class WebFetchTool:
    """AXMTool-compatible wrapper around :func:`fetch_page`.

    The MCP wrapping layer (``register_one``) invokes tools through a
    synchronous ``execute(**kwargs)`` seam, while the central fetching
    logic in :func:`fetch_page` is async. This wrapper bridges the two
    via :func:`asyncio.run` without duplicating any business logic — all
    fetching, mode dispatch, truncation and error handling stay in
    :func:`fetch_page`.
    """

    agent_hint = (
        "Fetch a web page with optional anti-bot bypass "
        "(modes: auto, basic, dynamic, stealth). Returns title, text "
        "and status_code, or a structured error."
    )

    @property
    def name(self) -> str:
        """Tool name used for MCP registration."""
        return "web_fetch"

    def execute(self, *, url: str, mode: str = "auto", **_: object) -> ToolResult:
        """Fetch a web page, delegating to :func:`fetch_page`.

        Args:
            url: URL to fetch (required).
            mode: Fetching mode — ``auto``, ``basic``, ``dynamic``,
                or ``stealth``. Defaults to ``auto``.
        """
        result = asyncio.run(fetch_page(url=url, mode=mode))
        success = bool(result.get("success", False))
        error = result.get("error")
        data = {k: v for k, v in result.items() if k not in {"success", "error"}}
        return ToolResult(
            success=success,
            data=data,
            error=str(error) if error is not None else None,
        )

name property

Tool name used for MCP registration.

execute(*, url, mode='auto', **_)

Fetch a web page, delegating to :func:fetch_page.

Parameters:

Name	Type	Description	Default
url	str	URL to fetch (required).	required
mode	str	Fetching mode — auto, basic, dynamic, or stealth. Defaults to auto.	'auto'

Source code in packages/axm-mcp/src/axm_mcp/web_fetch.py

def execute(self, *, url: str, mode: str = "auto", **_: object) -> ToolResult:
    """Fetch a web page, delegating to :func:`fetch_page`.

    Args:
        url: URL to fetch (required).
        mode: Fetching mode — ``auto``, ``basic``, ``dynamic``,
            or ``stealth``. Defaults to ``auto``.
    """
    result = asyncio.run(fetch_page(url=url, mode=mode))
    success = bool(result.get("success", False))
    error = result.get("error")
    data = {k: v for k, v in result.items() if k not in {"success", "error"}}
    return ToolResult(
        success=success,
        data=data,
        error=str(error) if error is not None else None,
    )

fetch_page(*, url, mode='auto') async

Fetch a web page with optional anti-bot bypass.

Uses Scrapling as backend with automatic escalation.

Parameters:

Name	Type	Description	Default
url	str	URL to fetch (required).	required
mode	str	Fetching mode — auto, basic, dynamic, or stealth. Defaults to auto (same as basic).	'auto'

Returns:

Type	Description
FetchResult	Dict with success, url, title, text,
FetchResult	and status_code on success.
FetchResult	Dict with success=False and error on failure.

Source code in packages/axm-mcp/src/axm_mcp/web_fetch.py

async def fetch_page(
    *,
    url: str,
    mode: str = "auto",
) -> FetchResult:
    """Fetch a web page with optional anti-bot bypass.

    Uses Scrapling as backend with automatic escalation.

    Args:
        url: URL to fetch (required).
        mode: Fetching mode — ``auto``, ``basic``, ``dynamic``,
            or ``stealth``. Defaults to ``auto`` (same as ``basic``).

    Returns:
        Dict with ``success``, ``url``, ``title``, ``text``,
        and ``status_code`` on success.
        Dict with ``success=False`` and ``error`` on failure.
    """
    if not _HAS_SCRAPLING:
        logger.error(
            "scrapling is not installed — install with: "
            "pip install 'scrapling[fetchers]'",
        )
        return {
            "success": False,
            "error": (
                "scrapling is not installed. "
                "Install with: pip install 'scrapling[fetchers]'"
            ),
        }

    try:
        match mode:
            case "auto" | "basic":
                page = Fetcher.get(url)
            case "dynamic":
                page = await DynamicFetcher.async_fetch(url)
            case "stealth":
                page = await StealthyFetcher.async_fetch(url)
            case _:
                return {
                    "success": False,
                    "error": (
                        f"Unknown mode: {mode!r}. Use: auto, basic, dynamic, stealth."
                    ),
                }

        title: str = page.css("title::text").get() or ""
        text: str = page.get_all_text() or ""
        if len(text) > _MAX_TEXT_CHARS:
            text = text[:_MAX_TEXT_CHARS] + "\n... [truncated]"
        status: int = getattr(page, "status", 200)

        return {
            "success": True,
            "url": url,
            "title": title,
            "text": text,
            "status_code": status,
            "mode": mode,
        }
    except Exception as exc:  # noqa: BLE001
        logger.warning("web_fetch failed for %s: %s", url, exc)
        return {
            "success": False,
            "error": str(exc),
            "url": url,
            "mode": mode,
        }