Initial release: airxdb

This commit is contained in:
admin
2026-05-18 11:45:03 +08:00
commit ba540f42ca
9 changed files with 1632 additions and 0 deletions

View File

@@ -0,0 +1,43 @@
{
"name": "airxdb",
"version": "0.2.2",
"description": "Midscene-based GUI debugging workflow for browser, desktop, and remote SSH GUI issues, with screenshot evidence for AirDbg, model-family gates, Computer MCP smoke tests, remote device screenshot setup, Windows screenshot repair, and reports.",
"author": {
"name": "14816",
"email": "noreply@example.com",
"url": "https://airlongdian.fun/plugins/airxdb"
},
"homepage": "https://airlongdian.fun/plugins/airxdb",
"repository": "http://git.airlongdian.fun/admin/airxdb.git",
"license": "MIT",
"keywords": [
"airxdb",
"midscene",
"gui",
"ui-debug",
"playwright",
"mcp"
],
"skills": "./skills/",
"interface": {
"displayName": "AirXDB",
"shortDescription": "Midscene GUI debug workflow with local and remote screenshot evidence",
"longDescription": "AirXDB helps debug graphical interfaces with Midscene.js. It captures screenshot evidence for AirDbg even without model configuration, checks Midscene model and model-family configuration for semantic actions, smoke-tests Computer MCP, uses a remote device helper over SSH for remote GUI evidence and auto-configures missing remote screenshot tools when possible, repairs known Windows screenshot asset gaps, works with AirDbg to reproduce browser, desktop, and remote UI issues visually, generate reports, choose the right Midscene mode, and keep AGENTS.md, ADR, C4 module, and GUI debug logs current.",
"developerName": "14816",
"category": "Productivity",
"capabilities": [
"Interactive",
"Write"
],
"websiteURL": "https://airlongdian.fun/plugins/airxdb",
"privacyPolicyURL": "https://openai.com/policies/row-privacy-policy/",
"termsOfServiceURL": "https://openai.com/policies/row-terms-of-use/",
"defaultPrompt": [
"Use AirXDB with AirDbg to reproduce a browser, desktop, or remote GUI problem visually.",
"Use AirXDB to check the Midscene setup and choose the right GUI automation mode before acting.",
"Use AirXDB to capture screenshot evidence and a reproducible visual report for the current issue."
],
"brandColor": "#0EA5A4",
"screenshots": []
}
}

109
commands/airxdb.md Normal file
View File

@@ -0,0 +1,109 @@
---
description: Enter, exit, inspect, screenshot, remote-screenshot, or smoke-test AirXDB GUI debug mode
argument-hint: [enter|exit|status|screenshot|remote-setup|remote-status|remote-screenshot|smoke]
allowed-tools: [Read, Glob, Grep, Bash, Write, Edit]
---
# /airxdb
控制当前工作区的 AirXDB 图形界面调试模式。
用户传入参数:`$ARGUMENTS`
- `enter` 或空参数:进入 AirXDB 模式并初始化 GUI 调试上下文。
- `status`:检查 AirXDB 状态和关键文件是否存在。
- `screenshot`:连接 Computer MCP 并截图,为 `airdbg` 提供 GUI 错误诊断证据;不要求 Midscene 模型配置。
- `remote-setup`:检查远程设备 SSH 和截图工具;缺失时自动尝试配置远程截图工具。
- `remote-status`:只检查远程设备配置和工具状态,不自动安装。
- `remote-screenshot`:通过 SSH 在远程设备截图,并把截图/JSON 报告拉回当前项目。
- `smoke`:启动 Computer MCP smoke test默认视觉定位 Windows taskbar Start button 并移动鼠标。
- `exit`:退出 AirXDB 模式。
## 执行步骤
1. 解析 `$ARGUMENTS`,默认动作为 `enter`
2. 在当前项目根目录运行:
```bash
python "$HOME/plugins/airxdb/scripts/airxdb_mode.py" --mode <enter|exit|status> --project .
```
如果 `python` 不存在,尝试 `py``python3`
3. `enter` 后必须:
- 加载或初始化 `AirPlan/AGENTS.md`、ADR、C4 module、`AirPlan/docs/debug/gui-debug-log.md`
- 检查输出中的 `midscene_config`
- 如果出现 `midscene_config_required=true``midscene_config=missing:...``midscene_semantic_config=missing:...`,先向用户索取 Midscene 模型配置,再执行视觉语义动作
- 判断是浏览器 GUI 还是桌面 GUI
- 判断是否已有 Playwright
- 选择 Midscene 模式Playwright / Bridge / Computer / MCP
- 与用户确认是否要和 `airdbg` 配合推进
4. 如果 `$ARGUMENTS` 包含 `remote``ssh``远程`,或用户已经说明目标 GUI 在远程设备、测试机、服务器、VM、SSH 主机上,优先运行远程设备 helper
```bash
python "$HOME/plugins/airxdb/scripts/airxdb_remote_device.py" --project . --action <setup|status|screenshot>
```
首次缺少远程配置时helper 会生成 `AirPlan/state/airxdb/remote-device.env.example` 并提示设置 `AIRXDB_REMOTE_SSH_TARGET`。有 SSH 目标后helper 会探测远端截图工具;缺失时自动尝试安装 `scrot`,只使用非交互式 `sudo -n`,无法自动配置时停止并提示用户。
5. `screenshot` 且目标是本机桌面时运行:
```bash
python "$HOME/plugins/airxdb/scripts/airxdb_computer_mcp_smoke.py" --project . --action screenshot
```
截图结果写入 `AirPlan/docs/debug/airxdb-artifacts/`,并应追加到 `AirPlan/docs/debug/gui-debug-log.md`,作为给 `airdbg` 的诊断证据。
6. `smoke` 时运行:
```bash
python "$HOME/plugins/airxdb/scripts/airxdb_computer_mcp_smoke.py" --project . --action mousemove --prompt "Windows taskbar Start button"
```
如果 `python` 不存在,尝试 `py``python3` 或用户提供的 Python 绝对路径。`smoke` 会加载 `AirPlan/state/airxdb/midscene.local.env`,启动 `@midscene/computer-mcp`,必要时修复 Windows 截图脚本缺失,并把截图/JSON 报告写入 `AirPlan/docs/debug/airxdb-artifacts/`
7. GUI 调试过程中必须维护:
- `AGENTS.md`
- `docs/architecture/adr/`
- `docs/architecture/c4/module.md`
- `AirPlan/docs/debug/gui-debug-log.md`
## 输出文案
进入模式:
```text
AirXDB 模式已开启:已初始化或检查 AGENTS.md、ADR、C4 module 和 GUI debug log。若 Midscene 模型配置缺失,请先提供 MIDSCENE_MODEL_NAME、MIDSCENE_MODEL_BASE_URL、MIDSCENE_MODEL_API_KEY、MIDSCENE_MODEL_FAMILY之后请说明目标界面、复现步骤以及你希望使用 Playwright、Bridge、Desktop 还是 MCP 路线。
```
smoke 成功:
```text
AirXDB smoke test 通过Computer MCP 已连接桌面,语义视觉动作完成,截图和 JSON 报告已写入 docs/debug/airxdb-artifacts/。
```
screenshot 成功:
```text
AirXDB 截图取证完成Computer MCP 已连接桌面并保存截图/JSON 报告,可交给 airdbg 做错误诊断。
```
remote-screenshot 成功:
```text
AirXDB 远程截图取证完成:已通过 SSH 获取远程设备截图/JSON 报告,并写入 docs/debug/airxdb-artifacts/,可交给 airdbg 做错误诊断。
```
退出模式:
```text
AirXDB 模式已退出:已返回标准 Codex 流程。
```
状态检查:
```text
AirXDB 状态:<enabled|disabled>
关键文件:逐项列出 ok/missing。
```

View File

@@ -0,0 +1,410 @@
#!/usr/bin/env python3
"""Run a small AirXDB Computer MCP smoke test with screenshot artifacts."""
from __future__ import annotations
import argparse
import base64
import json
import os
import shutil
import socket
import subprocess
import tarfile
import tempfile
import threading
import time
import urllib.error
import urllib.request
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional, Tuple
REQUIRED_BASE_ENV = [
"MIDSCENE_MODEL_NAME",
"MIDSCENE_MODEL_BASE_URL",
"MIDSCENE_MODEL_API_KEY",
]
REQUIRED_SEMANTIC_ENV = [
*REQUIRED_BASE_ENV,
"MIDSCENE_MODEL_FAMILY",
]
VALID_MODEL_FAMILIES = [
"doubao-vision",
"doubao-seed",
"gemini",
"qwen2.5-vl",
"qwen3-vl",
"qwen3.5",
"qwen3.6",
"vlm-ui-tars",
"vlm-ui-tars-doubao",
"vlm-ui-tars-doubao-1.5",
"glm-v",
"auto-glm",
"auto-glm-multilingual",
"gpt-5",
]
def now_stamp() -> str:
return datetime.now().strftime("%Y%m%d-%H%M%S")
def load_env(project_root: Path) -> Dict[str, str]:
env = os.environ.copy()
env_file = project_root / "AirPlan" / "state" / "airxdb" / "midscene.local.env"
if not env_file.exists():
return env
for raw_line in env_file.read_text(encoding="utf-8-sig").splitlines():
line = raw_line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
key = key.strip()
value = value.strip().strip('"').strip("'")
if key and value and key not in env:
env[key] = value
return env
def infer_model_family(model_name: str) -> str:
normalized = model_name.strip().lower()
if normalized.startswith("gpt-5"):
return "gpt-5"
if "qwen3" in normalized:
return "qwen3-vl"
if "qwen2.5" in normalized or "qwen-2.5" in normalized:
return "qwen2.5-vl"
if "gemini" in normalized:
return "gemini"
if "doubao" in normalized:
return "doubao-seed"
if "glm" in normalized:
return "glm-v"
return ""
def validate_env(env: Dict[str, str], semantic: bool) -> List[str]:
if not semantic:
return []
required = REQUIRED_SEMANTIC_ENV if semantic else REQUIRED_BASE_ENV
missing = [name for name in required if not env.get(name)]
family = env.get("MIDSCENE_MODEL_FAMILY", "")
if semantic and family and family not in VALID_MODEL_FAMILIES:
missing.append("MIDSCENE_MODEL_FAMILY(valid value)")
return missing
def redact(text: str, env: Dict[str, str]) -> str:
secret = env.get("MIDSCENE_MODEL_API_KEY")
if secret:
text = text.replace(secret, "<redacted>")
return text
def command_name(name: str) -> str:
if os.name == "nt":
candidate = shutil.which(f"{name}.cmd")
if candidate:
return candidate
return shutil.which(name) or name
def free_port() -> int:
sock = socket.socket()
sock.bind(("127.0.0.1", 0))
port = sock.getsockname()[1]
sock.close()
return int(port)
def npm_cache_root(env: Dict[str, str]) -> Optional[Path]:
local_app_data = env.get("LOCALAPPDATA")
if local_app_data:
return Path(local_app_data) / "npm-cache"
home = env.get("HOME") or env.get("USERPROFILE")
return Path(home) / ".npm" if home else None
def midscene_dist_dirs(project_root: Path, env: Dict[str, str]) -> List[Path]:
candidates: List[Path] = []
local = project_root / "node_modules" / "@midscene" / "computer-mcp" / "dist"
if local.exists():
candidates.append(local)
cache = npm_cache_root(env)
if cache:
candidates.extend(cache.glob("_npx/*/node_modules/@midscene/computer-mcp/dist"))
unique: List[Path] = []
seen = set()
for path in candidates:
resolved = str(path.resolve())
if resolved not in seen:
seen.add(resolved)
unique.append(path)
return unique
def ensure_windows_screenshot_assets(project_root: Path, env: Dict[str, str]) -> Tuple[str, List[str]]:
if os.name != "nt":
return "skipped-non-windows", []
dist_dirs = midscene_dist_dirs(project_root, env)
missing_dirs = [
path for path in dist_dirs
if not (path / "screenCapture_1.3.2.bat").exists() or not (path / "app.manifest").exists()
]
if not missing_dirs:
return "ok", [str(path) for path in dist_dirs]
if not dist_dirs:
return "missing-midscene-package", []
npm = command_name("npm")
with tempfile.TemporaryDirectory(prefix="airxdb-screenshot-desktop-") as tmp:
tmp_path = Path(tmp)
subprocess.run(
[npm, "pack", "screenshot-desktop@1.15.3", "--pack-destination", str(tmp_path)],
cwd=str(project_root),
env=env,
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
encoding="utf-8",
errors="replace",
)
tgz = next(tmp_path.glob("screenshot-desktop-1.15.3.tgz"))
with tarfile.open(tgz, "r:gz") as archive:
archive.extract("package/lib/win32/screenCapture_1.3.2.bat", path=tmp_path)
archive.extract("package/lib/win32/app.manifest", path=tmp_path)
bat = tmp_path / "package" / "lib" / "win32" / "screenCapture_1.3.2.bat"
manifest = tmp_path / "package" / "lib" / "win32" / "app.manifest"
for dist in missing_dirs:
shutil.copy2(bat, dist / "screenCapture_1.3.2.bat")
shutil.copy2(manifest, dist / "app.manifest")
return "repaired", [str(path) for path in missing_dirs]
def decode_response(raw: str, content_type: str) -> Optional[Dict[str, Any]]:
if "text/event-stream" in content_type or raw.startswith("event:") or raw.startswith("data:"):
data_lines = [line[5:].strip() for line in raw.splitlines() if line.startswith("data:")]
raw = "\n".join(data_lines).strip()
return json.loads(raw) if raw.strip() else None
class MCPClient:
def __init__(self, url: str, env: Dict[str, str], timeout: int) -> None:
self.url = url
self.env = env
self.timeout = timeout
self.session_id: Optional[str] = None
self.seq = 1
def post(self, method: str, params: Optional[Dict[str, Any]] = None, expect_id: bool = True) -> Optional[Dict[str, Any]]:
payload: Dict[str, Any] = {"jsonrpc": "2.0", "method": method, "params": params or {}}
if expect_id:
payload["id"] = self.seq
self.seq += 1
headers = {
"Content-Type": "application/json",
"Accept": "application/json, text/event-stream",
}
if self.session_id:
headers["mcp-session-id"] = self.session_id
request = urllib.request.Request(
self.url,
data=json.dumps(payload).encode("utf-8"),
headers=headers,
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=self.timeout) as response:
raw = response.read().decode("utf-8", "replace")
raw = redact(raw, self.env)
self.session_id = response.headers.get("mcp-session-id") or self.session_id
return decode_response(raw, response.headers.get("content-type", ""))
except urllib.error.HTTPError as error:
raw = error.read().decode("utf-8", "replace")
raise RuntimeError(f"HTTP {error.code} for {method}: {redact(raw, self.env)}") from error
def call_tool(self, name: str, arguments: Optional[Dict[str, Any]] = None) -> Optional[Dict[str, Any]]:
return self.post("tools/call", {"name": name, "arguments": arguments or {}})
def content_texts(result: Optional[Dict[str, Any]]) -> List[str]:
content = (result or {}).get("result", {}).get("content", [])
return [item.get("text", "") for item in content if item.get("type") == "text"]
def save_images(result: Optional[Dict[str, Any]], output_dir: Path, stem: str) -> List[str]:
saved: List[str] = []
content = (result or {}).get("result", {}).get("content", [])
for index, item in enumerate(content):
if item.get("type") != "image" or not item.get("data"):
continue
mime = item.get("mimeType", "image/png")
ext = ".jpg" if "jpeg" in mime or "jpg" in mime else ".png"
path = output_dir / f"{stem}-{index}{ext}"
path.write_bytes(base64.b64decode(item["data"]))
saved.append(str(path))
return saved
def wait_for_port(port: int, proc: subprocess.Popen[str], logs: List[str], timeout: int) -> None:
deadline = time.time() + timeout
while time.time() < deadline:
if proc.poll() is not None:
raise RuntimeError(f"computer-mcp exited {proc.returncode}\n" + "\n".join(logs[-80:]))
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.2):
return
except OSError:
time.sleep(0.25)
raise TimeoutError("computer-mcp HTTP server did not open a port\n" + "\n".join(logs[-80:]))
def start_log_pump(stream: Any, prefix: str, logs: List[str], env: Dict[str, str]) -> None:
def pump() -> None:
for line in iter(stream.readline, ""):
logs.append(prefix + redact(line.rstrip(), env))
threading.Thread(target=pump, daemon=True).start()
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run AirXDB Computer MCP smoke test.")
parser.add_argument("--project", default=".")
parser.add_argument("--prompt", default="Windows taskbar Start button")
parser.add_argument("--action", choices=["connect", "screenshot", "mousemove", "act"], default="mousemove")
parser.add_argument("--output-dir", default="")
parser.add_argument("--timeout", type=int, default=240)
parser.add_argument("--no-repair-screenshot-assets", action="store_true")
return parser.parse_args()
def main() -> None:
args = parse_args()
project_root = Path(args.project).expanduser().resolve()
output_dir = Path(args.output_dir).expanduser().resolve() if args.output_dir else project_root / "AirPlan" / "docs" / "debug" / "airxdb-artifacts"
output_dir.mkdir(parents=True, exist_ok=True)
env = load_env(project_root)
semantic = args.action in {"mousemove", "act"}
missing = validate_env(env, semantic)
if missing:
print("airxdb_smoke=blocked")
print("missing_midscene_env=" + ",".join(missing))
suggestion = infer_model_family(env.get("MIDSCENE_MODEL_NAME", ""))
if suggestion:
print(f"suggested_midscene_model_family={suggestion}")
print("valid_midscene_model_families=" + ",".join(VALID_MODEL_FAMILIES))
raise SystemExit(2)
port = free_port()
logs: List[str] = []
npx = command_name("npx")
proc = subprocess.Popen(
[npx, "-y", "@midscene/computer-mcp", "--mode", "http", "--host", "127.0.0.1", "--port", str(port)],
cwd=str(project_root),
env=env,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
encoding="utf-8",
errors="replace",
)
assert proc.stdout is not None
assert proc.stderr is not None
start_log_pump(proc.stdout, "OUT ", logs, env)
start_log_pump(proc.stderr, "ERR ", logs, env)
steps: List[Dict[str, Any]] = []
asset_status = "skipped"
asset_paths: List[str] = []
try:
wait_for_port(port, proc, logs, min(args.timeout, 60))
if not args.no_repair_screenshot_assets:
asset_status, asset_paths = ensure_windows_screenshot_assets(project_root, env)
client = MCPClient(f"http://127.0.0.1:{port}/mcp", env, args.timeout)
client.post(
"initialize",
{
"protocolVersion": "2024-11-05",
"capabilities": {},
"clientInfo": {"name": "airxdb-computer-mcp-smoke", "version": "0.2.1"},
},
)
client.post("notifications/initialized", {}, expect_id=False)
actions: List[Tuple[str, Dict[str, Any], str]] = [
("computer_connect", {}, "connect"),
]
if args.action == "mousemove":
actions.append(("MouseMove", {"locate": {"prompt": args.prompt}}, "mousemove"))
elif args.action == "act":
actions.append(("act", {"prompt": args.prompt}, "act"))
if args.action in {"screenshot", "mousemove", "act"}:
actions.append(("take_screenshot", {}, "final"))
for tool, arguments, stem in actions:
result = client.call_tool(tool, arguments)
steps.append(
{
"tool": tool,
"arguments": arguments,
"texts": content_texts(result),
"images": save_images(result, output_dir, f"{now_stamp()}-{stem}"),
}
)
time.sleep(0.3)
try:
result = client.call_tool("computer_disconnect", {})
steps.append({"tool": "computer_disconnect", "arguments": {}, "texts": content_texts(result), "images": []})
except Exception as error:
steps.append({"tool": "computer_disconnect", "arguments": {}, "texts": [str(error)], "images": []})
failed = [
step for step in steps
if any(text.lower().startswith(("warning:", "failed", "error")) for text in step["texts"])
]
status = "failed" if failed else "ok"
report = {
"status": status,
"action": args.action,
"prompt": args.prompt,
"modelName": env.get("MIDSCENE_MODEL_NAME", ""),
"modelFamily": env.get("MIDSCENE_MODEL_FAMILY", ""),
"assetRepair": {"status": asset_status, "paths": asset_paths},
"steps": steps,
"logsTail": logs[-80:],
}
report_path = output_dir / f"{now_stamp()}-airxdb-smoke.json"
report_path.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
print(f"airxdb_smoke={status}")
print(f"capture_mode={'semantic' if semantic else 'screenshot'}")
print(f"report={report_path}")
print(f"asset_repair={asset_status}")
for step in steps:
print(f"step={step['tool']}")
for text in step["texts"]:
print("text=" + text.replace("\n", " | ")[:1600])
for image in step["images"]:
print(f"image={image}")
raise SystemExit(1 if failed else 0)
finally:
proc.terminate()
try:
proc.wait(timeout=8)
except subprocess.TimeoutExpired:
proc.kill()
if __name__ == "__main__":
main()

429
scripts/airxdb_mode.py Normal file
View File

@@ -0,0 +1,429 @@
#!/usr/bin/env python3
"""Bootstrap AirXDB GUI debug context files."""
from __future__ import annotations
import argparse
import json
import os
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, Tuple
MARKER_BEGIN = "<!-- AIRXDB:BEGIN -->"
MARKER_END = "<!-- AIRXDB:END -->"
REQUIRED_MIDSCENE_ENV = [
"MIDSCENE_MODEL_NAME",
"MIDSCENE_MODEL_BASE_URL",
"MIDSCENE_MODEL_API_KEY",
]
SEMANTIC_MIDSCENE_ENV = [
*REQUIRED_MIDSCENE_ENV,
"MIDSCENE_MODEL_FAMILY",
]
OPTIONAL_MIDSCENE_ENV = [
"MCP_SERVER_REQUEST_TIMEOUT",
]
MODEL_FAMILY_VALUES = [
"doubao-vision",
"doubao-seed",
"gemini",
"qwen2.5-vl",
"qwen3-vl",
"qwen3.5",
"qwen3.6",
"vlm-ui-tars",
"vlm-ui-tars-doubao",
"vlm-ui-tars-doubao-1.5",
"glm-v",
"auto-glm",
"auto-glm-multilingual",
"gpt-5",
]
def now_iso() -> str:
return datetime.now(timezone.utc).isoformat()
def airxdb_agents_block() -> str:
return f"""{MARKER_BEGIN}
## AirXDB GUI Debug Workflow
1. Use AirXDB for `/airxdb` sessions that debug browser or desktop GUI issues with Midscene.js.
2. Before working, load:
- `AirPlan/AGENTS.md`
- `AirPlan/docs/architecture/adr/`
- `AirPlan/docs/architecture/c4/module.md`
- `AirPlan/docs/debug/gui-debug-log.md`
3. Classify the target first:
- Web + Playwright
- Web + Chrome Bridge
- Desktop Computer / Playground
- MCP
4. Before semantic visual actions, require Midscene model config:
- `MIDSCENE_MODEL_NAME`
- `MIDSCENE_MODEL_BASE_URL`
- `MIDSCENE_MODEL_API_KEY`
- `MIDSCENE_MODEL_FAMILY`
5. Screenshot capture is allowed without semantic model config and should be used as AirDbg diagnostic evidence.
6. Reproduce visually first and keep Midscene report paths, screenshots, and commands.
7. Use AirXDB with AirDbg when GUI reproduction and code-level fixing are both needed.
8. Update ADR when long-term GUI automation or bridge/MCP choices become architecture context.
9. Update C4 module when UI automation boundaries, browser bridge layers, or desktop control boundaries change.
10. Keep GUI debug logs resumable and concise.
{MARKER_END}
"""
def c4_module_template() -> str:
return """# C4 Module
## System Context
- TODO: Describe the product, users, and UI surfaces involved in the GUI issue.
## Containers
- TODO: Describe browser, frontend app, desktop app, automation runner, and external services.
## Modules
| Module | Responsibility | Public Interfaces | Dependencies | Data Ownership | GUI Debug Notes |
| --- | --- | --- | --- | --- | --- |
| TODO | TODO | TODO | TODO | TODO | TODO |
## Automation / Observability Boundaries
- TODO: Record Playwright, Bridge, Desktop, MCP, report generation, and screenshot boundaries.
## Change Log
- TODO: Record GUI-debug-related boundary changes.
"""
def adr_template() -> str:
return """# ADR-0001: AirXDB GUI Debug Governance
- Status: Accepted
- Date: TODO
## Context
GUI debugging needs stable visual reproduction, report evidence, and durable project context across sessions.
## Decision
Use AirXDB to maintain `AirPlan/AGENTS.md`, C4 module docs, ADR records, and `AirPlan/docs/debug/gui-debug-log.md` during Midscene-based GUI debugging.
## Consequences
- GUI issues can be reproduced with report evidence.
- Long-term GUI automation choices become traceable.
- AirDbg can consume AirXDB evidence for code-level fixes.
## Alternatives
- Screenshot-only chat debugging: rejected because it is hard to resume and verify.
"""
def gui_debug_log_template() -> str:
return """# GUI Debug Log
Append entries for AirXDB sessions.
## Entry Template
### YYYY-MM-DD: short GUI issue title
- Target surface: TODO
- Midscene mode: Playwright / Bridge / Computer / MCP
- Symptom: TODO
- Expected: TODO
- Actual: TODO
- Reproduction: TODO
- Report path: TODO
- Key observations: TODO
- Hand-off to AirDbg: TODO
- Validation after fix: TODO
- ADR/C4 updates: TODO
- Residual risk: TODO
"""
def midscene_env_example() -> str:
return """# AirXDB Midscene model configuration
#
# Copy this file to `.airxdb/midscene.local.env` and fill values locally,
# or export the same variables in your shell before running AirXDB.
# Never commit real API keys.
MIDSCENE_MODEL_NAME=
MIDSCENE_MODEL_BASE_URL=
MIDSCENE_MODEL_API_KEY=
# Required for semantic visual actions such as act, Tap, Input,
# KeyboardPress, MouseMove with locate prompts, and aiLocate.
# Common values:
# - gpt-5 for GPT-5.x visual models, including gpt-5.4
# - qwen2.5-vl
# - qwen3-vl
# - gemini
# - doubao-seed
MIDSCENE_MODEL_FAMILY=
# Optional:
MCP_SERVER_REQUEST_TIMEOUT=120000
"""
def airxdb_gitignore_template() -> str:
return """midscene.local.env
"""
def write_if_missing(path: Path, content: str) -> bool:
if path.exists():
return False
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8", newline="\n")
return True
def upsert_agents_md(path: Path) -> str:
block = airxdb_agents_block().rstrip() + "\n"
if path.exists():
original = path.read_text(encoding="utf-8")
existed = True
else:
original = "# AGENTS.md\n\n"
existed = False
begin = original.find(MARKER_BEGIN)
end = original.find(MARKER_END)
if begin >= 0 and end > begin:
end += len(MARKER_END)
updated = original[:begin].rstrip() + "\n\n" + block + original[end:].lstrip()
status = "updated"
else:
updated = original.rstrip() + "\n\n" + block
status = "updated" if existed else "created"
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(updated, encoding="utf-8", newline="\n")
return status
def artifact_map(project_root: Path) -> Dict[str, Path]:
return {
"AGENTS.md": project_root / "AirPlan" / "AGENTS.md",
"c4_module": project_root / "AirPlan" / "docs" / "architecture" / "c4" / "module.md",
"adr_dir": project_root / "AirPlan" / "docs" / "architecture" / "adr",
"adr_0001": project_root / "AirPlan" / "docs" / "architecture" / "adr" / "ADR-0001-airxdb-gui-debug-governance.md",
"gui_debug_log": project_root / "AirPlan" / "docs" / "debug" / "gui-debug-log.md",
"model_env_example": project_root / "AirPlan" / "state" / "airxdb" / "midscene.env.example",
"airxdb_gitignore": project_root / "AirPlan" / "state" / "airxdb" / ".gitignore",
"model_env_local": project_root / "AirPlan" / "state" / "airxdb" / "midscene.local.env",
"state": project_root / "AirPlan" / "state" / "airxdb" / "state.json",
}
def load_local_midscene_env(project_root: Path) -> str:
env_file = artifact_map(project_root)["model_env_local"]
if not env_file.exists():
return "missing"
loaded = 0
for raw_line in env_file.read_text(encoding="utf-8-sig").splitlines():
line = raw_line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
key = key.strip()
value = value.strip().strip('"').strip("'")
if key and key not in os.environ and value:
os.environ[key] = value
loaded += 1
return f"loaded:{loaded}"
def infer_model_family(model_name: str) -> str:
normalized = model_name.strip().lower()
if not normalized:
return ""
if normalized.startswith("gpt-5"):
return "gpt-5"
if "qwen3" in normalized:
return "qwen3-vl"
if "qwen2.5" in normalized or "qwen-2.5" in normalized:
return "qwen2.5-vl"
if "gemini" in normalized:
return "gemini"
if "doubao" in normalized:
return "doubao-seed"
if "glm" in normalized:
return "glm-v"
return ""
def status_from_missing(missing: list[str], invalid: bool = False) -> str:
if invalid:
return "invalid:MIDSCENE_MODEL_FAMILY"
return "ok" if not missing else "missing:" + ",".join(missing)
def midscene_config_health() -> Dict[str, object]:
checked_names = SEMANTIC_MIDSCENE_ENV + OPTIONAL_MIDSCENE_ENV
present = {name: bool(os.environ.get(name)) for name in checked_names}
missing_basic = [name for name in REQUIRED_MIDSCENE_ENV if not present[name]]
missing_semantic = [name for name in SEMANTIC_MIDSCENE_ENV if not present[name]]
model_name = os.environ.get("MIDSCENE_MODEL_NAME", "").strip()
model_family = os.environ.get("MIDSCENE_MODEL_FAMILY", "").strip()
valid_model_family = (not model_family) or model_family in MODEL_FAMILY_VALUES
suggested_model_family = infer_model_family(model_name)
basic_ready = not missing_basic
semantic_ready = basic_ready and not missing_semantic and valid_model_family
return {
"ready": semantic_ready,
"basicReady": basic_ready,
"semanticReady": semantic_ready,
"required": REQUIRED_MIDSCENE_ENV,
"requiredForSemanticActions": SEMANTIC_MIDSCENE_ENV,
"optional": OPTIONAL_MIDSCENE_ENV,
"present": present,
"missing": missing_basic,
"missingForSemanticActions": missing_semantic,
"validModelFamily": valid_model_family,
"modelFamily": model_family,
"suggestedModelFamily": suggested_model_family,
"validModelFamilies": MODEL_FAMILY_VALUES,
}
def write_state(path: Path, enabled: bool, project_root: Path) -> None:
artifacts = artifact_map(project_root)
health = {
name: artifacts[name].exists()
for name in [
"AGENTS.md",
"c4_module",
"adr_dir",
"adr_0001",
"gui_debug_log",
"model_env_example",
]
}
config_health = midscene_config_health()
payload = {
"enabled": enabled,
"updatedAt": now_iso(),
"projectRoot": str(project_root),
"artifactHealth": health,
"midsceneConfig": config_health,
"screenshotCapture": {
"requiresModelConfig": False,
"recommendedScript": "airxdb_computer_mcp_smoke.py --action screenshot",
},
}
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
def enter_mode(project_root: Path) -> Tuple[str, Dict[str, str]]:
artifacts = artifact_map(project_root)
results: Dict[str, str] = {}
results["midscene_local_env"] = load_local_midscene_env(project_root)
results["AGENTS.md"] = upsert_agents_md(artifacts["AGENTS.md"])
results["c4_module"] = "created" if write_if_missing(artifacts["c4_module"], c4_module_template()) else "exists"
artifacts["adr_dir"].mkdir(parents=True, exist_ok=True)
results["adr_dir"] = "exists"
results["adr_0001"] = "created" if write_if_missing(artifacts["adr_0001"], adr_template()) else "exists"
results["gui_debug_log"] = "created" if write_if_missing(artifacts["gui_debug_log"], gui_debug_log_template()) else "exists"
results["model_env_example"] = "created" if write_if_missing(artifacts["model_env_example"], midscene_env_example()) else "exists"
results["airxdb_gitignore"] = "created" if write_if_missing(artifacts["airxdb_gitignore"], airxdb_gitignore_template()) else "exists"
config_health = midscene_config_health()
results["midscene_basic_config"] = status_from_missing(config_health["missing"]) # type: ignore[arg-type]
results["midscene_semantic_config"] = status_from_missing(
config_health["missingForSemanticActions"], # type: ignore[arg-type]
invalid=not bool(config_health["validModelFamily"]),
)
results["midscene_config"] = results["midscene_semantic_config"]
write_state(artifacts["state"], True, project_root)
return "enabled", results
def exit_mode(project_root: Path) -> Tuple[str, Dict[str, str]]:
artifacts = artifact_map(project_root)
write_state(artifacts["state"], False, project_root)
return "disabled", {}
def status_mode(project_root: Path) -> Tuple[str, Dict[str, str]]:
artifacts = artifact_map(project_root)
local_env_status = load_local_midscene_env(project_root)
state_file = artifacts["state"]
enabled = False
if state_file.exists():
try:
payload = json.loads(state_file.read_text(encoding="utf-8"))
enabled = bool(payload.get("enabled"))
except json.JSONDecodeError:
enabled = False
results = {
name: ("ok" if path.exists() else "missing")
for name, path in artifacts.items()
if name not in ["state", "model_env_local"]
}
results["midscene_local_env"] = local_env_status
config_health = midscene_config_health()
results["midscene_basic_config"] = status_from_missing(config_health["missing"]) # type: ignore[arg-type]
results["midscene_semantic_config"] = status_from_missing(
config_health["missingForSemanticActions"], # type: ignore[arg-type]
invalid=not bool(config_health["validModelFamily"]),
)
results["midscene_config"] = results["midscene_semantic_config"]
return ("enabled" if enabled else "disabled"), results
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Manage AirXDB GUI debug artifacts.")
parser.add_argument("--mode", choices=["enter", "exit", "status"], default="enter")
parser.add_argument("--project", default=".")
return parser.parse_args()
def main() -> None:
args = parse_args()
project_root = Path(args.project).expanduser().resolve()
if args.mode == "enter":
mode_state, result = enter_mode(project_root)
elif args.mode == "exit":
mode_state, result = exit_mode(project_root)
else:
mode_state, result = status_mode(project_root)
print(f"airxdb_mode={mode_state}")
print(f"project_root={project_root}")
for key, value in result.items():
print(f"{key}={value}")
config_health = midscene_config_health()
if mode_state == "enabled" and not config_health["semanticReady"]:
missing = ",".join(config_health["missingForSemanticActions"]) # type: ignore[arg-type]
print("midscene_config_required=true")
print(f"missing_midscene_env={missing}")
if not config_health["validModelFamily"]:
print(f"invalid_midscene_model_family={config_health['modelFamily']}")
if config_health["suggestedModelFamily"]:
print(f"suggested_midscene_model_family={config_health['suggestedModelFamily']}")
print("valid_midscene_model_families=" + ",".join(MODEL_FAMILY_VALUES))
print("midscene_config_prompt=请先提供 Midscene 模型配置MIDSCENE_MODEL_NAME、MIDSCENE_MODEL_BASE_URL、MIDSCENE_MODEL_API_KEY、MIDSCENE_MODEL_FAMILY可选 MCP_SERVER_REQUEST_TIMEOUT。不要把真实 API key 写入 ADR/C4/debug log。")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,128 @@
#!/usr/bin/env python3
"""AirXDB remote GUI device helper over SSH."""
from __future__ import annotations
import argparse, base64, json, os, shlex, shutil, subprocess
from datetime import datetime, timezone
from pathlib import Path
def stamp(): return datetime.now().strftime('%Y%m%d-%H%M%S')
def iso(): return datetime.now(timezone.utc).isoformat()
def q(s): return shlex.quote(str(s))
def read_env(path):
vals = {}
if path.exists():
for raw in path.read_text(encoding='utf-8-sig').splitlines():
line = raw.strip()
if line and not line.startswith('#') and '=' in line:
k, v = line.split('=', 1); vals[k.strip()] = v.strip().strip('"').strip("'")
for k in ['AIRXDB_REMOTE_SSH_TARGET','AIRXDB_REMOTE_SSH_PORT','AIRXDB_REMOTE_SSH_OPTIONS','AIRXDB_REMOTE_WORKDIR','AIRXDB_REMOTE_SCREENSHOT_TOOL','AIRXDB_REMOTE_DISPLAY']:
if os.environ.get(k): vals[k] = os.environ[k]
return vals
def write_env(path, updates):
path.parent.mkdir(parents=True, exist_ok=True)
old = path.read_text(encoding='utf-8') if path.exists() else ''
keys = set(updates); lines = []
for raw in old.splitlines():
key = raw.split('=',1)[0].strip() if '=' in raw and not raw.strip().startswith('#') else None
if key not in keys: lines.append(raw)
if lines and lines[-1].strip(): lines.append('')
for k, v in updates.items():
if v: lines.append(f'{k}={v}')
path.write_text('\n'.join(lines).rstrip()+'\n', encoding='utf-8', newline='\n')
def ensure_files(project):
d = project/'AirPlan'/'state'/'airxdb'; d.mkdir(parents=True, exist_ok=True)
ex = d/'remote-device.env.example'
if not ex.exists():
ex.write_text('# AirXDB remote GUI device configuration\nAIRXDB_REMOTE_SSH_TARGET=user@host\nAIRXDB_REMOTE_SSH_PORT=22\nAIRXDB_REMOTE_SSH_OPTIONS=\nAIRXDB_REMOTE_WORKDIR=\nAIRXDB_REMOTE_SCREENSHOT_TOOL=auto\nAIRXDB_REMOTE_DISPLAY=\n', encoding='utf-8', newline='\n')
gi = d/'.gitignore'; old = gi.read_text(encoding='utf-8') if gi.exists() else ''
for item in ['remote-device.env','midscene.local.env']:
if item not in old.splitlines(): old = (old.rstrip()+f'\n{item}\n').lstrip()
gi.write_text(old, encoding='utf-8', newline='\n')
def ssh_args(env, cmd):
target = env.get('AIRXDB_REMOTE_SSH_TARGET','')
if not target: raise SystemExit('AIRXDB_REMOTE_SSH_TARGET is required')
args = [shutil.which('ssh') or 'ssh']
port = env.get('AIRXDB_REMOTE_SSH_PORT','22')
if port: args += ['-p', port]
opts = env.get('AIRXDB_REMOTE_SSH_OPTIONS','').strip()
if opts: args += shlex.split(opts, posix=True)
return args + [target, cmd]
def run(env, cmd, timeout=60):
return subprocess.run(ssh_args(env, cmd), capture_output=True, text=True, encoding='utf-8', errors='replace', timeout=timeout)
def remote_home(env):
r = run(env, 'printf %s "$HOME"', 15)
return r.stdout.strip() if r.returncode == 0 else ''
def workdir(env):
if env.get('AIRXDB_REMOTE_WORKDIR'): return env['AIRXDB_REMOTE_WORKDIR']
home = remote_home(env); return (home.rstrip('/')+'/.airxdb') if home else '.airxdb'
def detect_tool(env):
tool = env.get('AIRXDB_REMOTE_SCREENSHOT_TOOL','auto').strip()
if tool and tool != 'auto':
r = run(env, f'command -v {q(tool)}', 15); return tool if r.returncode == 0 else ''
script = 'for t in gnome-screenshot spectacle scrot grim import screencapture; do command -v "$t" >/dev/null 2>&1 && { printf %s "$t"; exit 0; }; done'
r = run(env, script, 15)
return r.stdout.strip().splitlines()[-1] if r.returncode == 0 and r.stdout.strip() else ''
def install_tool(env):
script = """set -e
if command -v scrot >/dev/null 2>&1; then exit 0; fi
if [ "$(id -u 2>/dev/null || echo 1)" = "0" ]; then SUDO=""; else SUDO="sudo -n"; fi
if command -v apt-get >/dev/null 2>&1; then $SUDO apt-get update && $SUDO apt-get install -y scrot;
elif command -v dnf >/dev/null 2>&1; then $SUDO dnf install -y scrot;
elif command -v yum >/dev/null 2>&1; then $SUDO yum install -y scrot;
elif command -v apk >/dev/null 2>&1; then $SUDO apk add scrot;
elif command -v pacman >/dev/null 2>&1; then $SUDO pacman -Sy --noconfirm scrot;
else exit 42; fi"""
return run(env, script, 300)
def setup(project, env, auto=True):
if not env.get('AIRXDB_REMOTE_SSH_TARGET'):
return {'status':'blocked','reason':'missing_remote_target','example':str(project/'AirPlan'/'state'/'airxdb'/'remote-device.env.example')}
if not shutil.which('ssh'): return {'status':'blocked','reason':'ssh_not_found'}
probe = run(env, 'printf ok', 20)
if probe.returncode: return {'status':'blocked','reason':'ssh_probe_failed','stderr':probe.stderr.strip()}
wd = workdir(env); mk = run(env, f'mkdir -p {q(wd)}', 20)
if mk.returncode: return {'status':'blocked','reason':'remote_workdir_failed','stderr':mk.stderr.strip()}
tool = detect_tool(env); install = 'skipped'
if not tool and auto:
inst = install_tool(env); install = 'ok' if inst.returncode == 0 else f'failed:{inst.returncode}'; tool = detect_tool(env)
upd = {'AIRXDB_REMOTE_WORKDIR': wd}
if tool: upd['AIRXDB_REMOTE_SCREENSHOT_TOOL'] = tool
write_env(project/'AirPlan'/'state'/'airxdb'/'remote-device.env', upd)
return {'status':'ok' if tool else 'blocked','target':env.get('AIRXDB_REMOTE_SSH_TARGET',''),'workdir':wd,'screenshotTool':tool,'autoConfigure':install,'hint':'' if tool else 'Install scrot/gnome-screenshot/grim/spectacle/import/screencapture or set AIRXDB_REMOTE_SCREENSHOT_TOOL.'}
def shot_cmd(tool, out, display):
pre = f'export DISPLAY={q(display)}; ' if display else ''
table = {'gnome-screenshot':f'gnome-screenshot -f {q(out)}','spectacle':f'spectacle -b -n -o {q(out)}','scrot':f'scrot {q(out)}','grim':f'grim {q(out)}','import':f'import -window root {q(out)}','screencapture':f'screencapture -x {q(out)}'}
if tool not in table: raise SystemExit(f'unsupported screenshot tool: {tool}')
return (pre if tool != 'screencapture' else '') + table[tool]
def emit(prefix, data):
print(f'{prefix}_status={data.get("status","unknown")}')
for k, v in data.items():
if k != 'status': print(f'{k}={json.dumps(v, ensure_ascii=False) if isinstance(v,(dict,list)) else v}')
def main():
ap = argparse.ArgumentParser(); ap.add_argument('--project', default='.'); ap.add_argument('--action', choices=['setup','status','screenshot'], default='setup'); ap.add_argument('--output-dir', default=''); ap.add_argument('--timeout', type=int, default=60); ap.add_argument('--no-auto-configure', action='store_true'); a = ap.parse_args()
project = Path(a.project).expanduser().resolve(); ensure_files(project); env = read_env(project/'AirPlan'/'state'/'airxdb'/'remote-device.env')
info = setup(project, env, auto=(a.action!='status' and not a.no_auto_configure))
if a.action in ['setup','status'] or info.get('status') != 'ok': emit('airxdb_remote', info); raise SystemExit(0 if info.get('status')=='ok' else 2)
outdir = Path(a.output_dir).expanduser().resolve() if a.output_dir else project/'AirPlan'/'docs'/'debug'/'airxdb-artifacts'; outdir.mkdir(parents=True, exist_ok=True)
remote = info['workdir'].rstrip('/') + '/' + stamp() + '-airxdb-remote.png'; cap = run(env, shot_cmd(info['screenshotTool'], remote, env.get('AIRXDB_REMOTE_DISPLAY','')), a.timeout)
data = {'status':'ok' if cap.returncode == 0 else 'failed','target':info['target'],'tool':info['screenshotTool'],'remoteFile':remote,'stdout':cap.stdout.strip(),'stderr':cap.stderr.strip(),'capturedAt':iso()}
if cap.returncode == 0:
b64 = run(env, f'base64 < {q(remote)}', a.timeout)
local = outdir/(stamp()+'-airxdb-remote.png'); local.write_bytes(base64.b64decode(''.join(b64.stdout.split()))); data['screenshot'] = str(local)
report = outdir/(stamp()+'-airxdb-remote-device.json'); report.write_text(json.dumps(data, ensure_ascii=False, indent=2)+'\n', encoding='utf-8'); data['report'] = str(report)
if data['status'] == 'ok':
log = project/'AirPlan'/'docs'/'debug'/'gui-debug-log.md'; log.parent.mkdir(parents=True, exist_ok=True); log.open('a', encoding='utf-8', newline='\n').write(f"\n## {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}: AirXDB remote screenshot\n\n- Remote target: `{data['target']}`\n- Screenshot: `{data.get('screenshot','')}`\n- Report: `{data['report']}`\n- AirDbg handoff: TODO\n- Residual risk: remote screenshots may contain sensitive data.\n")
emit('airxdb_remote', data); raise SystemExit(0 if data['status']=='ok' else 1)
if __name__ == '__main__': main()

View File

@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""Install the AirXDB plugin into the current user's home-local plugin directory."""
from __future__ import annotations
import argparse
import json
import shutil
from pathlib import Path
from typing import Any, Dict
PLUGIN_NAME = "airxdb"
def copy_plugin(source: Path, target: Path) -> None:
if source.resolve() == target.resolve():
return
if target.exists():
shutil.rmtree(target)
ignore = shutil.ignore_patterns("__pycache__", "*.pyc", ".DS_Store", ".git")
shutil.copytree(source, target, ignore=ignore)
def marketplace_payload() -> Dict[str, Any]:
return {
"name": "local-airxdb",
"interface": {"displayName": "Local AirXDB Plugins"},
"plugins": [],
}
def update_marketplace(path: Path) -> None:
if path.exists():
payload = json.loads(path.read_text(encoding="utf-8"))
else:
payload = marketplace_payload()
payload.setdefault("name", "local-airxdb")
payload.setdefault("interface", {}).setdefault("displayName", "Local AirXDB Plugins")
plugins = payload.setdefault("plugins", [])
entry = {
"name": PLUGIN_NAME,
"source": {
"source": "local",
"path": f"./plugins/{PLUGIN_NAME}",
},
"policy": {
"installation": "INSTALLED_BY_DEFAULT",
"authentication": "ON_INSTALL",
},
"category": "Productivity",
}
for index, existing in enumerate(plugins):
if existing.get("name") == PLUGIN_NAME:
plugins[index] = entry
break
else:
plugins.append(entry)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Install AirXDB as a home-local Codex plugin.")
parser.add_argument("--source", default=str(Path(__file__).resolve().parents[1]))
parser.add_argument("--home", default=str(Path.home()))
return parser.parse_args()
def main() -> None:
args = parse_args()
source = Path(args.source).expanduser().resolve()
home = Path(args.home).expanduser().resolve()
target = home / "plugins" / PLUGIN_NAME
marketplace = home / ".agents" / "plugins" / "marketplace.json"
copy_plugin(source, target)
update_marketplace(marketplace)
print(f"installed_plugin={target}")
print(f"marketplace={marketplace}")
if __name__ == "__main__":
main()

263
skills/airxdb/SKILL.md Normal file
View File

@@ -0,0 +1,263 @@
---
name: airxdb
description: Midscene-based GUI debugging workflow. Use when the user invokes /airxdb, asks to debug browser UI, desktop UI, canvas UI, visual regressions, flaky interface interactions, remote GUI/device debugging over SSH, or reproduce graphical issues with Midscene.js. Load or initialize AirPlan/AGENTS.md, AirPlan/docs/architecture/adr/ decision records, and AirPlan/docs/architecture/c4/module.md; collaborate with AirDbg for GUI issues; choose the right Midscene mode for Playwright, Chrome bridge mode, desktop computer automation, MCP, or the AirXDB remote device helper; auto-configure remote screenshot tooling when missing; generate visual reproduction steps and reports; and maintain AirPlan/AGENTS.md, ADR, C4 module, and GUI debug logs whenever GUI-debug tooling or interface behavior changes.
---
# AirXDB
## 核心约束
- 全程使用中文与用户交流,代码、命令、日志、路径、包名保持原文。
- `/airxdb` 专用于图形界面和视觉交互层面的调试,不替代 `airdbg` 的通用根因分析职责。
- 优先与 `airdbg` 配合:
- `airxdb` 负责界面复现、视觉定位、交互自动化、Midscene 报告与截图证据。
- `airdbg` 负责代码层根因、最小修复、测试验证和通用调试收尾。
- 先加载或初始化上下文:`AirPlan/AGENTS.md``AirPlan/docs/architecture/adr/``AirPlan/docs/architecture/c4/module.md``AirPlan/docs/debug/gui-debug-log.md`
- 如果这些文件不存在先分析当前项目并初始化它们C4 module 要反映真实模块边界尤其是前端、UI、桌面桥接、自动化测试相关边界。
- Midscene 路线优先使用视觉复现和 HTML 报告收集证据,不把 GUI 调试退化成纯日志猜测。
- 截图取证是 AirXDB 的一等能力:即使没有语义模型配置,也可以连接 Computer MCP 截图,为 `airdbg` 提供错误现场、布局状态、弹窗、焦点和多显示器信息。
- 远程设备、测试机、VM 或 SSH 主机上的 GUI 调试,先调用 `../../scripts/airxdb_remote_device.py`;缺少远程截图工具时允许脚本自动配置,无法无密码 `sudo` 或无包管理器时停止并提示用户。
- 优先做最小可验证复现和最小必要修复,不顺手大改界面架构。
- GUI 调试过程中,一定要维护 `AirPlan/AGENTS.md`、ADR、C4 module 和 `AirPlan/docs/debug/gui-debug-log.md`
- 如果需要 Midscene 包名、桥接/MCP 配置、常用命令,读取 [references/midscene-official-notes.md](references/midscene-official-notes.md)。
## 启动与初始化
进入 `/airxdb` 时运行:
```bash
python "$HOME/plugins/airxdb/scripts/airxdb_mode.py" --mode enter --project .
```
如果当前环境没有 `python`,尝试 `py``python3`。脚本不可用时,手动确保以下结构存在:
- `AirPlan/AGENTS.md`
- `AirPlan/docs/architecture/adr/`
- `AirPlan/docs/architecture/c4/module.md`
- `AirPlan/docs/debug/gui-debug-log.md`
- `AirPlan/state/airxdb/state.json`
初始化后读取已有内容作为上下文。不要覆盖用户已有正文;只补齐缺失结构或更新 AirXDB 标记块。
## 首次模型配置
AirXDB 第一次进入项目时必须检查 Midscene 模型配置。`airxdb_mode.py` 会输出 `midscene_config`,如果出现 `midscene_config_required=true``midscene_config=missing:...`
- 暂停执行 `act``Tap``Input``KeyboardPress`、视觉定位等语义动作。
- 用中文向用户索取配置:
- `MIDSCENE_MODEL_NAME`
- `MIDSCENE_MODEL_BASE_URL`
- `MIDSCENE_MODEL_API_KEY`
- `MIDSCENE_MODEL_FAMILY`
- 可选:`MCP_SERVER_REQUEST_TIMEOUT`
- 告知用户可以只在当前会话设置环境变量,或写入 `AirPlan/state/airxdb/midscene.local.env` 方便后续复用。
- `AirPlan/state/airxdb/midscene.local.env` 只保存本机密钥,默认由 `AirPlan/state/airxdb/.gitignore` 忽略;不要把真实 API key 写入 `AirPlan/AGENTS.md`、ADR、C4 或 debug log。
- 如果用户暂时不提供模型配置仍然可以做截图、MCP 连接、显示器枚举、环境探测等非语义取证操作;不能声称完成了 Midscene 视觉语义操作。
- `MIDSCENE_MODEL_FAMILY` 是语义视觉动作必填项。`gpt-5.4` 这类 GPT-5.x 视觉模型使用 `gpt-5`
- 常见合法 family`gpt-5``qwen2.5-vl``qwen3-vl``gemini``doubao-seed``vlm-ui-tars`
## 截图取证模式
当用户需要给 `airdbg` 提供错误诊断信息、GUI 现场、弹窗、焦点状态、布局错位或多显示器证据时,优先使用截图取证模式:
```bash
python "$HOME/plugins/airxdb/scripts/airxdb_computer_mcp_smoke.py" --project . --action screenshot
```
截图取证模式不要求 `MIDSCENE_MODEL_NAME``MIDSCENE_MODEL_BASE_URL``MIDSCENE_MODEL_API_KEY``MIDSCENE_MODEL_FAMILY`。它只验证 Computer MCP 桌面连接和截图能力,并将截图/JSON 报告写入 `AirPlan/docs/debug/airxdb-artifacts/`
截图后必须在 `AirPlan/docs/debug/gui-debug-log.md` 追加:
- 截图目标和平台
- 截图文件路径
- 当前界面关键观察
-`airdbg` 的诊断线索
- 是否还需要语义视觉动作或代码层修复
如果截图可能包含密钥、聊天内容、账号、客户数据或隐私信息,在对外分享前提醒用户脱敏。
## 远程设备截图模式
当用户说明目标在远程设备、测试机、服务器、VM、SSH 主机,或当前桌面不是目标 GUI 所在机器时,不要先使用本机 Computer MCP。先运行远程设备 helper
```bash
python "$HOME/plugins/airxdb/scripts/airxdb_remote_device.py" --project . --action setup
```
如果当前环境没有 `python`,尝试 `py``python3` 或用户提供的 Python 绝对路径。首次运行会生成 `AirPlan/state/airxdb/remote-device.env.example`;把连接信息写入 `AirPlan/state/airxdb/remote-device.env` 或当前环境变量:
- `AIRXDB_REMOTE_SSH_TARGET=user@host`
- `AIRXDB_REMOTE_SSH_PORT=22`
- `AIRXDB_REMOTE_SSH_OPTIONS=`
- `AIRXDB_REMOTE_WORKDIR=`
- `AIRXDB_REMOTE_SCREENSHOT_TOOL=auto`
- `AIRXDB_REMOTE_DISPLAY=`
远程 helper 行为:
- 检查本机 `ssh`、远程连通性和远程工作目录。
- 探测 `gnome-screenshot``spectacle``scrot``grim``import``screencapture`
- 工具缺失时自动尝试用远端包管理器安装 `scrot`,只使用非交互式 `sudo -n`;需要密码、管理员确认或无支持的包管理器时停止并提示用户。
- 将可复用配置写入 `AirPlan/state/airxdb/remote-device.env`,该文件由 `AirPlan/state/airxdb/.gitignore` 忽略。
远程截图:
```bash
python "$HOME/plugins/airxdb/scripts/airxdb_remote_device.py" --project . --action screenshot
```
远程截图和 JSON 报告写入 `AirPlan/docs/debug/airxdb-artifacts/`,并追加 `AirPlan/docs/debug/gui-debug-log.md`。远程截图只能证明远端截图链路和当前 GUI 现场;如果需要 Midscene 语义视觉动作,仍需单独确认远端或本机可用的 Midscene 接入方式。
## Computer MCP 快速验证
验证桌面 GUI 能力时优先运行 smoke test 脚本,而不是临时拼 MCP 客户端:
```bash
python "$HOME/plugins/airxdb/scripts/airxdb_computer_mcp_smoke.py" --project . --action mousemove --prompt "Windows taskbar Start button"
```
如果当前环境没有 `python`,尝试 `py``python3` 或用户提供的 Python 绝对路径。
脚本行为:
- 加载 `AirPlan/state/airxdb/midscene.local.env`,但输出和 JSON 报告会屏蔽 API key`--action screenshot` 不要求模型配置。
- 启动 `@midscene/computer-mcp` HTTP 服务。
- Windows 下自动检查并修复 `screenCapture_1.3.2.bat` / `app.manifest` 缺失问题;修复来源是 `screenshot-desktop@1.15.3` 官方 npm 包。
- 执行 `computer_connect`、可选语义动作、`take_screenshot``computer_disconnect`
- 证据写入 `docs/debug/airxdb-artifacts/`,包括截图和 `airxdb-smoke.json` 报告。
结果判断:
- `airxdb_smoke=ok` 才能说明 Computer MCP 视觉链路通过。
- 如果只完成截图,没有完成 `MouseMove` / `act` / `Tap` 等语义动作,只能说“截图/连接可用”,不能说“视觉语义操作已通过”。
- 如果报 `MIDSCENE_MODEL_FAMILY is not set to a visual language model`,先补 family`gpt-5.4``gpt-5`
- Windows 开始菜单 + 中文输入法场景下,输入查询词后可能需要双回车:第一次提交输入法,第二次执行启动。
## Midscene 模式选择
先判断目标界面和现有技术栈,再选 Midscene 模式:
1. Web + 已有 Playwright
- 优先使用 Midscene 的 Playwright 集成。
- 适合已有 E2E、页面复现、交互不稳定、视觉断言场景。
2. Web + 需要复用本地 Chrome 状态:
- 使用 Chrome Bridge Mode。
- 适合需要复用 cookies、扩展、已登录会话、人工介入浏览器态的场景。
3. 桌面应用 GUI
- 使用 Midscene Computer / Playground。
- 适合 Electron、Qt、WPF、原生应用和跨应用流程。
4. 远程设备 GUI
- 先使用 AirXDB remote device helper 取证和自动配置远端截图工具。
- 适合 SSH 可达的测试机、VM、服务器桌面或远程 Linux/macOS GUI。
5. 需要把 GUI 操作暴露给上层 Agent 或工具链:
- 使用 Midscene MCP。
- 浏览器优先 Web Bridge MCP桌面优先 Computer MCP。
不确定时,一次只问一个关键问题:
- 这是浏览器页面还是桌面应用?
- 项目里是否已有 Playwright
- 是否必须复用本机浏览器登录态?
- 是要快速复现,还是要沉淀成长期自动化脚本?
## GUI 调试流程
1. 明确问题边界:
- 哪个界面、哪条交互链路、什么平台。
- 期望行为和实际行为。
- 是否涉及视觉错位、点击不到、浮层遮挡、Canvas、焦点问题、窗口切换、多显示器等。
2. 加载上下文:
- 读取 `AGENTS.md`
- 读取相关 ADR。
- 读取 `docs/architecture/c4/module.md`
- 查看前端/桌面自动化相关代码、测试、构建配置。
3. 选择 Midscene 模式并做最小复现:
- Playwright
- Chrome Bridge
- Computer / Playground
- MCP
4. 生成可回放证据:
- Midscene HTML 报告
- 关键截图
- 复现命令
- 相关日志和报错
5. 将证据和观察写入 `AirPlan/docs/debug/gui-debug-log.md`
6. 如果问题只是界面复现层,继续用 `airxdb` 深挖。
7. 如果已定位到代码层根因,转入或并行配合 `airdbg` 做修复。
8. 修复后再次用 Midscene 复跑关键 GUI 路径,确认问题关闭。
## 与 AirDbg 的协作
- `airxdb` 先做:
- GUI 复现
- 视觉定位
- 界面交互脚本/桥接/MCP 配置
- 报告与截图证据
- `airdbg` 再做:
- 根因代码分析
- 修复实现
- 测试验证
- 风险收尾
如果当前问题同时包含“界面复现难”和“代码根因不明”,先用 `airxdb` 稳定复现,再把复现结论和报告交给 `airdbg`
## AGENTS.md 维护
在以下情况更新 `AGENTS.md`
- 发现新的 GUI 调试命令、Playwright 命令、Midscene 运行方式。
- 发现系统权限要求,例如桌面自动化权限、屏幕录制权限、多显示器限制。
- 发现影响后续 GUI 调试的重要约束,如浏览器桥接、登录态、测试环境、显示缩放。
- 发现远程设备 SSH 入口、远程截图工具、`AIRXDB_REMOTE_*` 配置方式或远端显示环境限制。
- 引入了新的 GUI 自动化脚本、报告目录或运行前置条件。
内容保持可执行、可复用,不写流水账。
## ADR 维护
目录:`docs/architecture/adr/`
需要 ADR 的情况:
- 决定长期采用某种 Midscene 接入方式,例如 Playwright 集成、Bridge Mode、Computer、MCP。
- GUI 调试方案改变了测试边界、前端交互契约、浏览器控制方式、桌面自动化权限模型。
- 为了稳定复现而新增长期保留的自动化脚本、报告流程或辅助基础设施。
ADR 保持简洁Context、Decision、Consequences、Alternatives。
## C4 Module 维护
文件:`docs/architecture/c4/module.md`
当 GUI 调试或修复改变以下内容时,必须更新:
- 前端模块边界
- 自动化测试边界
- 浏览器桥接/桌面控制边界
- 报告和调试基础设施
- UI 层和服务层之间的数据所有权或依赖
## GUI Debug Log 维护
文件:`AirPlan/docs/debug/gui-debug-log.md`
每次 AirXDB 会话至少追加:
- 问题摘要
- 目标平台和界面
- Midscene 模式Playwright / Bridge / Computer / MCP
- 是否使用远程设备 helper 以及远程目标、截图工具和限制
- 复现步骤或命令
- 报告文件路径
- 截图或关键观察
- 转交给 `airdbg` 的结论,或已完成的修复验证
- 剩余风险
## 输出格式
本轮 GUI 调试结束时,用中文简洁汇报:
- 选择了哪种 Midscene 模式,为什么。
- 复现是否成功,证据在哪里。
- 更新了哪些 `AGENTS.md` / ADR / C4 / GUI debug log。
- 是否需要切给 `airdbg` 继续做代码层修复。

View File

@@ -0,0 +1,3 @@
name: airxdb
short_description: Midscene GUI debug workflow with local and remote screenshot evidence
default_prompt: "使用 AirXDB 配合 AirDbg 做图形界面调试;远程设备优先调用 remote device helper 并自动配置截图工具。"

View File

@@ -0,0 +1,159 @@
# Midscene Official Notes
用于 `airxdb` 的轻量参考,不替代官方文档。
## 选择模式
- Web + 现有 Playwright 项目:
- 优先 Midscene Playwright 集成
- 适合浏览器页面复现、E2E、界面交互不稳定问题
- Web + 需要复用本机 Chrome 的 cookies / 已登录状态 / 扩展:
- 使用 Chrome Bridge Mode
- 桌面应用 GUI
- 使用 Midscene Computer 或 Playground
- 需要给上层 Agent / MCP 客户端暴露 GUI 操作:
- 浏览器用 Web Bridge MCP
- 桌面用 Computer MCP
## 关键能力
- Midscene 的 UI 操作以纯视觉为主,可用于 Web、移动端、桌面端和 Canvas。
- Midscene 支持生成 HTML 报告,适合作为 GUI debug 证据。
- Midscene 可通过 MCP 暴露截图和动作空间操作。
## 首次模型配置
AirXDB 第一次进入项目时先检查这些变量:
```bash
MIDSCENE_MODEL_NAME
MIDSCENE_MODEL_BASE_URL
MIDSCENE_MODEL_API_KEY
MIDSCENE_MODEL_FAMILY
MCP_SERVER_REQUEST_TIMEOUT
```
前三个是连接模型服务的基本配置。`MIDSCENE_MODEL_FAMILY` 是视觉语义动作必填项,`MCP_SERVER_REQUEST_TIMEOUT` 按模型服务情况补充。
常见 `MIDSCENE_MODEL_FAMILY`
- `gpt-5`GPT-5.x 视觉模型,例如 `gpt-5.4`
- `qwen2.5-vl`
- `qwen3-vl`
- `gemini`
- `doubao-seed`
- `vlm-ui-tars`
不要把真实 API key 写入 ADR、C4、debug log 或可提交文档。需要本机持久化时优先使用 `AirPlan/state/airxdb/midscene.local.env`
## Chrome Bridge Mode
- 官方说明:
- 需要 Midscene Chrome 插件
- 终端侧配置模型环境变量
- 适合复用本地浏览器登录态和页面状态
- 常用依赖:
```bash
npm install @midscene/web tsx --save-dev
```
- 常见入口:
```ts
import { AgentOverChromeBridge } from "@midscene/web/bridge-mode";
```
- 常见运行方式:
```bash
tsx demo-new-tab.ts
```
- 常见模型环境变量:
```bash
MIDSCENE_MODEL_BASE_URL
MIDSCENE_MODEL_API_KEY
MIDSCENE_MODEL_NAME
MIDSCENE_MODEL_FAMILY
```
## MCP
- 浏览器桥接 MCP
```text
@midscene/web-bridge-mcp
```
- 桌面 MCP
```text
@midscene/computer-mcp
```
- Computer MCP 常见配置核心:
```json
{
"command": "npx",
"args": ["-y", "@midscene/computer-mcp"]
}
```
- 常见 MCP 模型环境变量:
```bash
MIDSCENE_MODEL_BASE_URL
MIDSCENE_MODEL_API_KEY
MIDSCENE_MODEL_NAME
MIDSCENE_MODEL_FAMILY
MCP_SERVER_REQUEST_TIMEOUT
```
## 桌面自动化
- Midscene 支持 Windows、macOS、Linux 桌面自动化。
- 桌面控制包括鼠标、键盘、截图、多显示器。
- Linux 可在 Xvfb 下做无头执行。
- Windows 下 `@midscene/computer-mcp` 的 npx 缓存包可能缺 `dist/screenCapture_1.3.2.bat``dist/app.manifest`。AirXDB smoke test 会从 `screenshot-desktop@1.15.3` npm 包自动补齐。
- 桌面 GUI 调试可优先考虑:
- 快速试用Playground
- 持续脚本化Computer SDK / MCP
## AirXDB Smoke Test
```bash
python "$HOME/plugins/airxdb/scripts/airxdb_computer_mcp_smoke.py" --project . --action mousemove --prompt "Windows taskbar Start button"
```
输出:
- `airxdb_smoke=ok`Computer MCP 连接、截图、语义动作完成。
- `airxdb_smoke=blocked`:缺模型配置或 family 不合法。
- `asset_repair=repaired`:已补齐 Windows 截图脚本。
- `report=<path>`JSON 报告API key 已脱敏。
## AirXDB Screenshot Evidence
截图取证不需要模型配置:
```bash
python "$HOME/plugins/airxdb/scripts/airxdb_computer_mcp_smoke.py" --project . --action screenshot
```
用于:
-`airdbg` 提供 GUI 错误现场
- 捕获弹窗、遮挡、焦点、布局错位、任务栏/托盘状态
- 记录多显示器和当前桌面状态
截图可能包含敏感信息。写入 ADR/C4/debug log 时记录路径和观察,不复制密钥或隐私内容。
## GUI Debug 推荐策略
1. 先选模式,不要一上来混用多种接入。
2. 先做最小复现,再考虑长期自动化。
3. 保留 HTML 报告、截图和复现命令。
4. 若问题已定位到代码层,把证据交给 `airdbg` 做修复。