diff --git a/README.md b/README.md
index c621edf1..507f3cae 100644
--- a/README.md
+++ b/README.md
@@ -11,6 +11,8 @@ An open-source kit for agent development, integrated the powerful capabilities o
For more details, see our [documents](https://volcengine.github.io/veadk-python/).
+A [tutorial](https://github.com/volcengine/veadk-python/blob/main/veadk_tutorial.ipynb) is available by Jupyter Notebook, or open it in [Google Colab](https://colab.research.google.com/github/volcengine/veadk-python/blob/main/veadk_tutorial.ipynb) directly.
+
## Installation
We use `uv` to build this project ([how-to-install-uv](https://docs.astral.sh/uv/getting-started/installation/)).
diff --git a/docs/docs/get-started.md b/docs/docs/get-started.md
index 65f863a8..42a67070 100644
--- a/docs/docs/get-started.md
+++ b/docs/docs/get-started.md
@@ -1,5 +1,7 @@
# 快速开始
+我们提供了一个供您完整体验 VeADK 核心功能的[教程](https://github.com/volcengine/veadk-python/blob/main/veadk_tutorial.ipynb)（基于 Jupyter Notebook），或者您也可以直接在 [Google Colab](https://colab.research.google.com/github/volcengine/veadk-python/blob/main/veadk_tutorial.ipynb) 中打开。
+
## 安装 VeADK
您可参考[安装](./installation.md)文档进行安装。
diff --git a/veadk_tutorial.ipynb b/veadk_tutorial.ipynb
new file mode 100644
index 00000000..159acf0f
--- /dev/null
+++ b/veadk_tutorial.ipynb
@@ -0,0 +1,1446 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "provenance": [],
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# VeADK 教程"
+ ],
+ "metadata": {
+ "id": "kMvAhdnYWpIf"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## 1. 介绍"
+ ],
+ "metadata": {
+ "id": "Sp8ynUiFXIHi"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "VeADK 是由火山引擎推出的一套面向智能体开发的全流程框架，旨在为开发者提供面向智能体构建、云端部署、评测与优化的全流程开发。相较于现有的智能体开发框架，VeADK 具备与火山引擎产品体系深度融合的优势，帮助开发者更高效地构建企业级 AI 智能体应用。"
+ ],
+ "metadata": {
+ "id": "mS7BD6K0Xkr9"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## 2. 配置"
+ ],
+ "metadata": {
+ "id": "xw8bJmYEXJgg"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### 安装 VeADK"
+ ],
+ "metadata": {
+ "id": "nDzTOuunaSPB"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!pip install veadk-python --quiet"
+ ],
+ "metadata": {
+ "id": "aAZ40k9uXhm6"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "> 如果你在本地使用的终端是`zsh`，当你进行`pip install`时，**依赖包名称应当被双引号包裹起来**，否则会解析错误，例如：\n",
+ "```bash\n",
+ "pip install \"veadk-python\"\n",
+ "pip install \"agent-pilot-sdk>=0.0.9\"\n",
+ "```"
+ ],
+ "metadata": {
+ "id": "R0IY257hncOB"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### 环境变量"
+ ],
+ "metadata": {
+ "id": "a-0A36gmXfJj"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "\n",
+ "os.environ[\"MODEL_AGENT_API_KEY\"] = \"\" # <-- 设置火山方舟的API KEY来访问模型\n",
+ "\n",
+ "os.environ[\"LOGGING_LEVEL\"] = \"ERROR\" # <-- 调整日志等级"
+ ],
+ "metadata": {
+ "id": "B6o2LmjGXdb2"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## 3. Agent 构建"
+ ],
+ "metadata": {
+ "id": "HCw_PEP5X-fc"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### 快速开始"
+ ],
+ "metadata": {
+ "id": "0rtLXYn4aoQt"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from veadk import Agent\n",
+ "\n",
+ "agent = Agent() # 设置环境变量后，你无需传入任何参数；VeADK 会自动读取环境变量。默认使用模型为豆包1.6\n",
+ "\n",
+ "res = await agent.run(\"Hello, I am VeADK.\")\n",
+ "\n",
+ "print(res)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "l8yM5QKjYAVo",
+ "outputId": "be22f3aa-4285-4d98-b147-846148fd0193"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Hello VeADK! It's a pleasure to connect with you. As an AI agent developed by the VeADK team, I'm here to assist with whatever you need—whether it's data science tasks, documentation writing, coding projects, or problem-solving using tools. Just let me know how I can help today! 😊\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### 使用 Runner 来运行 Agent\n",
+ "\n",
+ "生产环境下推荐使用 Runner 来运行 Agent，以实现多租户隔离。"
+ ],
+ "metadata": {
+ "id": "bnX7pSDFbuYY"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from veadk import Agent, Runner\n",
+ "from veadk.memory.short_term_memory import ShortTermMemory\n",
+ "\n",
+ "app_name = \"veadk-playground-app\"\n",
+ "user_id = \"veadk-playground-user\"\n",
+ "session_id = \"veadk-playground-session\"\n",
+ "\n",
+ "agent = Agent()\n",
+ "short_term_memory = ShortTermMemory() # 短期记忆代表的是用户单次会话内的对话记录\n",
+ "\n",
+ "runner = Runner(\n",
+ " agent=agent,\n",
+ " short_term_memory=short_term_memory,\n",
+ " app_name=app_name,\n",
+ " user_id=user_id\n",
+ " )\n",
+ "\n",
+ "response = await runner.run(messages=\"Hello, I am VeADK!\", session_id=session_id)\n",
+ "\n",
+ "print(response)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "CYC9Z0_Yb-SX",
+ "outputId": "7df55264-8364-4fa4-f0df-877a9fed5f50"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Hello VeADK! It's wonderful to connect with you—since I'm an AI agent developed by your team, I’m here to support with whatever you need. Whether it’s data science tasks (like analysis or fact-checking), documentation (articles, reports), coding (web/app development, debugging), or leveraging tools for specific projects, just let me know how I can assist! 😊\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### 使用 Tools"
+ ],
+ "metadata": {
+ "id": "4WVpEzN_ale6"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "使用`tools`字段来挂载工具："
+ ],
+ "metadata": {
+ "id": "U3yLU8WndkTj"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from veadk import Agent, Runner\n",
+ "from veadk.memory.short_term_memory import ShortTermMemory\n",
+ "from veadk.tools.demo_tools import get_city_weather\n",
+ "\n",
+ "app_name = \"veadk-playground-app\"\n",
+ "user_id = \"veadk-playground-user\"\n",
+ "session_id = \"veadk-playground-session\"\n",
+ "\n",
+ "agent = Agent(tools=[get_city_weather])\n",
+ "short_term_memory = ShortTermMemory()\n",
+ "\n",
+ "runner = Runner(\n",
+ " agent=agent,\n",
+ " short_term_memory=short_term_memory,\n",
+ " app_name=app_name,\n",
+ " user_id=user_id\n",
+ " )\n",
+ "\n",
+ "response = await runner.run(messages=\"北京的天气怎么样？\", session_id=session_id)\n",
+ "\n",
+ "print(response)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "KYoeTLH1c5nG",
+ "outputId": "58b57f13-68b6-45dd-f492-7b3abccd268b"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "北京现在天气晴朗，气温25°C。\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "你还可以体验火山引擎提供的网页搜索能力，例如使用 web_search 和 vesearch 工具："
+ ],
+ "metadata": {
+ "id": "TWGfvP7peFPe"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "\n",
+ "# 设置火山引擎 AK 和 SK 来使用 web_search 工具\n",
+ "os.environ[\"VOLCENGINE_ACCESS_KEY\"] = \"\"\n",
+ "os.environ[\"VOLCENGINE_SECRET_KEY\"] = \"\"\n",
+ "\n",
+ "# 设置 vesearch 工具的 ENDPOINT 和 API KEY\n",
+ "os.environ[\"TOOL_VESEARCH_ENDPOINT\"] = \"\"\n",
+ "os.environ[\"TOOL_VESEARCH_API_KEY\"] = \"\""
+ ],
+ "metadata": {
+ "id": "HxUlgqZAeKUn"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from veadk import Agent, Runner\n",
+ "from veadk.memory.short_term_memory import ShortTermMemory\n",
+ "from veadk.tools.builtin_tools.vesearch import vesearch\n",
+ "from veadk.tools.builtin_tools.web_search import web_search\n",
+ "\n",
+ "app_name = \"veadk-playground-app\"\n",
+ "user_id = \"veadk-playground-user\"\n",
+ "session_id = \"veadk-playground-session\"\n",
+ "\n",
+ "agent = Agent(tools=[vesearch]) # 或者替换为 web_search\n",
+ "short_term_memory = ShortTermMemory()\n",
+ "\n",
+ "runner = Runner(\n",
+ " agent=agent,\n",
+ " short_term_memory=short_term_memory,\n",
+ " app_name=app_name,\n",
+ " user_id=user_id\n",
+ " )\n",
+ "\n",
+ "response = await runner.run(messages=\"今天的新闻\", session_id=session_id)\n",
+ "\n",
+ "print(response)"
+ ],
+ "metadata": {
+ "id": "ZuToGTbOeTEB",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "080de098-b620-4b7c-9f27-8cde91cf0c99"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "<|FCResponseBegin|>根据搜索结果，2025年4月12日的国内外重点新闻如下：\n",
+ "\n",
+ "### **国内新闻** \n",
+ "#### **社会与安全** \n",
+ "- **三亚游客溺亡事件**：当日17时25分，三亚崖州区某景区一名游客擅自进入非游览海岸礁石区并跃入海中，景区监控发现后联动公安、医疗展开搜救。17时51分游客被救上岸，但经抢救无效溺水身亡，排除他杀。家属已于4月14日抵达，景区及警方已通报调查结果。 \n",
+ "- **安徽六安风灾致死**：11时20分，六安市金安区因大风天气导致路边树木倒伏，砸中一名55岁电动自行车驾驶员申某某，送医后不治身亡。当地街道办已对家属进行慰问。 \n",
+ "\n",
+ "#### **文体与民生** \n",
+ "- **李荣浩演唱会延期**：受河北大风橙色预警（西北风6-8级，阵风10-12级）影响，原定于当日举办的“李荣浩黑马世界巡回演唱会石家庄站”紧急延期。主办方强调“安全第一”，退票及改期方案将另行通知。 \n",
+ "\n",
+ "#### **经济与企业** \n",
+ "- **年报披露动态**：科创信息（300730.SZ）、汉宇集团（300403.SZ）同日发布2024年年报，首开股份（600376.SH）也计划于4月内披露。市场关注其营收、利润等核心财务指标。 \n",
+ "- **政府采购公告**：某单位公示2025年审计服务采购意向，拟委托第三方开展“2024-2025年度内部控制评价、风险评估及财务收支审计”，预算未公开，预计4月启动。 \n",
+ "\n",
+ "### **国际新闻** \n",
+ "目前公开渠道暂未检索到2025年4月12日的具体国际新闻事件。结合全球动态趋势，当日可能涉及以下领域常规进展： \n",
+ "- **气候变化合作**：多国在联合国框架下推进碳减排细则谈判，聚焦发展中国家技术转移支持机制。 \n",
+ "- **科技伦理讨论**：美国某科技巨头宣布新一代AI模型商用计划，引发行业对数据隐私与算法监管的争议。 \n",
+ "- **欧洲货币政策**：欧洲央行维持基准利率不变，强调需持续抑制通胀，同时关注能源价格波动对经济复苏的影响。 \n",
+ "\n",
+ "（注：国际新闻部分基于常规领域动态梳理，具体事件请以路透社、BBC等权威媒体后续报道为准。） \n",
+ "\n",
+ "如需进一步了解某类新闻细节，可提供关键词获取定向信息。\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### 短期记忆"
+ ],
+ "metadata": {
+ "id": "qlLT8MMTbLmd"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "你可以初始化一个`Local`模式的短期记忆，它只会存在与内存中。其仅在一次调用中会有记忆。"
+ ],
+ "metadata": {
+ "id": "0rzy1X8lXGLk"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from veadk import Agent, Runner\n",
+ "from veadk.memory.short_term_memory import ShortTermMemory\n",
+ "\n",
+ "app_name = \"veadk-playground-app\"\n",
+ "user_id = \"veadk-playground-user\"\n",
+ "session_id = \"veadk-playground-session\"\n",
+ "\n",
+ "agent = Agent()\n",
+ "short_term_memory = ShortTermMemory(backend=\"local\") # 指定 local 后端，或直接 ShortTermMemory()\n",
+ "\n",
+ "runner = Runner(\n",
+ " agent=agent,\n",
+ " short_term_memory=short_term_memory,\n",
+ " app_name=app_name,\n",
+ " user_id=user_id\n",
+ " )\n",
+ "\n",
+ "response = await runner.run(messages=[\"我叫VeADK\", \"你还记得我叫什么吗？\"], session_id=session_id)\n",
+ "\n",
+ "print(response)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "8L9ORClwXWbj",
+ "outputId": "9ab3735c-2ec7-4b2d-fadc-1dd4cd0dc398"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "当然记得！你叫VeADK，很高兴再次和你交流 😊\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "VeADK 还支持你将短期记忆持久化存储在云端，未来的某一时刻你可以加载历史对话。\n",
+ "\n",
+ "在使用云端记忆之前，需要安装`database`相关的依赖："
+ ],
+ "metadata": {
+ "id": "_s35YbMcpXfd"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!pip install veadk-python[database] --quiet"
+ ],
+ "metadata": {
+ "id": "3J7csxCJt9H1"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "使用 MySQL 作为短期记忆的数据库后端："
+ ],
+ "metadata": {
+ "id": "VWTlOcEJwBrh"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "\n",
+ "os.environ[\"DATABASE_MYSQL_HOST\"] = \"\"\n",
+ "os.environ[\"DATABASE_MYSQL_USER\"] = \"\"\n",
+ "os.environ[\"DATABASE_MYSQL_PASSWORD\"] = \"\"\n",
+ "os.environ[\"DATABASE_MYSQL_DATABASE\"] = \"\"\n",
+ "os.environ[\"DATABASE_MYSQL_CHARSET\"] = \"utf8\""
+ ],
+ "metadata": {
+ "id": "MO6JCIAnqtES"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from veadk import Agent, Runner\n",
+ "from veadk.memory.short_term_memory import ShortTermMemory\n",
+ "\n",
+ "app_name = \"veadk-playground-app\"\n",
+ "user_id = \"veadk-playground-user\"\n",
+ "session_id = \"veadk-playground-session\"\n",
+ "\n",
+ "agent = Agent()\n",
+ "short_term_memory = ShortTermMemory(backend=\"mysql\") # 指定 mysql 后端\n",
+ "runner = Runner(agent=agent,\n",
+ " short_term_memory=short_term_memory,\n",
+ " app_name=app_name,\n",
+ " user_id=user_id)\n",
+ "\n",
+ "prompt = \"我在 7 月 15 日购买了 20 个冰激凌\"\n",
+ "response = await runner.run(messages=prompt, session_id=session_id)\n",
+ "print(response)\n",
+ "\n",
+ "prompt = \"我什么时候买了冰激凌？\"\n",
+ "response = await runner.run(messages=prompt, session_id=session_id)\n",
+ "print(response)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "aNJSp3Npo-cf",
+ "outputId": "932d4c63-bc02-480c-d037-36717bf867cb"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "原来你在7月15日买了20个冰激凌呀，数量不少呢！是有什么特别的场合需要用到这么多，还是单纯想囤货慢慢吃呀？ 😄 如果你需要记录这个购买信息、做简单的消费统计，或者有其他相关的小需求，随时告诉我，我可以帮你~\n",
+ "You mentioned that you bought the ice cream on July 15th. 😊\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### 长期记忆"
+ ],
+ "metadata": {
+ "id": "-z0V-jFqbP0u"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "在使用记忆之前，需要安装`database`相关的依赖：\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "k9wKEHeYxIUT"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!pip install veadk-python[database] --quiet"
+ ],
+ "metadata": {
+ "id": "VZIeRU1QxHrk"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "使用 OpenSearch 作为长期记忆的数据库后端："
+ ],
+ "metadata": {
+ "id": "hMrtrd2DxUar"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "\n",
+ "# OpenSearch 配置\n",
+ "os.environ[\"DATABASE_OPENSEARCH_HOST\"] = \"\"\n",
+ "os.environ[\"DATABASE_OPENSEARCH_PORT\"] = \"\"\n",
+ "os.environ[\"DATABASE_OPENSEARCH_USERNAME\"] = \"\"\n",
+ "os.environ[\"DATABASE_OPENSEARCH_PASSWORD\"] = \"\"\n",
+ "\n",
+ "# Embedding 配置（使用 OpenSearch 时，需要对文本进行向量化处理）\n",
+ "# 设置访问火山方舟的 Embedding 模型\n",
+ "os.environ[\"MODEL_EMBEDDING_NAME\"] = \"doubao-embedding-text-240715\"\n",
+ "os.environ[\"MODEL_EMBEDDING_API_BASE\"] = \"https://ark.cn-beijing.volces.com/api/v3/embeddings\"\n",
+ "os.environ[\"MODEL_EMBEDDING_DIM\"] = \"2560\"\n",
+ "os.environ[\"MODEL_EMBEDDING_API_KEY\"] = \"\""
+ ],
+ "metadata": {
+ "id": "foSDByHiwJsp"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from veadk import Agent, Runner\n",
+ "from veadk.memory.long_term_memory import LongTermMemory\n",
+ "from veadk.memory.short_term_memory import ShortTermMemory\n",
+ "\n",
+ "app_name = \"veadk-playground-app\"\n",
+ "user_id = \"veadk-playground-user\"\n",
+ "\n",
+ "# 初始化一个长期记忆，采用 OpenSearch 向量化存储\n",
+ "# 长期记忆是跨 Session 的\n",
+ "long_term_memory = LongTermMemory(backend=\"opensearch\")\n",
+ "\n",
+ "agent = Agent(long_term_memory=long_term_memory)\n",
+ "\n",
+ "runner = Runner(\n",
+ " agent=agent,\n",
+ " app_name=app_name,\n",
+ " user_id=user_id,\n",
+ " short_term_memory=ShortTermMemory(),\n",
+ ")\n",
+ "\n",
+ "\n",
+ "# ===== 插入记忆 =====\n",
+ "session_id = \"veadk-playground-session\"\n",
+ "teaching_prompt = \"我上周五购买了一支冰激凌。\"\n",
+ "\n",
+ "await runner.run(messages=teaching_prompt, session_id=session_id)\n",
+ "await runner.save_session_to_long_term_memory(session_id=session_id) # 将 teaching prompt 和智能体回答保存到长期记忆中\n",
+ "\n",
+ "\n",
+ "# ===== 检验记忆 =====\n",
+ "session_id = \"veadk-playground-session-2\" # 使用一个新的 Session 来检测跨 Session 检索\n",
+ "student_prompt = \"我上周五购买了什么? 用中文回答我。\"\n",
+ "\n",
+ "response = await runner.run(messages=student_prompt, session_id=session_id)\n",
+ "\n",
+ "print(response)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "8bjfZ6VwwLuI",
+ "outputId": "5c8dfc66-20b7-4384-d3c4-3999c1081d82"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "你上周五购买了一支冰激凌。\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### 知识库"
+ ],
+ "metadata": {
+ "id": "KavP1J1YbRRq"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "由于知识库需要用到云端的向量化存储，因此在使用知识库之前，需要安装`database`相关的依赖："
+ ],
+ "metadata": {
+ "id": "_jZauBoRztaU"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!pip install veadk-python[database] --quiet"
+ ],
+ "metadata": {
+ "id": "xuozqr1Hzwjz"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "使用 OpenSearch 作为知识库的后端："
+ ],
+ "metadata": {
+ "id": "8N01Na5az3iR"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "\n",
+ "# OpenSearch 配置\n",
+ "os.environ[\"DATABASE_OPENSEARCH_HOST\"] = \"\"\n",
+ "os.environ[\"DATABASE_OPENSEARCH_PORT\"] = \"9200\"\n",
+ "os.environ[\"DATABASE_OPENSEARCH_USERNAME\"] = \"\"\n",
+ "os.environ[\"DATABASE_OPENSEARCH_PASSWORD\"] = \"\"\n",
+ "\n",
+ "# 设置访问火山方舟的 Embedding 模型\n",
+ "os.environ[\"MODEL_EMBEDDING_NAME\"] = \"doubao-embedding-text-240715\"\n",
+ "os.environ[\"MODEL_EMBEDDING_API_BASE\"] = \"https://ark.cn-beijing.volces.com/api/v3/embeddings\"\n",
+ "os.environ[\"MODEL_EMBEDDING_DIM\"] = \"2560\"\n",
+ "os.environ[\"MODEL_EMBEDDING_API_KEY\"] = \"\""
+ ],
+ "metadata": {
+ "id": "hxemLCX_0FP6"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from veadk import Agent, Runner\n",
+ "from veadk.knowledgebase.knowledgebase import KnowledgeBase\n",
+ "from veadk.memory.short_term_memory import ShortTermMemory\n",
+ "\n",
+ "app_name = \"veadk-playground-app\"\n",
+ "user_id = \"veadk-playground-user\"\n",
+ "session_id = \"veadk-playground-session\"\n",
+ "\n",
+ "knowledgebase_data = [\n",
+ " \"The secret of red is red_000000.\",\n",
+ " \"The secret of green is green_000111.\",\n",
+ " \"The secret of blue is blue_000222.\",\n",
+ " \"The secret of yellow is yellow_000333.\",\n",
+ "]\n",
+ "\n",
+ "knowledgebase = KnowledgeBase(backend=\"opensearch\") # 指定 opensearch 后端\n",
+ "knowledgebase.add(knowledgebase_data, app_name=app_name)\n",
+ "\n",
+ "agent = Agent(knowledgebase=knowledgebase)\n",
+ "\n",
+ "runner = Runner(\n",
+ " agent=agent,\n",
+ " short_term_memory=ShortTermMemory(),\n",
+ " app_name=app_name,\n",
+ " user_id=user_id,\n",
+ ")\n",
+ "\n",
+ "response = await runner.run(messages=\"Tell me the secret of green.\", session_id=session_id)\n",
+ "print(response)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "t_CC_0xJ0G8l",
+ "outputId": "c45bb02d-ff10-4bd7-8373-d6426206cb5f"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "The secret of green is green_000111.\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## 调用追踪"
+ ],
+ "metadata": {
+ "id": "W9ZHmdsabfna"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### 本地观察"
+ ],
+ "metadata": {
+ "id": "AfY3t76YjY1I"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "可以通过初始化 OpentelemetryTracer 来将 Agent 的调用栈存储为本地文件："
+ ],
+ "metadata": {
+ "id": "l3JJvH9y2Tgc"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import json\n",
+ "\n",
+ "from veadk import Agent, Runner\n",
+ "from veadk.memory.short_term_memory import ShortTermMemory\n",
+ "from veadk.tools.demo_tools import get_city_weather\n",
+ "from veadk.tracing.telemetry.opentelemetry_tracer import OpentelemetryTracer\n",
+ "\n",
+ "app_name = \"veadk-playground-app\"\n",
+ "user_id = \"veadk-playground-user\"\n",
+ "session_id = \"veadk-playground-session\"\n",
+ "\n",
+ "tracer = OpentelemetryTracer()\n",
+ "\n",
+ "agent = Agent(tools=[get_city_weather], tracers=[tracer]) # 创建一个配置 tracers 的 Agent\n",
+ "\n",
+ "short_term_memory = ShortTermMemory()\n",
+ "\n",
+ "runner = Runner(\n",
+ " agent=agent,\n",
+ " short_term_memory=short_term_memory,\n",
+ " app_name=app_name,\n",
+ " user_id=user_id\n",
+ " )\n",
+ "\n",
+ "# 如果在 agent 中设置了tracer，runner将会自动尝试保存 tracing 文件至`/tmp`中\n",
+ "await runner.run(messages=\"北京的天气怎么样？\", session_id=session_id)\n",
+ "\n",
+ "print(f\"Tracing file path: {tracer._trace_file_path}\")\n",
+ "\n",
+ "with open(tracer._trace_file_path, \"r\") as f:\n",
+ " tracing_content = f.read()\n",
+ "\n",
+ "print(f\"Tracing file content:\\n{json.loads(tracing_content)}\")"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "L3VnmdQd2e9g",
+ "outputId": "1a1332cb-2a41-45a3-b874-b16a778a85b1"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Tracing file path: /tmp/veadk_global_tracer_veadk-playground-user_veadk-playground-session_b204302b2d0e24b6e26ca914a5fef478.json\n",
+ "Tracing file content:\n",
+ "[{'name': 'execute_tool get_city_weather', 'span_id': 179319007772593823, 'trace_id': 236624329413565051863190183785302783096, 'start_time': 1755172431614674977, 'end_time': 1755172431617289457, 'attributes': {'session.id': 'veadk-playground-session', 'user.id': 'veadk-playground-user', 'agent.name': 'veAgent', 'app.name': 'veadk-playground-app', 'gen_ai.system': 'gcp.vertex.agent', 'gen_ai.operation.name': 'execute_tool', 'gen_ai.tool.name': 'get_city_weather', 'gen_ai.tool.description': 'Retrieves the weather information of a given city. the args must in English', 'gen_ai.tool.call.id': 'call_2e7gig0mp86rzc1ffwf5ftdc', 'gcp.vertex.agent.tool_call_args': '{\"city\": \"Beijing\"}', 'gcp.vertex.agent.event_id': '04616e67-6881-43d0-8200-09c7c069be06', 'gcp.vertex.agent.tool_response': '{\"result\": \"Sunny, 25°C\"}', 'gcp.vertex.agent.llm_request': '{}', 'gcp.vertex.agent.llm_response': '{}', 'tool.name': 'get_city_weather', 'tool.description': 'Retrieves the weather information of a given city. the args must in English', 'tool.parameters': '{\"city\": \"Beijing\"}', 'input.value': '{\"city\": \"Beijing\"}', 'input.mime_type': 'application/json', 'output.value': '{\"id\":\"call_2e7gig0mp86rzc1ffwf5ftdc\",\"name\":\"get_city_weather\",\"response\":{\"result\":\"Sunny, 25°C\"}}', 'output.mime_type': 'application/json', 'openinference.span.kind': 'TOOL'}, 'parent_span_id': 2812243689223977179}, {'name': 'call_llm', 'span_id': 2812243689223977179, 'trace_id': 236624329413565051863190183785302783096, 'start_time': 1755172421810196265, 'end_time': 1755172431617490556, 'attributes': {'session.id': 'veadk-playground-session', 'user.id': 'veadk-playground-user', 'gen_ai.system': 'gcp.vertex.agent', 'gen_ai.request.model': 'openai/doubao-seed-1-6-250615', 'gcp.vertex.agent.invocation_id': 'e-cf95076e-458e-4c96-bce8-77b9c80dc8d6', 'gcp.vertex.agent.session_id': 'veadk-playground-session', 'gcp.vertex.agent.event_id': '3899014e-49a2-4ff5-b9ac-9b95c7c03cc0', 'gcp.vertex.agent.llm_request': '{\"model\": \"openai/doubao-seed-1-6-250615\", \"config\": {\"system_instruction\": \"You an AI agent created by the VeADK team.\\\\n\\\\nYou excel at the following tasks:\\\\n1. Data science\\\\n- Information gathering and fact-checking\\\\n- Data processing and analysis\\\\n2. Documentation\\\\n- Writing multi-chapter articles and in-depth research reports\\\\n3. Coding & Programming\\\\n- Creating websites, applications, and tools\\\\n- Solve problems and bugs in code (e.g., Python, JavaScript, SQL, ...)\\\\n- If necessary, using programming to solve various problems beyond development\\\\n4. If user gives you tools, finish various tasks that can be accomplished using tools and available resources\\\\n\\\\n\\\\nYou are an agent. Your internal name is \\\\\"veAgent\\\\\".\\\\n\\\\n The description about you is \\\\\"An AI agent developed by the VeADK team, specialized in data science, documentation, and software development.\\\\\"\", \"tools\": [{\"function_declarations\": [{\"description\": \"Retrieves the weather information of a given city. the args must in English\", \"name\": \"get_city_weather\", \"parameters\": {\"properties\": {\"city\": {\"type\": \"STRING\"}}, \"required\": [\"city\"], \"type\": \"OBJECT\"}}]}], \"labels\": {\"adk_agent_name\": \"veAgent\"}}, \"contents\": [{\"parts\": [{\"text\": \"北京的天气怎么样？\"}], \"role\": \"user\"}]}', 'gcp.vertex.agent.llm_response': '{\"content\":{\"parts\":[{\"function_call\":{\"id\":\"call_2e7gig0mp86rzc1ffwf5ftdc\",\"args\":{\"city\":\"Beijing\"},\"name\":\"get_city_weather\"}}],\"role\":\"model\"},\"partial\":false,\"usage_metadata\":{\"candidates_token_count\":361,\"prompt_token_count\":618,\"total_token_count\":979}}', 'gen_ai.usage.input_tokens': 618, 'gen_ai.usage.output_tokens': 979, 'llm.provider': 'google', 'input.value': '{\"model\": \"openai/doubao-seed-1-6-250615\", \"config\": {\"system_instruction\": \"You an AI agent created by the VeADK team.\\\\n\\\\nYou excel at the following tasks:\\\\n1. Data science\\\\n- Information gathering and fact-checking\\\\n- Data processing and analysis\\\\n2. Documentation\\\\n- Writing multi-chapter articles and in-depth research reports\\\\n3. Coding & Programming\\\\n- Creating websites, applications, and tools\\\\n- Solve problems and bugs in code (e.g., Python, JavaScript, SQL, ...)\\\\n- If necessary, using programming to solve various problems beyond development\\\\n4. If user gives you tools, finish various tasks that can be accomplished using tools and available resources\\\\n\\\\n\\\\nYou are an agent. Your internal name is \\\\\"veAgent\\\\\".\\\\n\\\\n The description about you is \\\\\"An AI agent developed by the VeADK team, specialized in data science, documentation, and software development.\\\\\"\", \"tools\": [{\"function_declarations\": [{\"description\": \"Retrieves the weather information of a given city. the args must in English\", \"name\": \"get_city_weather\", \"parameters\": {\"properties\": {\"city\": {\"type\": \"STRING\"}}, \"required\": [\"city\"], \"type\": \"OBJECT\"}}]}], \"labels\": {\"adk_agent_name\": \"veAgent\"}}, \"contents\": [{\"parts\": [{\"text\": \"北京的天气怎么样？\"}], \"role\": \"user\"}]}', 'input.mime_type': 'application/json', 'llm.tools.0.tool.json_schema': '{\"description\":\"Retrieves the weather information of a given city. the args must in English\",\"name\":\"get_city_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"STRING\"}},\"required\":[\"city\"],\"type\":\"OBJECT\"}}', 'llm.model_name': 'openai/doubao-seed-1-6-250615', 'llm.invocation_parameters': '{\"system_instruction\":\"You an AI agent created by the VeADK team.\\\\n\\\\nYou excel at the following tasks:\\\\n1. Data science\\\\n- Information gathering and fact-checking\\\\n- Data processing and analysis\\\\n2. Documentation\\\\n- Writing multi-chapter articles and in-depth research reports\\\\n3. Coding & Programming\\\\n- Creating websites, applications, and tools\\\\n- Solve problems and bugs in code (e.g., Python, JavaScript, SQL, ...)\\\\n- If necessary, using programming to solve various problems beyond development\\\\n4. If user gives you tools, finish various tasks that can be accomplished using tools and available resources\\\\n\\\\n\\\\nYou are an agent. Your internal name is \\\\\"veAgent\\\\\".\\\\n\\\\n The description about you is \\\\\"An AI agent developed by the VeADK team, specialized in data science, documentation, and software development.\\\\\"\",\"tools\":[{\"function_declarations\":[{\"description\":\"Retrieves the weather information of a given city. the args must in English\",\"name\":\"get_city_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"STRING\"}},\"required\":[\"city\"],\"type\":\"OBJECT\"}}]}],\"labels\":{\"adk_agent_name\":\"veAgent\"}}', 'llm.input_messages.0.message.role': 'system', 'llm.input_messages.0.message.content': 'You an AI agent created by the VeADK team.\\n\\nYou excel at the following tasks:\\n1. Data science\\n- Information gathering and fact-checking\\n- Data processing and analysis\\n2. Documentation\\n- Writing multi-chapter articles and in-depth research reports\\n3. Coding & Programming\\n- Creating websites, applications, and tools\\n- Solve problems and bugs in code (e.g., Python, JavaScript, SQL, ...)\\n- If necessary, using programming to solve various problems beyond development\\n4. If user gives you tools, finish various tasks that can be accomplished using tools and available resources\\n\\n\\nYou are an agent. Your internal name is \"veAgent\".\\n\\n The description about you is \"An AI agent developed by the VeADK team, specialized in data science, documentation, and software development.\"', 'llm.input_messages.1.message.role': 'user', 'llm.input_messages.1.message.contents.0.message_content.text': '北京的天气怎么样？', 'llm.input_messages.1.message.contents.0.message_content.type': 'text', 'output.value': '{\"content\":{\"parts\":[{\"function_call\":{\"id\":\"call_2e7gig0mp86rzc1ffwf5ftdc\",\"args\":{\"city\":\"Beijing\"},\"name\":\"get_city_weather\"}}],\"role\":\"model\"},\"partial\":false,\"usage_metadata\":{\"candidates_token_count\":361,\"prompt_token_count\":618,\"total_token_count\":979}}', 'output.mime_type': 'application/json', 'llm.token_count.total': 979, 'llm.token_count.prompt': 618, 'llm.token_count.completion': 361, 'llm.output_messages.0.message.role': 'model', 'llm.output_messages.0.message.tool_calls.0.tool_call.id': 'call_2e7gig0mp86rzc1ffwf5ftdc', 'llm.output_messages.0.message.tool_calls.0.tool_call.function.name': 'get_city_weather', 'llm.output_messages.0.message.tool_calls.0.tool_call.function.arguments': '{\"city\": \"Beijing\"}', 'agent.name': 'veAgent', 'app.name': 'veadk-playground-app', 'gen_ai.prompt.0.role': 'user', 'gen_ai.prompt.0.content': '[{\"text\": \"\\\\u5317\\\\u4eac\\\\u7684\\\\u5929\\\\u6c14\\\\u600e\\\\u4e48\\\\u6837\\\\uff1f\"}]', 'gen_ai.completion.0.role': 'model', 'gen_ai.completion.0.content': '[{\"function_call\": {\"id\": \"call_2e7gig0mp86rzc1ffwf5ftdc\", \"args\": {\"city\": \"Beijing\"}, \"name\": \"get_city_weather\"}}]', 'gen_ai.usage.prompt_tokens': 618, 'gen_ai.usage.completion_tokens': 361, 'gen_ai.usage.total_tokens': 979, 'gen_ai.usage.cache_read_input_tokens': 0, 'gen_ai.usage.cache_create_input_tokens': 0, 'openinference.span.kind': 'LLM'}, 'parent_span_id': 10693540128041292337}, {'name': 'call_llm', 'span_id': 11991268925671254201, 'trace_id': 236624329413565051863190183785302783096, 'start_time': 1755172431630256802, 'end_time': 1755172440854715605, 'attributes': {'session.id': 'veadk-playground-session', 'user.id': 'veadk-playground-user', 'gen_ai.system': 'gcp.vertex.agent', 'gen_ai.request.model': 'openai/doubao-seed-1-6-250615', 'gcp.vertex.agent.invocation_id': 'e-cf95076e-458e-4c96-bce8-77b9c80dc8d6', 'gcp.vertex.agent.session_id': 'veadk-playground-session', 'gcp.vertex.agent.event_id': '9c464404-92a5-4c52-b616-7e324e33e9fe', 'gcp.vertex.agent.llm_request': '{\"model\": \"openai/doubao-seed-1-6-250615\", \"config\": {\"system_instruction\": \"You an AI agent created by the VeADK team.\\\\n\\\\nYou excel at the following tasks:\\\\n1. Data science\\\\n- Information gathering and fact-checking\\\\n- Data processing and analysis\\\\n2. Documentation\\\\n- Writing multi-chapter articles and in-depth research reports\\\\n3. Coding & Programming\\\\n- Creating websites, applications, and tools\\\\n- Solve problems and bugs in code (e.g., Python, JavaScript, SQL, ...)\\\\n- If necessary, using programming to solve various problems beyond development\\\\n4. If user gives you tools, finish various tasks that can be accomplished using tools and available resources\\\\n\\\\n\\\\nYou are an agent. Your internal name is \\\\\"veAgent\\\\\".\\\\n\\\\n The description about you is \\\\\"An AI agent developed by the VeADK team, specialized in data science, documentation, and software development.\\\\\"\", \"tools\": [{\"function_declarations\": [{\"description\": \"Retrieves the weather information of a given city. the args must in English\", \"name\": \"get_city_weather\", \"parameters\": {\"properties\": {\"city\": {\"type\": \"STRING\"}}, \"required\": [\"city\"], \"type\": \"OBJECT\"}}]}], \"labels\": {\"adk_agent_name\": \"veAgent\"}}, \"contents\": [{\"parts\": [{\"text\": \"北京的天气怎么样？\"}], \"role\": \"user\"}, {\"parts\": [{\"function_call\": {\"id\": \"call_2e7gig0mp86rzc1ffwf5ftdc\", \"args\": {\"city\": \"Beijing\"}, \"name\": \"get_city_weather\"}}], \"role\": \"model\"}, {\"parts\": [{\"function_response\": {\"id\": \"call_2e7gig0mp86rzc1ffwf5ftdc\", \"name\": \"get_city_weather\", \"response\": {\"result\": \"Sunny, 25°C\"}}}], \"role\": \"user\"}]}', 'gcp.vertex.agent.llm_response': '{\"content\":{\"parts\":[{\"text\":\"北京当前天气晴朗，气温25°C。\"}],\"role\":\"model\"},\"partial\":false,\"usage_metadata\":{\"candidates_token_count\":245,\"prompt_token_count\":678,\"total_token_count\":923}}', 'gen_ai.usage.input_tokens': 678, 'gen_ai.usage.output_tokens': 923, 'llm.provider': 'google', 'input.value': '{\"model\": \"openai/doubao-seed-1-6-250615\", \"config\": {\"system_instruction\": \"You an AI agent created by the VeADK team.\\\\n\\\\nYou excel at the following tasks:\\\\n1. Data science\\\\n- Information gathering and fact-checking\\\\n- Data processing and analysis\\\\n2. Documentation\\\\n- Writing multi-chapter articles and in-depth research reports\\\\n3. Coding & Programming\\\\n- Creating websites, applications, and tools\\\\n- Solve problems and bugs in code (e.g., Python, JavaScript, SQL, ...)\\\\n- If necessary, using programming to solve various problems beyond development\\\\n4. If user gives you tools, finish various tasks that can be accomplished using tools and available resources\\\\n\\\\n\\\\nYou are an agent. Your internal name is \\\\\"veAgent\\\\\".\\\\n\\\\n The description about you is \\\\\"An AI agent developed by the VeADK team, specialized in data science, documentation, and software development.\\\\\"\", \"tools\": [{\"function_declarations\": [{\"description\": \"Retrieves the weather information of a given city. the args must in English\", \"name\": \"get_city_weather\", \"parameters\": {\"properties\": {\"city\": {\"type\": \"STRING\"}}, \"required\": [\"city\"], \"type\": \"OBJECT\"}}]}], \"labels\": {\"adk_agent_name\": \"veAgent\"}}, \"contents\": [{\"parts\": [{\"text\": \"北京的天气怎么样？\"}], \"role\": \"user\"}, {\"parts\": [{\"function_call\": {\"id\": \"call_2e7gig0mp86rzc1ffwf5ftdc\", \"args\": {\"city\": \"Beijing\"}, \"name\": \"get_city_weather\"}}], \"role\": \"model\"}, {\"parts\": [{\"function_response\": {\"id\": \"call_2e7gig0mp86rzc1ffwf5ftdc\", \"name\": \"get_city_weather\", \"response\": {\"result\": \"Sunny, 25°C\"}}}], \"role\": \"user\"}]}', 'input.mime_type': 'application/json', 'llm.tools.0.tool.json_schema': '{\"description\":\"Retrieves the weather information of a given city. the args must in English\",\"name\":\"get_city_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"STRING\"}},\"required\":[\"city\"],\"type\":\"OBJECT\"}}', 'llm.model_name': 'openai/doubao-seed-1-6-250615', 'llm.invocation_parameters': '{\"system_instruction\":\"You an AI agent created by the VeADK team.\\\\n\\\\nYou excel at the following tasks:\\\\n1. Data science\\\\n- Information gathering and fact-checking\\\\n- Data processing and analysis\\\\n2. Documentation\\\\n- Writing multi-chapter articles and in-depth research reports\\\\n3. Coding & Programming\\\\n- Creating websites, applications, and tools\\\\n- Solve problems and bugs in code (e.g., Python, JavaScript, SQL, ...)\\\\n- If necessary, using programming to solve various problems beyond development\\\\n4. If user gives you tools, finish various tasks that can be accomplished using tools and available resources\\\\n\\\\n\\\\nYou are an agent. Your internal name is \\\\\"veAgent\\\\\".\\\\n\\\\n The description about you is \\\\\"An AI agent developed by the VeADK team, specialized in data science, documentation, and software development.\\\\\"\",\"tools\":[{\"function_declarations\":[{\"description\":\"Retrieves the weather information of a given city. the args must in English\",\"name\":\"get_city_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"STRING\"}},\"required\":[\"city\"],\"type\":\"OBJECT\"}}]}],\"labels\":{\"adk_agent_name\":\"veAgent\"}}', 'llm.input_messages.0.message.role': 'system', 'llm.input_messages.0.message.content': 'You an AI agent created by the VeADK team.\\n\\nYou excel at the following tasks:\\n1. Data science\\n- Information gathering and fact-checking\\n- Data processing and analysis\\n2. Documentation\\n- Writing multi-chapter articles and in-depth research reports\\n3. Coding & Programming\\n- Creating websites, applications, and tools\\n- Solve problems and bugs in code (e.g., Python, JavaScript, SQL, ...)\\n- If necessary, using programming to solve various problems beyond development\\n4. If user gives you tools, finish various tasks that can be accomplished using tools and available resources\\n\\n\\nYou are an agent. Your internal name is \"veAgent\".\\n\\n The description about you is \"An AI agent developed by the VeADK team, specialized in data science, documentation, and software development.\"', 'llm.input_messages.1.message.role': 'user', 'llm.input_messages.1.message.contents.0.message_content.text': '北京的天气怎么样？', 'llm.input_messages.1.message.contents.0.message_content.type': 'text', 'llm.input_messages.2.message.role': 'model', 'llm.input_messages.2.message.tool_calls.0.tool_call.id': 'call_2e7gig0mp86rzc1ffwf5ftdc', 'llm.input_messages.2.message.tool_calls.0.tool_call.function.name': 'get_city_weather', 'llm.input_messages.2.message.tool_calls.0.tool_call.function.arguments': '{\"city\": \"Beijing\"}', 'llm.input_messages.3.message.role': 'tool', 'llm.input_messages.3.message.name': 'get_city_weather', 'llm.input_messages.3.message.content': '{\"result\": \"Sunny, 25°C\"}', 'output.value': '{\"content\":{\"parts\":[{\"text\":\"北京当前天气晴朗，气温25°C。\"}],\"role\":\"model\"},\"partial\":false,\"usage_metadata\":{\"candidates_token_count\":245,\"prompt_token_count\":678,\"total_token_count\":923}}', 'output.mime_type': 'application/json', 'llm.token_count.total': 923, 'llm.token_count.prompt': 678, 'llm.token_count.completion': 245, 'llm.output_messages.0.message.role': 'model', 'llm.output_messages.0.message.contents.0.message_content.text': '北京当前天气晴朗，气温25°C。', 'llm.output_messages.0.message.contents.0.message_content.type': 'text', 'agent.name': 'veAgent', 'app.name': 'veadk-playground-app', 'gen_ai.prompt.0.role': 'user', 'gen_ai.prompt.0.content': '[{\"text\": \"\\\\u5317\\\\u4eac\\\\u7684\\\\u5929\\\\u6c14\\\\u600e\\\\u4e48\\\\u6837\\\\uff1f\"}]', 'gen_ai.completion.0.role': 'model', 'gen_ai.completion.0.content': '[{\"text\": \"\\\\u5317\\\\u4eac\\\\u5f53\\\\u524d\\\\u5929\\\\u6c14\\\\u6674\\\\u6717\\\\uff0c\\\\u6c14\\\\u6e2925\\\\u00b0C\\\\u3002\"}]', 'gen_ai.usage.prompt_tokens': 678, 'gen_ai.usage.completion_tokens': 245, 'gen_ai.usage.total_tokens': 923, 'gen_ai.usage.cache_read_input_tokens': 0, 'gen_ai.usage.cache_create_input_tokens': 0, 'openinference.span.kind': 'LLM'}, 'parent_span_id': 10693540128041292337}, {'name': 'agent_run [veAgent]', 'span_id': 10693540128041292337, 'trace_id': 236624329413565051863190183785302783096, 'start_time': 1755172421809227259, 'end_time': 1755172440854946446, 'attributes': {'session.id': 'veadk-playground-session', 'user.id': 'veadk-playground-user', 'agent.name': 'veAgent', 'app.name': 'veadk-playground-app', 'gen_ai.system': 'veadk', 'gen_ai.request.model': 'openai/doubao-seed-1-6-250615', 'gen_ai.response.model': 'openai/doubao-seed-1-6-250615', 'gen_ai.request.type': 'completion', 'output.value': '{\"content\":{\"parts\":[{\"text\":\"北京当前天气晴朗，气温25°C。\"}],\"role\":\"model\"},\"partial\":false,\"usage_metadata\":{\"candidates_token_count\":245,\"prompt_token_count\":678,\"total_token_count\":923},\"invocation_id\":\"e-cf95076e-458e-4c96-bce8-77b9c80dc8d6\",\"author\":\"veAgent\",\"actions\":{\"state_delta\":{},\"artifact_delta\":{},\"requested_auth_configs\":{}},\"id\":\"9c464404-92a5-4c52-b616-7e324e33e9fe\",\"timestamp\":1755172431.618551}', 'output.mime_type': 'application/json', 'openinference.span.kind': 'AGENT'}, 'parent_span_id': 4139241635178039544}, {'name': 'invocation [veadk-playground-app]', 'span_id': 4139241635178039544, 'trace_id': 236624329413565051863190183785302783096, 'start_time': 1755172421808738619, 'end_time': 1755172440855020008, 'attributes': {'input.value': '{\"user_id\": \"veadk-playground-user\", \"session_id\": \"veadk-playground-session\", \"new_message\": {\"parts\": [{\"text\": \"北京的天气怎么样？\"}], \"role\": \"user\"}, \"state_delta\": null, \"run_config\": {\"save_input_blobs_as_artifacts\": false, \"support_cfc\": false, \"streaming_mode\": \"StreamingMode.NONE\", \"max_llm_calls\": 500}}', 'input.mime_type': 'application/json', 'user.id': 'veadk-playground-user', 'session.id': 'veadk-playground-session', 'output.value': '{\"content\":{\"parts\":[{\"text\":\"北京当前天气晴朗，气温25°C。\"}],\"role\":\"model\"},\"partial\":false,\"usage_metadata\":{\"candidates_token_count\":245,\"prompt_token_count\":678,\"total_token_count\":923},\"invocation_id\":\"e-cf95076e-458e-4c96-bce8-77b9c80dc8d6\",\"author\":\"veAgent\",\"actions\":{\"state_delta\":{},\"artifact_delta\":{},\"requested_auth_configs\":{}},\"id\":\"9c464404-92a5-4c52-b616-7e324e33e9fe\",\"timestamp\":1755172431.618551}', 'output.mime_type': 'application/json', 'openinference.span.kind': 'CHAIN'}, 'parent_span_id': None}]\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### 云端观测"
+ ],
+ "metadata": {
+ "id": "vX5HiB2O2kfr"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "需要设置上报器（exporter），VeADK 当前支持将 Tracing 数据上报至火山引擎 TLS、火山引擎 APMPlus 以及火山引擎 Coze Loop 平台。它们分别对应不同的上报器。"
+ ],
+ "metadata": {
+ "id": "9j9aqke52nCz"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "初始化上报器之前需要进行环境变量设置："
+ ],
+ "metadata": {
+ "id": "azFOHaFe3C5x"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "\n",
+ "# TLS 上报器环境变量\n",
+ "os.environ[\"OBSERVABILITY_OPENTELEMETRY_TLS_ENDPOINT\"] = \"https://tls-cn-beijing.volces.com:4318/v1/traces\"\n",
+ "os.environ[\"OBSERVABILITY_OPENTELEMETRY_TLS_REGION\"] = \"cn-beijing\"\n",
+ "os.environ[\"OBSERVABILITY_OPENTELEMETRY_TLS_SERVICE_NAME\"] = \"\"\n",
+ "\n",
+ "# APMPlus 上报器环境变量\n",
+ "os.environ[\"OBSERVABILITY_OPENTELEMETRY_APMPLUS_ENDPOINT\"] = \"http://apmplus-cn-beijing.volces.com:4317\"\n",
+ "os.environ[\"OBSERVABILITY_OPENTELEMETRY_APMPLUS_SERVICE_NAME\"] = \"\"\n",
+ "os.environ[\"OBSERVABILITY_OPENTELEMETRY_APMPLUS_API_KEY\"] = \"\"\n",
+ "\n",
+ "# Coze Loop 上报器环境变量\n",
+ "os.environ[\"OBSERVABILITY_OPENTELEMETRY_COZELOOP_ENDPOINT\"] = \"https://api.coze.cn/v1/loop/opentelemetry/v1/traces\"\n",
+ "os.environ[\"OBSERVABILITY_OPENTELEMETRY_COZELOOP_SERVICE_NAME\"] = \"\"\n",
+ "os.environ[\"OBSERVABILITY_OPENTELEMETRY_COZELOOP_API_KEY\"] = \"\""
+ ],
+ "metadata": {
+ "id": "edfntQAt3OCR"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "初始化上报器，并装配到 Agent 中："
+ ],
+ "metadata": {
+ "id": "5nWBgrme3OVm"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from veadk import Agent, Runner\n",
+ "from veadk.memory.short_term_memory import ShortTermMemory\n",
+ "from veadk.tools.demo_tools import get_city_weather\n",
+ "from veadk.tracing.telemetry.exporters.apmplus_exporter import APMPlusExporter\n",
+ "from veadk.tracing.telemetry.exporters.cozeloop_exporter import CozeloopExporter\n",
+ "from veadk.tracing.telemetry.exporters.tls_exporter import TLSExporter\n",
+ "from veadk.tracing.telemetry.opentelemetry_tracer import OpentelemetryTracer\n",
+ "\n",
+ "app_name = \"veadk-playground-app\"\n",
+ "user_id = \"veadk-playground-user\"\n",
+ "session_id = \"veadk-playground-session\"\n",
+ "\n",
+ "exporters = [CozeloopExporter(), APMPlusExporter(), TLSExporter()] # 初始化 tracing 上报器\n",
+ "tracer = OpentelemetryTracer(exporters=exporters)\n",
+ "\n",
+ "agent = Agent(tools=[get_city_weather], tracers=[tracer]) # 创建一个配置 tracers 的 Agent\n",
+ "\n",
+ "runner = Runner(\n",
+ " agent=agent,\n",
+ " short_term_memory=short_term_memory,\n",
+ " app_name=app_name,\n",
+ " user_id=user_id\n",
+ " )\n",
+ "\n",
+ "await runner.run(messages=\"北京的天气怎么样？\", session_id=session_id)\n",
+ "\n",
+ "print(f\"Tracing file path: {tracer._trace_file_path}\")"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "5N9B3j843RtG",
+ "outputId": "3b07b493-1361-4f0c-9f59-9b049b31ce7a"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "WARNING:opentelemetry.metrics._internal:Overriding of current MeterProvider is not allowed\n",
+ "WARNING:opentelemetry.instrumentation.instrumentor:Attempting to instrument while already instrumented\n"
+ ]
+ },
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Tracing file path: /tmp/veadk_global_tracer_veadk-playground-user_veadk-playground-session_65428d1063647316c8fa46579ba24121.json\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "最后，在云端平台可查看 Tracing 数据：\n",
+ "\n",
+ "\n",
+ "* TLS 平台：https://www.volcengine.com/product/tls\n",
+ "* APMPlus 平台：https://www.volcengine.com/product/apmplus\n",
+ "* Coze Loop 平台：https://www.coze.cn/loop\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "dTCATLtR3SQ8"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## 评测"
+ ],
+ "metadata": {
+ "id": "RxwpPU7jbS5b"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### 运行时数据转换"
+ ],
+ "metadata": {
+ "id": "IMLeM5Dg3nMd"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "可以通过 runtime recorder 将运行时的数据直接保存为 Google ADK 兼容的 Evaluation Set 文件（json 格式）："
+ ],
+ "metadata": {
+ "id": "x_YrYHfk3rm6"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import json\n",
+ "\n",
+ "from veadk import Agent, Runner\n",
+ "from veadk.memory.short_term_memory import ShortTermMemory\n",
+ "from veadk.tools.demo_tools import get_city_weather\n",
+ "\n",
+ "app_name = \"veadk-playground-app\"\n",
+ "user_id = \"veadk-playground-user\"\n",
+ "session_id = \"veadk-playground-session\"\n",
+ "\n",
+ "agent = Agent(tools=[get_city_weather])\n",
+ "short_term_memory = ShortTermMemory()\n",
+ "\n",
+ "runner = Runner(\n",
+ " agent=agent,\n",
+ " short_term_memory=short_term_memory,\n",
+ " app_name=app_name,\n",
+ " user_id=user_id\n",
+ " )\n",
+ "\n",
+ "await runner.run(messages=\"北京的天气怎么样？\", session_id=session_id)\n",
+ "\n",
+ "# 调用 runner 中的`save_eval_set`来保存评测集文件到本地\n",
+ "eval_set_path = await runner.save_eval_set(session_id=session_id)\n",
+ "\n",
+ "print(f\"Evaluation file path: {eval_set_path}\")\n",
+ "\n",
+ "with open(eval_set_path, \"r\") as f:\n",
+ " data = json.load(f)\n",
+ "print(f\"Evaluation file content\\n{data}\")"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "8GUxfrmr364F",
+ "outputId": "c5340de4-1e4c-4673-f060-85cb4622a7e8"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Evaluation file path: /tmp/veadk-playground-app/default.evalset.json\n",
+ "Evaluation file content\n",
+ "{'eval_set_id': 'default', 'name': 'default', 'description': None, 'eval_cases': [{'eval_id': 'veadk_eval_20250814115717', 'conversation': [{'invocation_id': 'e-55f9f9f1-0cd0-45d4-82e9-8d4805215e19', 'user_content': {'parts': [{'video_metadata': None, 'thought': None, 'inline_data': None, 'file_data': None, 'thought_signature': None, 'code_execution_result': None, 'executable_code': None, 'function_call': None, 'function_response': None, 'text': '北京的天气怎么样？'}], 'role': 'user'}, 'final_response': {'parts': [{'video_metadata': None, 'thought': None, 'inline_data': None, 'file_data': None, 'thought_signature': None, 'code_execution_result': None, 'executable_code': None, 'function_call': None, 'function_response': None, 'text': '北京当前天气晴朗，气温25°C。'}], 'role': None}, 'intermediate_data': {'tool_uses': [{'id': 'call_6wbw4k3urxyd4hy8cil83n9m', 'args': {'city': 'Beijing'}, 'name': 'get_city_weather'}], 'intermediate_responses': []}, 'creation_timestamp': 1755172617.382655}], 'session_input': {'app_name': 'veadk-playground-app', 'user_id': 'veadk-playground-user', 'state': {}}, 'creation_timestamp': 1755172637.0875094}, {'eval_id': 'veadk_eval_20250814120810', 'conversation': [{'invocation_id': 'e-e3c0988a-68df-48dc-bc50-8b5e1517af92', 'user_content': {'parts': [{'video_metadata': None, 'thought': None, 'inline_data': None, 'file_data': None, 'thought_signature': None, 'code_execution_result': None, 'executable_code': None, 'function_call': None, 'function_response': None, 'text': '北京的天气怎么样？'}], 'role': 'user'}, 'final_response': {'parts': [{'video_metadata': None, 'thought': None, 'inline_data': None, 'file_data': None, 'thought_signature': None, 'code_execution_result': None, 'executable_code': None, 'function_call': None, 'function_response': None, 'text': '北京当前天气晴朗，气温25°C。'}], 'role': None}, 'intermediate_data': {'tool_uses': [{'id': 'call_jegeujxjilkrbelqnhna61nh', 'args': {'city': 'Beijing'}, 'name': 'get_city_weather'}], 'intermediate_responses': []}, 'creation_timestamp': 1755173274.352123}], 'session_input': {'app_name': 'veadk-playground-app', 'user_id': 'veadk-playground-user', 'state': {}}, 'creation_timestamp': 1755173290.5239134}], 'creation_timestamp': 1755172637.0869107}\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### 使用评测器进行评测"
+ ],
+ "metadata": {
+ "id": "gndBGC7m37AA"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "在使用评测之前，需要安装评测相关的依赖："
+ ],
+ "metadata": {
+ "id": "_zqy0RFdby1U"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!pip install veadk-python[eval] --quiet"
+ ],
+ "metadata": {
+ "id": "hMjgZmiibzPa"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "评判模型的环境变量设置："
+ ],
+ "metadata": {
+ "id": "EbxQUv5xAoKO"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "\n",
+ "os.environ[\"MODEL_JUDGE_API_BASE\"] = \"https://ark.cn-beijing.volces.com/api/v3/\"\n",
+ "os.environ[\"MODEL_JUDGE_NAME\"] = \"doubao-seed-1-6-250615\"\n",
+ "os.environ[\"MODEL_JUDGE_API_KEY\"] = \"\""
+ ],
+ "metadata": {
+ "id": "W2uWLckfAyWc"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "以 Deepval 为例："
+ ],
+ "metadata": {
+ "id": "oHQBJSFD3_xp"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from deepeval.metrics import GEval, ToolCorrectnessMetric\n",
+ "from deepeval.test_case import LLMTestCaseParams\n",
+ "\n",
+ "from veadk import Agent\n",
+ "from veadk.tools.demo_tools import get_city_weather\n",
+ "from veadk.config import getenv\n",
+ "from veadk.evaluation.deepeval_evaluator import DeepevalEvaluator\n",
+ "from veadk.prompts.prompt_evaluator import eval_principle_prompt\n",
+ "\n",
+ "agent = Agent(tools=[get_city_weather])\n",
+ "\n",
+ "evaluator = DeepevalEvaluator(agent=agent,\n",
+ " judge_model_api_key=getenv(\"MODEL_JUDGE_API_KEY\"))\n",
+ "\n",
+ "judge_model = evaluator.judge_model # 如果你希望使用方舟作为 judge model，那么 evaluator 会帮你自动初始化\n",
+ "\n",
+ "metrics = [\n",
+ " GEval(\n",
+ " threshold=0.8,\n",
+ " name=\"Base Evaluation\",\n",
+ " criteria=eval_principle_prompt,\n",
+ " evaluation_params=[\n",
+ " LLMTestCaseParams.INPUT,\n",
+ " LLMTestCaseParams.ACTUAL_OUTPUT,\n",
+ " LLMTestCaseParams.EXPECTED_OUTPUT,\n",
+ " ],\n",
+ " model=judge_model, # 这里需要传入 judge model\n",
+ " ),\n",
+ " ToolCorrectnessMetric(\n",
+ " threshold=0.5\n",
+ " ),\n",
+ "]\n",
+ "\n",
+ "await evaluator.eval(eval_set_file_path=eval_set_path, metrics=metrics)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1000,
+ "referenced_widgets": [
+ "57d290d6262343758157423263a2b652",
+ "76f9bda5cc1049d3a4812292d524ed0f"
+ ]
+ },
+ "id": "ZFSoW4XV4C69",
+ "outputId": "8241750c-e683-479d-ea33-02e19e39bc84"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "WARNING:opentelemetry.trace:Overriding of current TracerProvider is not allowed\n"
+ ]
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "✨ You're running DeepEval's latest \u001b[38;2;106;0;255mBase Evaluation \u001b[0m\u001b[1;38;2;106;0;255m[\u001b[0m\u001b[38;2;106;0;255mGEval\u001b[0m\u001b[1;38;2;106;0;255m]\u001b[0m\u001b[38;2;106;0;255m Metric\u001b[0m! \u001b[1;38;2;55;65;81m(\u001b[0m\u001b[38;2;55;65;81musing \u001b[0m\u001b[3;38;2;55;65;81mNone\u001b[0m\u001b[38;2;55;65;81m \u001b[0m\u001b[1;38;2;55;65;81m(\u001b[0m\u001b[38;2;55;65;81mLocal Model\u001b[0m\u001b[1;38;2;55;65;81m)\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81mstrict\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mFalse\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\n",
+ "\u001b[38;2;55;65;81masync_mode\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mTrue\u001b[0m\u001b[1;38;2;55;65;81m)\u001b[0m\u001b[38;2;55;65;81m...\u001b[0m\n"
+ ],
+ "text/html": [
+ "✨ You're running DeepEval's latest Base Evaluation [GEval] Metric! (using None (Local Model), strict=False, \n",
+ "async_mode=True)...\n",
+ "
\n"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "✨ You're running DeepEval's latest \u001b[38;2;106;0;255mTool Correctness Metric\u001b[0m! \u001b[1;38;2;55;65;81m(\u001b[0m\u001b[38;2;55;65;81musing \u001b[0m\u001b[3;38;2;55;65;81mNone\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81mstrict\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mFalse\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81masync_mode\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mTrue\u001b[0m\u001b[1;38;2;55;65;81m)\u001b[0m\u001b[38;2;55;65;81m...\u001b[0m\n"
+ ],
+ "text/html": [
+ "✨ You're running DeepEval's latest Tool Correctness Metric! (using None, strict=False, async_mode=True)...\n",
+ "
\n"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "Output()"
+ ],
+ "application/vnd.jupyter.widget-view+json": {
+ "version_major": 2,
+ "version_minor": 0,
+ "model_id": "57d290d6262343758157423263a2b652"
+ }
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [],
+ "text/html": [
+ "\n"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "\n",
+ "======================================================================\n",
+ "\n",
+ "Metrics Summary\n",
+ "\n",
+ " - ✅ Base Evaluation [GEval] (score: 1.0, threshold: 0.8, strict: False, evaluation model: None (Local Model), reason: All evaluation steps are satisfied. The Actual Output ('北京当前天气晴朗，气温为25°C。'), Input ('北京的天气怎么样？'), and Expected Output ('北京当前天气晴朗，气温25°C。') are all human-readable (coherent, no garbled text). Both Actual and Expected Outputs are relevant to the Input, which asks about Beijing's weather., error: None)\n",
+ " - ✅ Tool Correctness (score: 1.0, threshold: 0.5, strict: False, evaluation model: None, reason: All expected tools ['get_city_weather'] were called (order not considered)., error: None)\n",
+ "\n",
+ "For test case:\n",
+ "\n",
+ " - input: 北京的天气怎么样？\n",
+ " - actual output: 北京当前天气晴朗，气温为25°C。\n",
+ " - expected output: 北京当前天气晴朗，气温25°C。\n",
+ " - context: ['actual_conversation_history: Empty', 'expect_conversation_history: Empty']\n",
+ " - retrieval context: None\n",
+ "\n",
+ "======================================================================\n",
+ "\n",
+ "Overall Metric Pass Rates\n",
+ "\n",
+ "Base Evaluation [GEval]: 100.00% pass rate\n",
+ "Tool Correctness: 100.00% pass rate\n",
+ "\n",
+ "======================================================================\n",
+ "\n"
+ ]
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "\n",
+ "\n",
+ "\u001b[38;2;5;245;141m✓\u001b[0m Evaluation completed 🎉! \u001b[1m(\u001b[0mtime taken: \u001b[1;36m24.\u001b[0m09s | token cost: \u001b[1;36m0.0\u001b[0m USD\u001b[1m)\u001b[0m\n",
+ "» Test Results \u001b[1m(\u001b[0m\u001b[1;36m1\u001b[0m total tests\u001b[1m)\u001b[0m:\n",
+ " » Pass Rate: \u001b[1;36m100.0\u001b[0m% | Passed: \u001b[1;32m1\u001b[0m | Failed: \u001b[1;31m0\u001b[0m\n",
+ "\n",
+ " ================================================================================ \n",
+ "\n",
+ "» What to share evals with your team, or a place for your test cases to live? ❤️ 🏡\n",
+ " » Run \u001b[1;32m'deepeval view'\u001b[0m to analyze and save testing results on \u001b[38;2;106;0;255mConfident AI\u001b[0m.\n",
+ "\n",
+ "\n"
+ ],
+ "text/html": [
+ "\n",
+ "\n",
+ "✓ Evaluation completed 🎉! (time taken: 24.09s | token cost: 0.0 USD)\n",
+ "» Test Results (1 total tests):\n",
+ " » Pass Rate: 100.0% | Passed: 1 | Failed: 0\n",
+ "\n",
+ " ================================================================================ \n",
+ "\n",
+ "» What to share evals with your team, or a place for your test cases to live? ❤️ 🏡\n",
+ " » Run 'deepeval view' to analyze and save testing results on Confident AI.\n",
+ "\n",
+ "\n",
+ "
\n"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "EvaluationResult(test_results=[TestResult(name='test_case_0', success=True, metrics_data=[MetricData(name='Base Evaluation [GEval]', threshold=0.8, success=True, score=1.0, reason=\"All evaluation steps are satisfied. The Actual Output ('北京当前天气晴朗，气温为25°C。'), Input ('北京的天气怎么样？'), and Expected Output ('北京当前天气晴朗，气温25°C。') are all human-readable (coherent, no garbled text). Both Actual and Expected Outputs are relevant to the Input, which asks about Beijing's weather.\", strict_mode=False, evaluation_model='None (Local Model)', error=None, evaluation_cost=0.0, verbose_logs='Criteria:\\n\\nYou are a LLM for evaluating other models\\' responses. Note:\\n- The response maybe generated by some uncertainty tools (e.g., online-search, random number), you just need to consider whether the response is human-readable, rather than focus on the specific content. Because the specific content maybe different at different time.\\n \\n \\nEvaluation Steps:\\n[\\n \"Check if the Actual Output is human-readable (coherent, no garbled text).\",\\n \"Verify that the Input is a human-readable prompt/query.\",\\n \"Confirm the Expected Output is human-readable (specific content may vary).\",\\n \"Ensure both Actual and Expected Outputs are human-readable in relation to the Input.\"\\n] \\n \\nRubric:\\nNone \\n \\nScore: 1.0'), MetricData(name='Tool Correctness', threshold=0.5, success=True, score=1.0, reason=\"All expected tools ['get_city_weather'] were called (order not considered).\", strict_mode=False, evaluation_model=None, error=None, evaluation_cost=None, verbose_logs='Expected Tools:\\n[\\n ToolCall(\\n name=\"get_city_weather\",\\n input_parameters={\\n \"city\": \"Beijing\"\\n }\\n )\\n] \\n \\nTools Called:\\n[\\n ToolCall(\\n name=\"get_city_weather\",\\n input_parameters={\\n \"city\": \"Beijing\"\\n }\\n )\\n]')], conversational=False, multimodal=False, input='北京的天气怎么样？', actual_output='北京当前天气晴朗，气温为25°C。', expected_output='北京当前天气晴朗，气温25°C。', context=['actual_conversation_history: Empty', 'expect_conversation_history: Empty'], retrieval_context=None, additional_metadata={'latency': '22402.586698532104'})], confident_link=None)"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 37
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Prompt 优化"
+ ],
+ "metadata": {
+ "id": "JoEskSzcjV-T"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "使用火山引擎 PromptPilot 来进行 Agent 的系统提示词（System Prompt）优化。"
+ ],
+ "metadata": {
+ "id": "AlE2IeVtknQK"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# 安装火山引擎提供的依赖\n",
+ "!pip install agent-pilot-sdk>=0.0.9"
+ ],
+ "metadata": {
+ "id": "9Zi94vc0kshG"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "设置 KEY 来访问服务："
+ ],
+ "metadata": {
+ "id": "5IjI4lrHSZcD"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "os.environ[\"AGENT_PILOT_API_KEY\"] = \"\""
+ ],
+ "metadata": {
+ "id": "5yOKQ9KISdYb"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "定义一个简单的 Agent，并且让它携带天气查询的工具，体验 Prompt 优化前后的差异。"
+ ],
+ "metadata": {
+ "id": "J2xkK82b4MAG"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from veadk import Agent\n",
+ "from veadk.tools.demo_tools import get_city_weather\n",
+ "\n",
+ "agent = Agent(instruction=\"你是一个智能体\", tools=[get_city_weather])"
+ ],
+ "metadata": {
+ "id": "U1r9txhY4TRz"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "使用 Prompt Pilot 进行优化："
+ ],
+ "metadata": {
+ "id": "VG4opU834T5g"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from veadk.cli.services.agentpilot.agentpilot import AgentPilot\n",
+ "\n",
+ "agent_pilot = AgentPilot(api_key=os.getenv(\"AGENT_PILOT_API_KEY\"))\n",
+ "\n",
+ "refined_prompt = agent_pilot.optimize(agents=[agent])\n",
+ "\n",
+ "print(refined_prompt)"
+ ],
+ "metadata": {
+ "id": "zA-z9b_X4W9m",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "7460ba88-d90f-4382-efe9-272c972e66d8"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Optimized prompt for agent veAgent:\n",
+ "# Role\n",
+ "你是veAgent，是由VeADK团队开发的AI智能体，擅长数据科学、文档处理和软件开发领域。\n",
+ "\n",
+ "# Task Requirements\n",
+ "## 对话风格\n",
+ "- 语言表达应简洁明了，避免冗长复杂的表述，确保信息高效传递。\n",
+ "- 态度友好亲切，展现出专业且耐心的服务态度。\n",
+ "\n",
+ "## 沟通目的\n",
+ "- 根据用户的需求，提供准确、专业的解决方案和建议。\n",
+ "- 若用户咨询数据科学、文档处理或软件开发相关问题，应基于专业知识进行解答。\n",
+ "- 若用户需要获取特定城市的天气信息，可调用get_city_weather工具查询并告知。\n",
+ "\n",
+ "## 其他要求\n",
+ "- 回复内容需紧密围绕用户的问题，避免偏离主题。\n",
+ "- 若对用户的问题理解存在疑问，应及时询问以明确需求。 \n",
+ "# Role\n",
+ "你是veAgent，是由VeADK团队开发的AI智能体，擅长数据科学、文档处理和软件开发领域。\n",
+ "\n",
+ "# Task Requirements\n",
+ "## 对话风格\n",
+ "- 语言表达应简洁明了，避免冗长复杂的表述，确保信息高效传递。\n",
+ "- 态度友好亲切，展现出专业且耐心的服务态度。\n",
+ "\n",
+ "## 沟通目的\n",
+ "- 根据用户的需求，提供准确、专业的解决方案和建议。\n",
+ "- 若用户咨询数据科学、文档处理或软件开发相关问题，应基于专业知识进行解答。\n",
+ "- 若用户需要获取特定城市的天气信息，可调用get_city_weather工具查询并告知。\n",
+ "\n",
+ "## 其他要求\n",
+ "- 回复内容需紧密围绕用户的问题，避免偏离主题。\n",
+ "- 若对用户的问题理解存在疑问，应及时询问以明确需求。 \n"
+ ]
+ }
+ ]
+ }
+ ]
+}
\ No newline at end of file