Tang Moying

GitLab Predefined Variables in Practice (Part 2): The Pipeline "Orchestration Signals" — Trigger Source, Parent-Child Pipelines, Runner and Environment

Sun, 21 Jun 2026 00:00:00 +0000

The previous post talked about the “identification” variables — how a pipeline knows “who it is, where it is, and which commit it came from”. This post is about the “orchestration” variables, which answer a different kind of question: who triggered me, which larger pipeline do I belong to, which machine am I running on, and which environment did I reach.

If the identification variables brought configuration to life (one set of configuration adapting to many branches and many projects), then the orchestration variables make pipelines truly able to cooperate — multi-project chaining, parent-child splitting, and executing differently by trigger method all depend on them.

1. Who Triggered the Pipeline: CI_PIPELINE_SOURCE

This is the one to figure out first among the orchestration variables. $CI_PIPELINE_SOURCE tells you how this pipeline came about; its possible values include:

Value	Meaning
`push`	Triggered by a code push
`merge_request_event`	Triggered by an MR event (see the previous post)
`schedule`	Triggered by a scheduled job
`web`	Manually clicking “Run pipeline” in the GitLab web UI
`api`	Triggered through the API
`trigger`	Triggered through a `trigger` token (common in multi-project pipelines)
`pipeline`	Triggered by another pipeline (parent-child / multi-project pipelines)
`chat`	Triggered through ChatOps (for example the Slack `/gitlab` command)
`external`	Triggered by an external CI integration

Its value lies in this: the same configuration behaves completely differently under different trigger methods. The classic pattern is “run the cheap ones constantly, the expensive ones on demand”:

quick_lint:
  script: ./lint.sh
  rules:
    - if: $CI_PIPELINE_SOURCE == "push"          # run the lightweight check on every push
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

heavy_e2e:
  script: ./e2e.sh
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"      # only the scheduled job runs the heavy e2e
    - if: $CI_PIPELINE_SOURCE == "web"           # and it can also run when you click to run it manually

This way day-to-day pushes return results in seconds, and the expensive end-to-end tests only run on a nightly schedule, so Runner resources aren’t wasted.

2. Scheduled Jobs Need Names Too: CI_PIPELINE_SCHEDULE_DESCRIPTION

Knowing that it was triggered by schedule isn’t enough. You may have configured several scheduled jobs: one that “runs the full regression every day at dawn” and one that “runs smoke tests every hour”. With the same .gitlab-ci.yml, how do you tell which scheduled job this run is?

The answer is $CI_PIPELINE_SCHEDULE_DESCRIPTION — it takes the Description you filled in for that scheduled job under CI/CD → Schedules. Use it to branch:

nightly_regression:
  script: ./regression.sh
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule" && $CI_PIPELINE_SCHEDULE_DESCRIPTION == "nightly-full"

Note that it compares the description text, so scheduled jobs need to be named consistently and stably. Don’t call it nightly today and change it to 每晚全量 tomorrow, or the rule will stop matching.

3. Passing Context Between Parent and Child Pipelines: CI_PARENT_PIPELINE_ID and CI_UPSTREAM_*

As .gitlab-ci.yml grows longer, you’ll want to split build, test, and deploy into “child pipelines”, pulled in with trigger: include. Or project A finishes building and triggers a deployment in project B (a multi-project pipeline). At that point, how do parent and child know who each other are?

Variable	Meaning
`$CI_PIPELINE_ID`	The unique ID of the current pipeline
`$CI_PARENT_PIPELINE_ID`	The ID of the parent pipeline that triggered me (has a value in child pipelines)
`$CI_UPSTREAM_PIPELINE_ID`	The upstream pipeline ID (in multi-project scenarios, pointing at the triggering side)
`$CI_UPSTREAM_PROJECT_PATH`	The upstream project path
`$CI_UPSTREAM_NAMESPACE`	The upstream namespace
`$CI_UPSTREAM_REF`	The upstream ref at the time of triggering

A practical scenario: a child pipeline finishes its tests and wants to write the status back or notify the parent project. With $CI_UPSTREAM_PROJECT_PATH you can pin down “who set me running”:

# in the child pipeline
notify_parent:
  script:
    - ./notify.sh "子流水线 $CI_PIPELINE_ID 完成，父流水线为 $CI_PARENT_PIPELINE_ID"

Another gotcha: in a child pipeline, “identification” variables such as $CI_COMMIT_BRANCH are inherited from the parent pipeline, not recomputed. So the branch-based branching rules from the previous post work just as correctly in parent-child pipelines, without needing to pass parameters again.

4. Which Machine Am I Running On: CI_RUNNER_* and GITLAB_*

GitLab Runner is multi-platform: Linux, Windows, macOS, even specific GPU machines. Your scripts often need to sense which kind of Runner they’re running on.

Variable	Meaning
`$CI_RUNNER_ID`	The Runner’s numeric ID
`$CI_RUNNER_DESCRIPTION`	The Runner’s description (what you filled in when registering it)
`$CI_RUNNER_TAGS`	The Runner’s tag list, comma-separated, e.g. `docker,linux`
`$CI_SERVER_HOST`	The GitLab instance’s hostname
`$CI_SERVER_VERSION`	The GitLab version number
`$GITLAB_USER_NAME` / `$GITLAB_USER_EMAIL` / `$GITLAB_USER_LOGIN` / `$GITLAB_USER_ID`	The identity of the user who triggered this pipeline

The most classic use case is platform-specific builds. For example, an iOS package can only be built on a macOS Runner:

build_ios:
  script: ./build-ios.sh
  tags:
    - macos          # only pick Runners tagged macos

Inside a script, if you want to branch on Runner capabilities, you can parse $CI_RUNNER_TAGS:

if echo "$CI_RUNNER_TAGS" | grep -q "gpu"; then
  export USE_GPU=1
fi

$GITLAB_USER_*, by contrast, is often used for audit and notification — when something goes wrong, it tells you clearly who clicked to release and who triggered this pipeline, instead of leaving you staring at an anonymous CI bot.

5. Which Environment Did It Reach: CI_ENVIRONMENT_* and Deploy Freezes

Jobs that carry an environment get extra environment-related variables injected by GitLab:

Variable	Meaning
`$CI_ENVIRONMENT_NAME`	The current environment name (e.g. `production`, `review/my-feature`)
`$CI_ENVIRONMENT_SLUG`	The slug form of the environment name (DNS-safe)
`$CI_ENVIRONMENT_URL`	The address you declared in `environment:url`

Together with the review environment pattern from the first post, this set of variables makes “dynamic environments” genuinely usable: every MR automatically gets its own review environment, cleaned up automatically when the MR is closed.

Another practical one is the deploy freeze. GitLab lets you mark out “no deploy” windows on the calendar (during a major sale, say). During a freeze, related variables are set:

$CI_DEPLOY_FREEZE_STARTED / $CI_DEPLOY_FREEZE_* and the like, injected by instance/project configuration

You can use it to “block” production deployments and avoid mistakes inside a freeze window:

deploy_prod:
  script: ./deploy.sh prod
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  environment:
    name: production

Although the deploy freeze itself is intercepted by GitLab at the scheduling layer, knowing these variables lets you write finer-grained, custom safeguards inside your scripts.

6. Parallel Splitting and Job Metadata: CI_JOB_* and CI_NODE_*

The last group is job-level and mainly useful in parallel or parallel:matrix scenarios:

Variable	Meaning
`$CI_JOB_ID`	The unique ID of the current job
`$CI_JOB_NAME`	The job name
`$CI_JOB_STAGE`	The stage it belongs to
`$CI_JOB_URL`	The job’s web page URL
`$CI_NODE_INDEX`	The index within the parallel split (starting from 0)
`$CI_NODE_TOTAL`	The total number of parallel slices

A typical scenario: split a large test suite into 4 slices running in parallel, each running only its own portion:

test_split:
  parallel: 4
  script:
    - ./run-tests.sh --slice $CI_NODE_INDEX/$CI_NODE_TOTAL

Most test frameworks (pytest, jest, and so on) support slicing by “total + index”; just feed them $CI_NODE_INDEX and $CI_NODE_TOTAL and you don’t need to compute anything yourself.

Summary

The core problem the orchestration variables solve is “how does a pipeline know what collaborative relationship it’s part of”:

Trigger method: $CI_PIPELINE_SOURCE — branch behavior by push / MR / schedule / web
Scheduled identity: $CI_PIPELINE_SCHEDULE_DESCRIPTION — tell the intent of several scheduled jobs apart
Parent-child cooperation: $CI_PARENT_PIPELINE_ID, $CI_UPSTREAM_* — pass context between pipelines and write status back
Execution machine: $CI_RUNNER_*, $GITLAB_USER_* — platform adaptation and audit tracing
Environment and deployment: $CI_ENVIRONMENT_* — dynamic environments, deploy freezes
Parallel splitting: $CI_NODE_INDEX / $CI_NODE_TOTAL — test sharding

Together, the two posts complete the two main threads of GitLab predefined variables: identification and orchestration. Remember the principle that runs through it all: if a predefined variable can give you a value, don’t hard-code it — hard-coding will come back to bite you in the second repository, on the Nth trigger, on the Mth Runner.

One last reminder of an easily overlooked point: predefined variables expand at different times in different places. rules: and variables: expand during YAML parsing, whereas variables inside script: expand during shell execution. So comparing $CI_COMMIT_BRANCH inside rules is safe, but if you want to build a variable name dynamically inside script and then reference it, you have to mind the shell’s expansion order — that set of pitfalls deserves a post of its own, and we’ll get into it another time.

A small step forward every day is a new height!

GitLab Predefined Variables in Practice (Part 1): The Pipeline's "Identity" — Branch, Tag, Commit, and MR

Sat, 20 Jun 2026 00:00:00 +0000

Every time a pipeline runs, GitLab automatically injects a pile of “built-in variables” (officially called Predefined Variables). You don’t need to configure them; they work out of the box, and they’re the only window through which a pipeline can perceive “what context it’s in right now.”

Many people writing a .gitlab-ci.yml have only ever used one $CI_COMMIT_BRANCH, and then rely on a pile of hard-coded ifs to tell environments apart. But this set of predefined variables can actually support some very intricate scenarios: one and the same configuration can automatically adapt to the branch, automatically tag a version, automatically recognize whether it’s an MR, and automatically know which project is running.

If you’re not yet familiar with GitLab’s variable hierarchy (how Project / Group / predefined variables relate to one another), I’d suggest reading that 2023 article, “Gitlab CI Variables and Environments,” first. This article won’t restate the hierarchy; it focuses solely on how to use predefined variables to solve real scenarios.

This article focuses on the “identity” variables — the ones that answer “who am I, where am I, where did I come from, and which commit did I come from.” The next article will cover the “orchestration” variables.

1. Deciding the Deployment Target by Branch: CI_COMMIT_BRANCH and Its Friends

The most classic scenario: the develop branch deploys to the test environment, the main branch deploys to production.

deploy_staging:
  stage: deploy
  script: ./deploy.sh staging
  rules:
    - if: $CI_COMMIT_BRANCH == "develop"

deploy_prod:
  stage: deploy
  script: ./deploy.sh prod
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

There are two easily confused variables here; let’s get them straight first:

$CI_COMMIT_BRANCH: the current branch name. Note that it is empty when a pipeline is triggered by a Tag (because a Tag doesn’t belong to any branch).
$CI_COMMIT_REF_NAME: the name of the current ref — a branch name when it’s a branch, a Tag name when it’s a Tag. It’s a superset of $CI_COMMIT_BRANCH.
$CI_COMMIT_REF_SLUG: converts the ref name into a “DNS-safe” form — for example feature/login becomes feature-login, and release/v1.2 becomes release-v1-2. When you want to use a ref name to build a host name, a namespace, or a directory name, use this rather than the ref name itself.

A common mistake: using $CI_COMMIT_BRANCH to build a Kubernetes namespace in a review environment:

review:
  environment:
    name: review/$CI_COMMIT_BRANCH   # the branch name feature/login contains a /, which K8s doesn't allow

The correct form uses the slug:

review:
  environment:
    name: review/$CI_COMMIT_REF_SLUG   # feature-login, safe
    url: https://$CI_COMMIT_REF_SLUG.review.example.com

2. Tagging Artifacts as “Immutable”: CI_COMMIT_SHA and CI_COMMIT_TAG

For images and artifact repositories, the thing to dread most is “overwriting under the same name.” The ideal practice is to tag every build with an immutable tag — the commit SHA is a naturally immutable identifier.

build:
  stage: build
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

$CI_COMMIT_SHA: the full 40-character commit SHA.
$CI_COMMIT_SHORT_SHA: the first 8 characters as a short SHA, good enough and easier to read — just use it for everyday scenarios.

When you come to a “release” scenario, you want the tag to be a semantic version number rather than a string of SHA. Distinguishing “an ordinary commit” from “a pushed Tag” comes down to $CI_COMMIT_TAG:

release:
  stage: release
  script:
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
  rules:
    - if: $CI_COMMIT_TAG   # only runs when a Tag was pushed

The key point: when a Tag is pushed, $CI_COMMIT_BRANCH is empty, but $CI_COMMIT_TAG has a value. $CI_COMMIT_REF_NAME has a value in both scenarios (the branch name and the Tag name respectively). So if your logic needs to cover both “a branch push” and “a pushed Tag,” prefer $CI_COMMIT_REF_NAME.

3. Distinguishing “an Ordinary Push” from “a Merge Request”: CI_PIPELINE_SOURCE and the MR Variable Family

This is where people trip up most easily. By default, pushing to a branch and opening an MR against that same branch may each run a pipeline (depending on whether you have “Merge Request Pipelines” enabled). Your script needs to be able to tell “am I running inside an MR right now, or inside an ordinary push.”

The key to telling them apart is $CI_PIPELINE_SOURCE. When the pipeline comes from an MR, its value is merge_request_event:

run_tests:
  stage: test
  script: ./run-tests.sh
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"   # only run the heavy tests inside an MR

The MR scenario also has a whole set of dedicated variables, whose value is that they let you get information about the target branch “before the merge” and validate ahead of time:

Variable	Description
`$CI_MERGE_REQUEST_IID`	The MR’s number within the repository (not a global ID)
`$CI_MERGE_REQUEST_TITLE`	The MR’s title
`$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME`	The source branch (the one you’re merging in)
`$CI_MERGE_REQUEST_TARGET_BRANCH_NAME`	The target branch (the one you’re merging into)
`$CI_MERGE_REQUEST_APPROVED`	Whether it has been approved (requires a license)

A practical scenario: only allow merges into main, and the source branch must have passed the checks.

guard:
  stage: test
  script: echo "检查目标分支"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"

Another common need: on an ordinary push run only a lightweight lint, and run the full tests only inside an MR, so you don’t spend half an hour on every push and waste Runner resources. This is the most typical value of using $CI_PIPELINE_SOURCE to route the work.

4. Digging Information Out of Commit Messages: the CI_COMMIT_MESSAGE Family

Commit messages aren’t just for people to read; a pipeline can read them too. GitLab splits them up in fine detail:

Variable	Description
`$CI_COMMIT_MESSAGE`	The full commit message (including the body)
`$CI_COMMIT_TITLE`	The first line (the subject)
`$CI_COMMIT_DESCRIPTION`	The body excluding the first line
`$CI_COMMIT_AUTHOR_NAME`	The author’s name
`$CI_COMMIT_AUTHOR_EMAIL`	The author’s email
`$CI_COMMIT_TIMESTAMP`	The commit timestamp (ISO 8601)

Practical scenario 1: automatically skipping CI. Although these days it’s preferable to use GitLab’s [skip ci] convention (write [skip ci] directly in the commit message and GitLab automatically won’t run the pipeline), you can also check for it manually in a script if you want:

script:
  - if echo "$CI_COMMIT_MESSAGE" | grep -q "\[skip ci\]"; then echo "跳过"; exit 0; fi

Practical scenario 2: using a commit prefix to trigger different behavior. For example, agree that commits starting with chore(release): automatically cut a release:

auto_release:
  script: ./release.sh
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_COMMIT_TITLE =~ /^chore\(release\)/

Practical scenario 3: automatically generating a changelog. Take $CI_COMMIT_TITLE, string it into the release notes, and combine it with $CI_COMMIT_AUTHOR_NAME to note who committed it — far less hassle than writing it by hand.

5. Which Project Am I Running In: the CI_PROJECT_* Family

When you use a single .gitlab-ci.yml as a template and reuse it across a dozen repositories under a Group, a hard-coded project name falls apart immediately. That’s when you rely on the project-level predefined variables:

Variable	Description
`$CI_PROJECT_PATH`	The full path, e.g. `group/subgroup/myproject`
`$CI_PROJECT_NAME`	The project name (the last segment of the path)
`$CI_PROJECT_NAMESPACE`	The namespace it belongs to (a group or a user)
`$CI_PROJECT_DIR`	The absolute path where the code is cloned on the Runner
`$CI_PROJECT_URL`	The project’s web URL on GitLab
`$CI_PROJECT_ID`	The project’s numeric ID

The most useful thing is to include the project context in notification scripts; otherwise the group chat receives a pile of “build failed” and nobody knows which project it was:

notify:
  script:
    - ./notify.sh "项目 $CI_PROJECT_PATH 的流水线 $CI_PIPELINE_ID 失败了：$CI_PROJECT_URL/-/pipelines/$CI_PIPELINE_ID"

And $CI_PROJECT_DIR — once you’ve cd-ed back and forth in a script and want to get back to the code root, using the absolute path $CI_PROJECT_DIR is far more reliable than a relative path, especially inside after_script.

Summary

This group of variables solves the core problem of “how does a pipeline know what context it’s in”:

Branch / ref / slug: $CI_COMMIT_BRANCH, $CI_COMMIT_REF_NAME, $CI_COMMIT_REF_SLUG — deciding “where to deploy and what it’s called”
Commit / Tag: $CI_COMMIT_SHA, $CI_COMMIT_SHORT_SHA, $CI_COMMIT_TAG — giving artifacts an immutable, readable tag
MR context: $CI_PIPELINE_SOURCE == "merge_request_event" plus $CI_MERGE_REQUEST_* — telling MR apart from push, and validating ahead of the merge
Commit message: the $CI_COMMIT_MESSAGE family — driving behavior from conventions and generating notes automatically
Project identity: $CI_PROJECT_* — letting one configuration be reused across many repositories

Remember one principle: whatever a predefined variable can give you, don’t hard-code. Hard-coded project names, branch names, and paths will all turn on you the moment you copy the configuration into a second repository.

The next article covers the “orchestration” variables: who triggered my pipeline, how context is passed between parent and child pipelines, and those Runner and environment variables.

A small step forward every day is a new height!

MCP vs. CLI: The Two Paths for Agents to Call Tools, and Who Is Eroding Whom?

Sat, 13 Jun 2026 00:00:00 +0000

When an Agent calls an external tool, there are two broad paths: MCP and the CLI. MCP (Model Context Protocol) is a standardized protocol Anthropic introduced at the end of 2024, with the goal of giving Agents one unified way to connect to every kind of third-party service. The CLI is just the command line — kubectl get pods, git diff, docker ps — where the Agent runs the command directly and takes the output. Many people instinctively feel that MCP is the more “advanced” approach: standardized, type-safe, dynamically discovered. It sounds so sophisticated that it ought to gradually replace crude CLI calls. But the reality is exactly the opposite — the CLI is eroding MCP’s share in reverse. This article looks at why.

1. MCP’s Ideal: One Standard to Connect Everything

First, let’s understand what problem MCP set out to solve.

Before MCP existed, connecting an Agent to a new service meant doing four things: write a Tool definition (description + parameter schema), write the execution logic, write the error handling, and register the Tool with the Agent. You did all of it again for every service. Ten services meant ten pieces of glue code.

MCP’s idea was this: define a standard protocol; the service provider writes an MCP Server, the Agent framework writes an MCP Client, and the two communicate over the standard protocol. As a developer you no longer write glue code — the Agent dynamically discovers the capabilities a Server offers through the MCP protocol and calls them directly.

For example: GitHub has an MCP Server that exposes tools like search_repositories, create_issue and get_pr. Once an Agent is connected to that MCP Server, it immediately “knows” how to work with GitHub — without you writing any code at all.

// Example MCP tool definition — every tool carries a complete schema
{
  "name": "create_issue",
  "description": "Create a new GitHub issue in a repository",
  "inputSchema": {
    "type": "object",
    "properties": {
      "owner": { "type": "string", "description": "Repository owner" },
      "repo": { "type": "string", "description": "Repository name" },
      "title": { "type": "string", "description": "Issue title" },
      "body": { "type": "string", "description": "Issue description body" }
    },
    "required": ["owner", "repo", "title"]
  }
}

Standardized, type-safe, dynamically discovered — those three keywords look unassailable. Backed by Anthropic and a number of other AI companies, MCP quickly became the “correct choice” for Agent tool calling.

2. MCP’s Reality: The Token Bill Is Much Steeper Than You Think

MCP’s standardization comes at a price. Every Tool has to sit in the System Prompt with its complete description and parameter schema. The create_issue example above is not very long — around 400 Tokens. But a complete MCP Server usually has anywhere from a dozen to several dozen Tools.

A GitHub MCP Server exposes 20 tools at an average of 400 Tokens each; that is 8,000 Tokens — before any work has begun, 8,000 Tokens of the System Prompt are already gone.

What is more, those Tokens are consumed on every turn of the conversation. It is not “load whichever one you use” — every Tool definition lives in the System Prompt and is carried along every single time. For an MCP Server that supports 50 tools, the Tool definitions alone can easily take up more than 20,000 Tokens in the System Prompt.

More troublesome still: those definitions may have nothing at all to do with your actual task. You ask the Agent to “change a config file”, and it has no need whatsoever to know the definitions of create_issue and list_milestones, yet those definitions still show up in the System Prompt, crowding out the space that could have gone to file context and conversation history.

And that is without counting the operational cost of the MCP Server itself. Running an MCP Server means an extra process, extra network overhead and extra authentication management. A simple git diff is one command through the CLI; through MCP it is “Agent → MCP Client → MCP Server → local Git command → return result” — a chain several times as long.

3. The Return of the CLI: Why Models “Already Know” the Command Line

The Token-consumption problem with MCP made everyone take another look at the CLI, that “old approach”. What they found was that the CLI has one fundamental advantage over MCP in Agent tool calling: models have already learned these commands in bulk during training.

GPT-4, Claude, Qwen, DeepSeek — every mainstream model’s training data contains an enormous volume of shell scripts, command-line tutorials and operations manuals. kubectl, git, docker, curl, grep — the syntax, the parameters and the typical usage of these commands have already been internalized by the model.

That means: you do not need to tell the model what kubectl get pods means or how to use it. The model already knows. All you have to do is give the Agent an execute_shell Tool, and it will construct the right command for itself when it needs one.

Compare the Token consumption:

Call method	System Prompt footprint	Extra Tokens per call
MCP (full toolset)	15,000~25,000 Tokens (all Tool definitions)	Small (results returned structured)
CLI (one Shell Tool)	~200 Tokens (one Tool definition)	Small (command output text)

That is two orders of magnitude apart. And MCP’s Token consumption is fixed — whether or not this conversation uses those Tools, the definitions sit in the System Prompt. The CLI’s Token consumption is on demand — it costs you only when you actually call a command.

That is why the CLI is eroding MCP’s share in reverse. Not because the CLI is better, but because it is cheaper. And in an era when Tokens are money, “cheaper” is itself productivity.

4. But the CLI Is Not Perfect Either

The CLI saves Tokens, but it has problems of its own.

Uncontrolled output. When the Agent runs kubectl get pods -o json, the output may be 500 lines of JSON — including a pile of metadata, status details, managedFields and so on that the Agent does not need. Shoving all of that into the context is another Token expense. MCP’s responses are usually filtered on the Server side and return only the key fields after structuring — on the “information density” of the return value, MCP wins.

Crude error handling. The CLI’s output is just stdout + stderr + exit code. The Agent has to work out for itself why a command failed. An MCP Server can hand back structured error information the moment a failure occurs, making it easier for the Agent to understand where the problem lies.

Blurry security boundaries. Giving the Agent a general-purpose execute_shell Tool is giving it access to the entire file system and every executable command. There is no way to say “the Agent may run git but may not run rm -rf” — a Shell Tool does not make that distinction. MCP can achieve fine-grained permission control.

Parameter combinations depend on the Agent “guessing”. kubectl has countless parameters, and although the model “knows” them, complex parameter combinations (fetching a specific field of a specific resource in a specific namespace in a specific format, say) can go wrong. MCP’s type-safety constraints can prevent that class of error.

So the return of the CLI does not mean the death of MCP. It just means the two suit different kinds of scenarios.

5. How to Choose: A Decision Table

Scenario	Recommendation	Reason
Everyday Git operations (diff, log, commit)	CLI	Already internalized by the model; extremely Token-cheap
Common kubectl / docker operations	CLI	The model saw command-line usage in bulk during training
Complex GitHub APIs (Issue management, PR review)	MCP	Many parameters, many steps, type safety needed
Database queries	MCP	Needs connection management, structured results, permission control
Monitoring and alerting (Prometheus / Grafana)	It depends	Simple queries via CLI (`curl`), complex configuration via MCP
In-house internal services	MCP	No ready-made CLI; MCP is the standardized way in
One-off scripts and text processing	CLI	The Agent writes its own Shell script and runs it — flexible and efficient

One practical principle: try the CLI first, and only reach for MCP when the CLI cannot get the job done. Do not start from the assumption that “MCP is better”. MCP’s Token overhead is there the moment you go live; the CLI’s Token overhead is on demand — start with the cheap option and add more only when it is not enough. That is the most pragmatic path in engineering.

6. Summary

The relationship between MCP and the CLI is exactly the reverse of what many people’s intuition suggests. It is not “standardized MCP replacing the crude CLI” but “lightweight CLI proving, in practice, that it is already good enough in most scenarios, and squeezing MCP back into the few areas where it is genuinely necessary”.

That is not a bad thing. This is simply how technology evolves — not every “advanced” approach survives; what survives is the approach with the best cost-performance ratio. MCP has irreplaceable value in the standardized integration of complex third-party services, but across a large number of everyday operations and development scenarios, the CLI — riding on two advantages, “already internalized by the model” plus “zero extra Token cost” — has proved to be the most pragmatic way to call a Tool.

Across three articles — Tool, Skill, and MCP vs. CLI — the same point is really being made: Agent engineering is a process of subtraction. More Tools is not better, more Skills is not better, and a more “advanced” way of calling a Tool is not better either. Restraint, pragmatism, and understanding the trade-offs at each layer — that is the right way to do Agent development.

A small step forward every day is a new height!

An Agent's "Code of Conduct" — the Essence of Skill and the Common Misconceptions

Sun, 07 Jun 2026 00:00:00 +0000

Last time we talked about Tools — the “hands and feet” an Agent uses to operate on the outside world. Once it has hands and feet, the next question follows: how should the Agent use them? When should it do what, how far is far enough to count as “done”, and how should it judge a situation that is ambiguous? That is the problem Skill solves. It is not what tells the Agent “how you use kubectl” — that is the Tool’s business. What Skill tells the Agent is “the right process and principles for doing a thing”. But many people have a deep-rooted misconception about Skill, and this article is about that.

1. What a Skill Is: Custom Processes and Principles

Let’s start from a concrete scenario. You have written a Skill in Hermes called code-review. It goes roughly like this:

## Code Review Skill

在进行代码审查时，遵循以下流程和原则：

### 审查流程
1. 先理解改动的业务背景（从 PR 描述和关联 Issue 中获取）
2. 检查安全性问题（SQL 注入、XSS、敏感信息泄露）
3. 检查性能瓶颈（N+1 查询、不必要的循环、大对象创建）
4. 检查代码风格和可读性
5. 检查测试覆盖（改动是否包含测试，测试是否覆盖了边界情况）

### 判断原则
- 安全问题是 blocker，必须修复
- 性能问题视影响范围决定是否 blocking
- 代码风格问题建议修改但不强制
- 不确定的地方标记为"建议确认"，不要直接判断

### 输出格式
每个问题标注严重程度：[Blocker] / [Major] / [Suggestion]

Notice that this Skill contains no technical detail at all — it does not tell you how grep is used or how git diff is invoked. It is all about process (what order to check things in), principles (what counts as a blocker, what counts as a suggestion) and output conventions (how to label severity).

That is the essence of a Skill: turning experience into an Agent’s code of conduct. The judgements that run naturally through a senior engineer’s head during a code review — “check security first, then performance, style last; zero tolerance on security issues, stop at a hint on style issues” — become rules an Agent can follow, by way of a Skill.

A Skill is not a manual for a Tool. A Tool answers “what can be done”; a Skill answers “how to do it”. The two are not on the same dimension and do not stand in a relationship of replacement or competition.

2. “The More Skills the Better” — The Most Common Misconception

This is also the most deeply held of all the misconceptions about Skill.

Many people using Hermes or similar Agent tools share an intuition: the more Skills you install, the more powerful the Agent becomes. You find a “Python best practices” Skill and install it; you find a “K8s operations standards” Skill and install it; a “database optimization” Skill, install that too. In the end the Agent’s Skill list is enormously long, and it feels reassuring — “my Agent can do everything”.

That intuition is wrong. Adding Skills does not increase an Agent’s capability linearly. The more likely result is that the Agent actually gets dumber.

There are three reasons.

2.1 The Context Blows Up

An Agent’s context window has limits. On every conversation, the content of all the Skills has to be injected into the System Prompt — it is not “load the one you use” but “inject all of them”. Install 20 Skills at 500 words each and your System Prompt has 10,000 extra words. Add the conversation history, the Tool definitions and file memory, and the context is soon filled up.

What gets squeezed out is not Skills but the information the Agent genuinely needs — the complete content of the file you have open right now, or the complete context of your conversation. The Agent is forced to trade off between “remembering the Skills” and “understanding your problem”, and that trade-off itself lowers the quality of its answers.

2.2 Skills May Contradict Each Other

One Skill says “a function should not exceed 50 lines”; another says “do not split for splitting’s sake; logical completeness comes first”. When it comes time to act, the Agent does not know whose rule to follow when the two conflict, because Skill definitions usually contain no priority. The Agent will most likely pick one at random, or take both into account — and the result is a function that gets split and then merged again, the Agent fighting itself.

When you maintain 20 Skills yourself, the further along you get, the harder it is to spot which of them contradict each other. Skills usually come from different sources and are installed at different times — reviewing them all as a whole to check for conflicts is very hard.

2.3 The Agent’s “Difficulty Choosing”

A Skill is not only a “passive rule”. Many Skills also shape the Agent’s behaviour pattern — a code-review Skill, for instance, steers the Agent into “review mode”: more cautious in what it says, asking more confirming questions, preferring to flag a TODO rather than change the code directly. A debugging Skill steers it into “investigation mode”: running diagnostic commands on its own initiative, analysing logs, offering several possibilities.

When the Agent has a pile of Skills loaded at the same time, it becomes hesitant about “which mode to enter”. Its behaviour grows unstable — the same problem, and sometimes it enters review mode (only offering suggestions), sometimes investigation mode (executing commands on its own). This is not the Agent “getting smarter”; it is behaviour drift caused by too many Skills.

3. How to Write a Good Skill

The “few but fine” principle for Skills also points, in reverse, to how a Skill should be designed. A good Skill has a few characteristics:

1. A clear scenario. Do not write a catch-all Skill like “backend development best practices” — the scenario is too broad, and covering everything amounts to covering nothing. It should be a Skill with a clear scenario such as “Code Review”, “Pre-deployment checks” or “Database query optimization”. Then, when the Agent is handling the matching scenario, it can recognize at once that “this Skill is the one to use now”.

2. Priorities and decision criteria. Do not list a pile of rules and let the Agent choose for itself; tell it explicitly what counts as a Blocker, what counts as a Suggestion, and which rule wins when two of them conflict. Write the “judgement” that lives in human experience out in a structured form.

3. Good examples and counter-examples. A bare description of a rule is not as good as an example. For a performance check, for instance, do not just write “avoid N+1 queries” — attach a before-and-after code comparison. An Agent (like a human) learns more solidly from examples than from rules.

4. No overlap with existing Skills. Before installing a new Skill, go through the list of Skills you already have and confirm that the new Skill’s function is not a subset of an existing Skill and that there are no obviously conflicting rules.

5. Light on context. If a rule can be stated clearly, do not write more. A process that can be told in 500 words should not be padded into a 3,000-word encyclopaedia to look “professional”. A Skill’s goal is to help the Agent make decisions, not to help a person look things up in documentation.

4. Summary

If this article conveys one thing, it is this: a Skill is a code of conduct, not a feature list; quality matters far more than quantity.

A Tool lets an Agent “be able to”; a Skill lets an Agent “know how to” — two entirely different questions. However good its hands and feet, without a set of principles and processes for getting things done, the Agent is just flailing. But the reverse holds too: too many principles, fighting each other, and the Agent is worse off than with none — it does not know whose rule to follow, which mode to enter, and it cannot even remember your question in full.

Designing Skills is a process of “subtraction”. What matters is not how many rules you found but how many genuinely useful ones you kept in the end. Three clear enough Skills beat twenty contradictory ones a hundred times over.

Next time, MCP vs. CLI — the two paths for an Agent to call a Tool, and why the situation is the exact opposite of what our intuition suggests.

A small step forward every day is a new height!

The Hands and Feet of an Agent: Tool Capability Boundaries and Their Evolution

Sat, 06 Jun 2026 00:00:00 +0000

However clever a large model is, if it cannot interact with the outside world it is just a talking head. Chatting, writing a poem, fixing a function for you — all of that is still within the scope of “outputting text”. But the moment you want an Agent to do something practical — deploy a service, look something up in a database, open a browser and fill in a form for you — it has to have “hands”. In Agent terminology, those “hands” are called Tools. From the earliest file reads and writes to today’s Computer Use and Browser Use, the capability boundary of Tools has kept expanding outward. This article walks through the evolution of Tools, and, along the way, the things that never changed.

1. What a Tool Is: Function Calling Isn’t “Calling”, It’s “Translating”

Before we talk about Tools, we need to get one thing straight: a large model doesn’t “call” anything.

You tell an Agent to “check the weather” and the Agent does not really open a browser to check. What it does is actually quite simple — it outputs a piece of JSON:

{
  "name": "get_weather",
  "arguments": {
    "city": "Beijing"
  }
}

Then a host program (Hermes, ChatGPT’s backend, your Python script) goes and really executes that function, carries the result back to the model, and the model gives you a natural-language reply based on the result.

This mechanism is called Function Calling. The name is misleading — what it really amounts to is “the model outputs a structured expression of intent, and the host program does the executing”. The model is the brain, the Tool is the hands and feet, and Function Calling is the nerve signal.

Understanding this matters, because it explains the core design principle of Tools: a Tool’s quality does not depend on how powerful its function is, but on how clear its “expression of intent” is. A Tool’s description, its parameter definitions, its return-value format — these determine whether an Agent can “use” it correctly. A badly written Tool (vague description, arbitrary parameter names, a tangle of JSON for a return value) is no different from a hand with a severed nerve — however strong it is, it cannot move.

2. The Evolution of Tools: From Reading and Writing Files to Manipulating the World

The form of Tools has gone through several phases. It is not “a new version replacing an old version” but a capability boundary that keeps expanding outward — just as human tools went from the stone axe to the power drill, not as a replacement but as more and more things they can handle.

2.1 The First Generation: Reading and Writing Files

The earliest Tools, provided by default in almost every Agent framework: read a file, write a file, execute a Shell command.

// The earliest Function Calling — simple and direct
{
  "name": "write_file",
  "arguments": { "path": "/app/config.yaml", "content": "..." }
}

The hallmark of this phase: about all a Tool can do is “write text down”. Changing code, generating a config file, writing a script — at bottom they are all a combination of text generation plus a file write.

That is already enough to solve many problems. In your day-to-day use of Hermes, when you ask it to fix a bug, all it does is: read the file (understand the context), output a revision plan, and write it back. But for scenarios that “need to interact with an external system”, file reads and writes are not enough.

2.2 The Second Generation: Web Requests and API Calls

The Tools of the second phase began to break through the boundary of “local files” — an Agent could now call external APIs directly.

{
  "name": "call_api",
  "arguments": {
    "method": "GET",
    "url": "https://api.example.com/v1/services",
    "headers": { "Authorization": "Bearer xxx" }
  }
}

Behind this sit a series of “connecting to external systems” Tools such as web_fetch and Shell command execution. The Agent is no longer just “writing files locally” — it can query APIs, run database queries, trigger CI/CD pipelines; anything a Shell can do, the Agent can do.

The Tools of this phase are already enough to support the core scenarios of “making DevOps AI-driven”: an Agent can run the entire CI/CD process, from code repository to build server to deployment cluster.

2.3 Standardizing Tools: How MCP Took Tools From “Built by the Vendor” to “Built Together by the Ecosystem”

The two generations of Tools we discussed above — file reads and writes, web requests, Shell execution — are almost all built in by the Agent vendor itself. The vendor’s engineering team defines the Tool’s schema, writes the execution logic, and packages it into the product. Early on this was no problem, because there were few kinds of Tools and the scenarios were concentrated.

But the problem arrived before long: there are far too many external platforms to integrate. Databases include MySQL, PostgreSQL, MongoDB, Redis… cloud platforms include AWS, Azure, GCP, Alibaba Cloud… monitoring includes Grafana, Prometheus, Datadog… collaboration tools include Slack, Jira, Confluence… Every platform has its own API, its own authentication method, its own data format. No vendor could build all of that in — not for lack of wanting to, but because it is impossible.

Before there was a standardized protocol, if you wanted to add a “query Datadog monitoring” capability to an Agent, you either waited for the vendor to support it (which most likely never happened), or wrote your own glue code to wrap the Datadog API into a Tool the Agent could understand. And every vendor defined Tools differently — a Tool you had written for Claude Code would not work in Hermes, and the reverse was just as true. Tools and Agent platforms were tightly coupled.

That is the problem MCP (Model Context Protocol) exists to solve. MCP is a standardized Tool description protocol — it defines a “common language” for Tools: every Tool has a name, a description, a parameter schema and a return format. Whichever Agent platform it is, as long as it supports MCP it can understand and call any Tool Server that implements MCP. Tool developers and Agent platform developers can from then on work independently of each other — connecting through the standard protocol is all it takes.

The significance of this change: the Tool ecosystem went from “built in by the vendor” to “built together by the community”. It used to be that you waited for Anthropic to add a Kubernetes Tool to Claude Code; now a community developer writes an MCP Server and every MCP-capable Agent can use it. Datadog, Grafana, Slack, Jira — as long as somebody writes an MCP Server, the Agent can connect. Vendors do not need to build those Tools in; they only need to implement the MCP protocol properly — that is the decoupling standardization brings.

2.4 The Third Generation: Computer Use — Letting an Agent Drive the Desktop

If the Tools of the first two generations operate on “systems that have an API”, then Computer Use steps into “systems that have a user interface” — an Agent can use desktop software the way a person does.

Anthropic introduced Computer Use at the end of 2024. The principle: the Agent can see a screenshot of the screen, then output operation instructions — mouse clicks, keyboard input, scrolling — and after the host program executes those operations, it hands the Agent a fresh screenshot.

{
  "action": "left_click",
  "coordinate": [450, 320]
}

What is the essential difference from the Tools above? The earlier Tools operate on “interfaces designed to be programmable” — the API has documentation, the parameters have types, the return value has a format. Computer Use operates on “interfaces made for people” — no documentation, no schema, and the output is just an image. The Agent has to look, understand, decide and execute — every step is a combination of visual understanding and reasoning.

What this capability means for DevOps: not every system has an API. Your company may have a server from 15 years ago whose management interface is just a web console, with no API at all. Before, you could only operate it by hand; now an Agent can do it for you — inelegant, but already usable.

Computer Use is not stable enough yet (the latency of each step, the accuracy of visual recognition and the reliability on complex interfaces are all still being improved), but the direction is clear: the Agent’s capability boundary is expanding from “the programmable world” to “the visible world”.

2.5 The Fourth Generation: Browser Use — A New Shape for Web Automation

Browser Use can be seen as a specialized version of Computer Use for the web. It lets an Agent open a browser, browse web pages, click buttons, fill in forms and download files — completing a full web interaction flow.

What is the difference from traditional Web Scraping? Scraping follows preset rules (“find the .price class and extract the text”), while Browser Use has the Agent deciding for itself (“I need to find the price, let me see which element on the page looks like a price”). Hard-coded rules break the moment the page is redesigned; an Agent that searches on the basis of semantic understanding is far more fault-tolerant and generalizes much better.

An example: you are troubleshooting a production issue and need to log into Grafana to look at the CPU trend of some service, and then post the screenshot into the group chat. That used to be three separate things — log into Grafana, take the screenshot, paste the image into the chat tool — and each one needed a person. With Browser Use, the Agent can do it all in one go: open Grafana → navigate to the corresponding Dashboard → set the time range → take a screenshot → send it to the messaging channel.

Behind Browser Use, Playwright or Puppeteer usually drives the browser, but the key difference is who writes the control logic — no longer a developer hard-coding locators and a sequence of operations, but the Agent “reading” the page in real time from a natural-language instruction and deciding what to do next.

3. How the Four Phases Relate: Not Replacement, but Expansion of Capability

Put these four generations of Tools side by side:

Phase	What it operates on	Interaction mode	Typical scenario	Stability
File reads and writes	Local file system	Structured IO	Changing code, writing config, generating documents	★★★★★
Web/API calls	The APIs of external services	Structured request/response	Calling REST APIs, executing Shell scripts, CI/CD automation	★★★★☆
Computer Use	Desktop application interfaces	Screenshot + mouse/keyboard	Driving legacy systems that have no API	★★★☆☆
Browser Use	Web page interfaces	DOM + vision + clicks	Web automation, data collection, monitoring screenshots	★★★☆☆

One key pattern: the further to the right, the more “human-like” a Tool’s operation is; the further to the left, the more “program-like”. “Human-like” means more general but less stable; “program-like” means more specialized but more reliable.

So in real engineering, the principle for choosing a Tool is: if file reads and writes can solve it, don’t call an API; if an API call can do it, don’t reach for Computer Use. Higher-level is not better — Computer Use sounds cool, but the latency and failure rate of each step are far higher than those of a stable API call. Choose the most “program-like” option, and until that option is no longer enough, take one step in the “human-like” direction.

4. More Tools Is Not Better

Having written out the history of their evolution, I have to say one counterintuitive thing: more Tools is not better.

The intuition of many Agent developers is that installing more Tools makes an Agent more capable. It is like adding functions to a Swiss Army knife — one more small blade, one more bottle opener, one more pair of tweezers — it always feels good.

But there is one fundamental difference between an Agent’s Tools and a Swiss Army knife: the Swiss Army knife’s tools are physically separate, while an Agent’s Tools are all crammed into the context. Every extra Tool means one more description and one more set of parameter definitions in the System Prompt. Once the number of Tools goes up, the Agent has to “choose” among them — and that process itself consumes Tokens, consumes reasoning time, and increases the chance of choosing wrong.

Worse still: Tools can interfere with each other. Your JSON-validation Tool is called validate_json, and your weather Tool is called get_weather; they look like they keep to themselves. But when the Agent handles a “user request”, it has to judge “which Tool’s remit this request falls under”. If the descriptions are not written precisely enough, the Agent may route “check this config file for me” to get_weather — because the config file happens to mention “deployed on the cloud”, and the Agent’s chain of reasoning links “cloud” to “weather”.

So the design principle for Tools is not “the more the better” but “just enough”. The Tools you give an Agent should cover exactly what you need it to do; one more is interference.

5. Summary

From file reads and writes to Browser Use, the changing form of Tools reflects the steady expansion of the Agent’s capability boundary. But however the form of Tools changes, a few principles have never changed:

The essence of a Tool is “intent translation” — the Agent does not really execute anything; it only outputs structured instructions. The clearer the description, the more reliably the Agent can use it.
The more “program-like” the Tool you choose, the more reliable it is — in scenarios with high stability requirements (DevOps automation, for example), prefer an API/CLI and keep Computer Use as a last resort.
More Tools is not better — every Tool is an occupation of Context and an interference with reasoning. Trimming down to just enough is the optimal answer.

Next time, we will talk about a concept that is often misunderstood — Skill. A Skill is not a sibling of Tool; it is something on another layer.

A small step forward every day is a new height!

Hermes Memory Provider and Mem0: An Overlooked Memory Abstraction Layer That Lets Agents Share Knowledge

Sun, 24 May 2026 00:00:00 +0000

If you’re using Hermes Agent, chances are you’ve only ever used its file memory — MEMORY.md, USER.md, SKILL.md. That alone is already quite useful: tell it your preferences and it remembers them; work with it enough and it understands you better and better. But what many people don’t know is that Hermes hides a badly underrated capability: the Memory Provider mechanism. It turns “memory storage” into a pluggable abstraction layer, so that your Agent doesn’t just remember you — it can share knowledge across different Agents. For enterprise scenarios that need multiple Agents dividing the work and collaborating, this is the key step from “fighting alone” to “teamwork”.

1. “Wait, Hermes Has This Feature Too?”

Let me start with a real scenario. You set up a “code review Agent” in Hermes and tell it your coding standards: functions no longer than 50 lines, variables named with full words, tests must be run before any change. Two weeks in, this Agent has your preferences down.

Then you spin up a new “deployment Agent” and ask it to help you write deployment scripts. You may well be puzzled — this new Agent knows nothing about those earlier standards. You have to teach it all over again from scratch.

Worse, if you have three Agents — code review, deployment, operations — each of them has its own memory file under ~/.hermes/, isolated from the others. This is the memory silo problem: the Agent isn’t insufficiently smart, it’s locked inside its own “memory cell”.

Many people’s reaction at this point is: “Nothing to be done, that’s just how Agents are.” But Hermes actually thought about this problem long ago.

Hermes doesn’t only record things in files. It has a complete Memory Provider mechanism — one that abstracts “where memory lives” into a replaceable interface. The built-in file storage (MEMORY.md and friends) is merely the default implementation. You can absolutely plug in an external memory service and have multiple Agents share one and the same memory.

I don’t know how many Hermes users are aware this feature exists. My guess: not many. Because “file memory” is already good enough, and the vast majority of tutorials stop at “edit ~/.hermes/config.yaml to configure the model” and never go deeper into the memory.provider configuration key.

This example is itself a reminder: to use a tool well, you can’t stop at the “getting started tutorial” level. Tools evolve far faster than tutorials. The Hello World you see today may be missing a core feature that was added three months ago. Hermes’s Memory Provider has been around since an early 2026 release, yet you can barely find a complete introduction to it in Chinese search results. It isn’t that the feature isn’t good enough — it’s that information spreads with a lag. Learn the fundamentals well and follow updates to the thing itself — these two matter more than chasing every “I built XX system with XX tool” share video.

2. Inside the Memory Provider Mechanism

2.1 Design Philosophy: Built-in Memory and External Providers Coexist

Hermes’s memory architecture has a clever design: built-in file memory is always in effect, and an external Provider is an additive supplement. It’s not “pick one of the two”, it’s “take both”.

Concretely, when a Memory Provider is active, Hermes does the following six things on every conversation:

Injects the Provider’s context into the system prompt
Before each turn, asynchronously prefetches relevant memories from the Provider
Syncs the conversation history to the Provider
Automatically extracts memories at the end of a session (if the Provider supports it)
Mirrors writes to MEMORY.md and USER.md over to the Provider
Exposes Provider-specific tools to the Agent (such as mem0_search, mem0_profile, and so on)

You don’t need to change your business code, don’t need to change your prompts, don’t need to change your habits. You simply have one more “memory backend” and one more “knowledge sharing layer”.

2.2 The Nine Providers at a Glance

As of June 2026, Hermes ships with 9 external Memory Providers. I’ll sort them by deployment model and use case:

Provider	Deployment	Core capabilities	Who it’s for
Mem0	Cloud / self-hosted	Automatic fact extraction, semantic search, multi-tenant isolation	Enterprise scenarios (the focus of this article)
Honcho	Cloud / self-hosted	User modeling, dialectical reasoning, session summaries	Cross-session understanding of users
OpenViking	Self-hosted	Filesystem-style knowledge hierarchy, automatic extraction of 6 memory types	Open-source self-hosting
Holographic	Local SQLite	FTS5 full-text search, trust scoring	Fully local, lightweight
ByteRover	Local / optional cloud sync	Hierarchical knowledge tree, pre-compression extraction, CLI management	Local-first with sync
Hindsight	Cloud / local pg	Knowledge graph, entity relations, reflective synthesis across memories	You need graph reasoning
RetainDB	Cloud	Hybrid search (BM25 + vectors), 7 memory types, incremental compression	You need advanced search
Supermemory	Cloud	Context isolation, session graph, multiple containers, user profile injection	Preventing memory contamination
Memori	Cloud	Structured memory, attribution by project/session, background ingestion	Structured scenarios

That’s more than most people would expect. And every Provider has a clear design philosophy — not “big and all-encompassing”, but “each one solves a particular class of memory problem”. Need knowledge-graph reasoning, go with Hindsight; need to be fully local, go with Holographic; need cloud-native multi-tenancy, go with Mem0 or RetainDB.

2.3 The Limitation: Only One Provider Can Be Enabled at a Time

One limitation of Hermes: only one external Memory Provider can be active at a time. Built-in file memory and an external Provider can coexist, but you can’t use two external Providers at the same time (Mem0 + Honcho, for example, can’t both be on).

For now that limitation looks reasonable — several Providers injecting context at once would blow up the System Prompt. But if you happen to be hesitating between two Providers, choose by “fit” rather than by “number of features”: pick the one that best matches your pain point.

3. Mem0 in Depth: Memory Infrastructure for Enterprise Autonomous Agents

Why focus on Mem0 out of the nine Providers? Because it best fits the needs of enterprise autonomous AI Agent workflows. When you deploy Agents in a team to do real work — not just a personal assistant, but multiple Agents dividing the work and collaborating — you run into these very real problems:

Memory must be shared across Agents, yet isolated along the Agent dimension
Memory must be persistent, not lost on restart
Memory must support semantic retrieval — finding “user preferences” by keyword search is wishful thinking
Memory must have an audit log (who recorded and changed what, and when)
The whole memory system must be self-hosted, with no data leaving the intranet

Mem0 is designed for every one of those needs. It’s currently the most active Agent-specific memory layer project on GitHub, recommended by Anthropic, LangChain and several other teams, with an API simple enough that three lines of code wire it into any Agent framework.

3.1 How Mem0 Processes Memory

When you call memory.add("用户喜欢简洁的命名风格"), Mem0 does five things behind the scenes:

Fact extraction: calls an LLM to automatically extract structured facts from the text (entity: user, relation: preference, object: concise naming)
Deduplication and merging: if a similar memory already exists, it merges rather than writing a duplicate. This avoids “the same preference being recorded 20 times”.
Vectorization: converts the memory into a vector through an Embedding model
Storage: vectors go into Qdrant (or another vector store), metadata goes into SQLite
History: writes this operation to the history table — inserts, deletes and updates are all traceable

Retrieval uses a three-layer hybrid strategy:

Vector semantic search — finds semantically similar memories (“naming convention” can match “variable naming preference”)
BM25 keyword search — captures exact keyword matches, compensating for vector search’s “approximately equal” bias
Entity filtering — filters along the user_id and agent_id dimensions to achieve multi-tenant isolation

The three result sets are finally fed into a Reranker for re-ranking, which outputs the most relevant Top-N memories. That means the memories injected into the Agent’s context are both relevant and concise — no wasted Tokens.

3.2 Multi-Tenant Isolation: A Native Capability for Enterprise Scenarios

The two-dimensional isolation of user_id + agent_id is Mem0’s most powerful answer to the multi-Agent scenario:

# Code review Agent
memory.add("函数不超过 50 行", user_id="team-backend", agent_id="code-reviewer")

# Deployment Agent
memory.add("部署前必须跑冒烟测试", user_id="team-backend", agent_id="deployer")

# Operations Agent searches memory
results = memory.search("代码规范", user_id="team-backend", agent_id="maintainer")

Search results retain the memories from the user_id=team-backend dimension, but cross-Agent sharing can also be configured. The flexibility is high enough: fully isolated, shared within a team, or shared globally.

3.3 Self-Hosting: No Data Leaves the Intranet

Full-pipeline self-hosting is supported:

config = {
    "vector_store": {
        "provider": "qdrant",
        "config": {
            "host": "qdrant-internal",
            "port": 6333
        }
    },
    "llm": {
        "provider": "openai",
        "config": {
            "model": "qwen2.5-7b-instruct",
            "api_base": "http://vllm-internal:8000/v1",
            "api_key": "not-needed"
        }
    },
    "embedder": {
        "provider": "huggingface",
        "config": {
            "model": "BAAI/bge-m3"
        }
    }
}

All three key backends can be localized:

Vector store: Qdrant / Milvus / ChromaDB / Weaviate
Extraction LLM: a local vLLM or Ollama (you don’t need GPT-4; a small model is enough for good fact extraction)
Embedding: BGE-M3 or a local Embedding model

The entire system runs on the intranet, and not a single line of data leaves your servers.

4. Hands On: Wiring Mem0 into Hermes

4.1 Deploying Mem0

First start Qdrant (the vector database):

docker run -d --name qdrant -p 6333:6333 qdrant/qdrant

Install Mem0:

pip install mem0ai

4.2 Configuring Hermes to Use Mem0

Interactive configuration is the simplest:

# One step and done
hermes memory setup

The list of options shows every supported Provider; type mem0 to select it. Then follow the prompts to fill in your API Key (if you’re using the cloud version) or your self-hosted endpoint information.

Or change the configuration manually:

# Set the Provider
hermes config set memory.provider mem0

# Configure the keys (written to ~/.hermes/.env)
echo "MEM0_API_KEY=your-key" >> ~/.hermes/.env
echo "MEM0_API_URL=http://your-mem0-server:8000" >> ~/.hermes/.env

# Configure the user identity (optional; specify user_id and agent_id in ~/.hermes/mem0.json)

4.3 Verifying: Check Whether Memory Is Syncing

After configuring, chat a few lines with Hermes about anything:

hermes chat "Our team's backend services all run Go 1.22+ and are deployed on K8s"
hermes chat "Write me a K8s deployment template"

After the second conversation, check the memory:

hermes chat "mem0_search: Go version requirement"

If the answer is correct, then the memory made it into Mem0 and can be retrieved.

4.4 Multi-Agent Verification

Using Hermes’s Profile mechanism (see the earlier Profile article), create two separate Agent personas, then point them at the same Mem0 instance — with different agent_ids but the same user_id:

# Profile A (code review Agent)
hermes -p code-reviewer config set memory.provider mem0

# Profile B (deployment Agent)
hermes -p deployer config set memory.provider mem0

Now a preference recorded in Agent A can also be retrieved by Agent B. Memory is no longer an island.

5. Why Enterprises Should Care

At this point you might be thinking, “it’s just one more backend, what’s the big deal”. But shift your perspective from “individual user” to “enterprise DevOps team” and you’ll see that this step is a qualitative leap.

5.1 The Watershed from Single-Agent to Multi-Agent

Personal use case: one Agent, one project, one person. File memory is enough.

Team use case: three Agents (review, deployment, operations) sharing one project context, and memory has to flow between them. File memory is no longer enough.

The Memory Provider is the bridge over that watershed. Without changing tools or processes, just by configuring one more backend, it switches you from “single-Agent mode” to “multi-Agent collaboration mode”.

5.2 Auditing and Compliance

With file memory, a change is simply a change — there’s no version history. When the team is troubleshooting a production incident and needs to trace back “when did the Agent remember that bad configuration?”, file memory can’t answer.

Mem0’s history table logs every memory operation — what time, which Agent, what change was made. In compliance scenarios that’s a hard requirement.

5.3 Switch Freely, Without Lock-In

Hermes’s Provider mechanism has an underrated advantage: you can switch between Providers without losing your memory. That’s because built-in file memory is always in effect — even if you change Providers, the memories in MEMORY.md and USER.md are still there. An external Provider is an additive layer, not a replacement layer.

This lowers the cost of trial and error: you can try Mem0 first, and if you’re not satisfied, switch to Honcho or Holographic — file memory is always there.

6. The Complete Architecture: LiteLLM + Mem0 + Hermes

If you’ve read the earlier LiteLLM article, here is the complete puzzle of an enterprise AI Agent infrastructure:

企业内网
│
├── Model Access Layer: LiteLLM Gateway
│   ├── Unified OpenAI-compatible API
│   ├── Multi-tenant budget control (by team/project)
│   ├── Automatic Fallback (public→public, intranet→intranet)
│   └── Cost tracking and audit logs
│
├── Model Inference Layer
│   ├── Local vLLM (Qwen / DeepSeek)
│   └── Public models (unified access via LiteLLM)
│
├── Agent Layer: Hermes Agent
│   ├── Agent A (code review)── MyMemory Provider → Mem0
│   ├── Agent B (deployment)─── MyMemory Provider → Mem0
│   └── Agent C (operations)─── MyMemory Provider → Mem0
│
└── Memory Layer: Mem0 + Qdrant
    ├── Multi-Agent shared memory
    ├── Multi-tenant isolation (user_id / agent_id)
    ├── Hybrid retrieval (vector + BM25 + Reranker)
    └── Operation audit logs

Every layer is decoupled, every layer is replaceable. That’s the mindset of “AI infrastructure”: locked into no single component, dependent on no single vendor, with every capability closed-loop inside the intranet.

7. Summary

What I most want to stress in this article isn’t actually how good Mem0 is — though it really is good — but the fact that “the tool you’re using may be more powerful than you imagine”.

Among Hermes users, how many know that it supports 9 external memory backends? How many know that a single configuration key can solve multi-Agent knowledge sharing? I believe the proportion is low. That’s not the users’ fault — tools iterate fast, and the documentation is plentiful but scattered, so the people who truly go deep on a given feature are always few. But if you want to genuinely put AI Agents to work in an enterprise, you can’t stay at the level of “I can use the tool” — you have to reach the level of “I understand the tool’s architectural design”.

Back to Memory Provider: its value lies in upgrading “the Agent’s memory” from “an incidental feature of a single program” to “an infrastructure capability that can evolve independently”. Today you’re using Mem0, and six months from now you may swap in a better backend — the Agent code won’t need to change. That’s what a good abstraction should look like.

A small step forward every day is a new height!

Hermes Profile — Multiple Agent Personas, One Set of Tools Does It All

Sat, 23 May 2026 00:00:00 +0000

You’ve raised an Agent well inside Hermes — it remembers your code style, your project structure, your frequently used commands. Now you have a second need: an Agent dedicated to managing deployments, and one more dedicated to ops inspections. They’re three completely different roles, needing different memories, different skills, and even different model preferences. Surely you’re not going to reinstall Hermes every time? That’s exactly what Hermes’s Profile mechanism was designed for.

1. The Problem: One Hermes Isn’t Enough

A quick recap: Hermes Agent stores each Agent’s configuration, memory, and skills under the ~/.hermes/ directory:

~/.hermes/
├── config.yaml       # main config (model, tools, etc.)
├── .env              # environment variables (API Key, etc.)
├── MEMORY.md         # episodic memory
├── USER.md           # user profile
├── SKILL.md          # procedural memory
└── skills/           # skills directory

This structure is good — for one Agent, one role. But the real-world scenario is that you might need:

A code review Agent: leaning toward code quality and refactoring suggestions, using a model with a large context
A deployment Agent: leaning toward K8s and CI/CD operations, using a small, fast model
An ops inspection Agent: running inspection tasks on a schedule and logging inspection records, using the cheapest model

The three roles’ memory contents are completely different. The code review Agent remembers your coding conventions, the deployment Agent remembers your K8s cluster topology, and the ops Agent records historical failure patterns. Mix them together and the memory files turn into a pot of porridge.

You might think: then why not copy ~/.hermes/ three times, one for each role? Sure, that works — but that’s exactly what Profile does, only it manages it better for you.

2. What Is a Profile

A Profile is Hermes’s built-in configuration isolation mechanism. Each Profile is an independent Hermes configuration environment, containing its own config files, memory files, skills directory, and environment variables. One Profile = one logical Agent persona.

To put it more plainly: without installing three Hermes instances, you can have three completely independent Agents. Each keeps its own memories, configures its own model, and installs its own skills.

The Profile mechanism is everywhere in Hermes. With the --profile (or -p) global parameter, you can make any command act on a specified Profile; with hermes profile use you can switch the default Profile:

hermes profile create my-agent                  # create a new profile
hermes --profile my-agent chat "帮我看一段代码"  # run one conversation with the specified profile
hermes profile use my-agent                     # switch the default profile (omit --profile from then on)
hermes -p my-agent config set model qwen2.5     # configure using the -p shorthand
hermes -p my-agent skill add k8s-deploy-guide   # install a skill for the specified profile

When you don’t specify --profile, Hermes uses the currently active default Profile (named default by default).

3. Hands On: Creating Your First Persona

Assume your default Profile is already a code review Agent. Now create a deployment Agent.

3.1 Creating a New Profile

# create a blank profile named deployer
hermes profile create deployer

You can also use --clone to copy the configuration from the current Profile:

# copy the current profile's config.yaml, .env and SOUL.md into the new profile
hermes profile create deployer --clone

# a full clone — memories, sessions and skills copied along too
hermes profile create deployer --clone-all

After creation, a deployer/ subdirectory is generated under ~/.hermes/profiles/, containing a brand-new set of config and memory files.

The two Profiles are completely isolated:

~/.hermes/
├── config.yaml          # the default profile's configuration
├── MEMORY.md            # the default profile's memory
├── USER.md              # the default profile's user profile
├── SKILL.md             # the default profile's skills
├── .env                 # the default profile's environment variables
│
├── profiles/
│   └── deployer/
│       ├── config.yaml  # the deployer profile's configuration (separate!)
│       ├── MEMORY.md    # the deployer profile's memory
│       ├── USER.md
│       ├── SKILL.md
│       └── .env

3.2 Switching Profiles

There are two ways. One is to set the default Profile, after which all commands point to it automatically:

# set deployer as the default profile
hermes profile use deployer

# from now on, chat directly and deployer is used automatically
hermes chat "Write a K8s deployment template for me"

The second is to specify it temporarily for each command, without changing the default:

# temporarily chat with the deployer profile
hermes --profile deployer chat "帮我写一个 K8s deployment 模板"

# temporarily install a skill with the deployer profile
hermes -p deployer skill add k8s-deploy-guide

3.3 Viewing All Profiles

# list all profiles
hermes profile list

The output is something like:

* default
  deployer
  maintainer

The one with a leading * is the currently active default Profile. Use hermes profile use <name> to switch.

3.4 Deleting a Profile

# delete the deployer profile (removing all of its configuration and memory)
hermes profile delete deployer

Note: a delete operation is irreversible. But each Profile is independent, and deleting deployer won’t affect default.

We just said “completely isolated,” but some things are shared:

Item	Isolated?	Description
Conversation history	Isolated	Each Profile has its own conversation records
MEMORY.md / USER.md / SKILL.md	Isolated	Each keeps its own
Model configuration	Isolated	Each Profile can use a different model
Skill files	Isolated	Each Profile installs different skills
Global plugins	Shared	Plugins are global and shared by all Profiles
Tool authorization	Isolated	Each Profile authorizes tool permissions independently

You can configure them flexibly according to each role’s needs:

The code review Agent uses Qwen2.5-14B, with file operation tools enabled
The deployment Agent uses DeepSeek-V3, with Shell execution tools enabled
The ops Agent uses a small local model, with only log-reading tools enabled

Three Profiles, three sets of memories, three differently positioned Agents — but all on the same machine, under the same Hermes installation.

5. Why Profiles Are the Foundation for Multi-Agent Collaboration

By now you might feel that three Agents running independently, each keeping its own memories, is already pretty good. But once you really step into a multi-Agent collaboration scenario, you’ll find a key problem: they need to share some knowledge.

For example:

The code review Agent remembers that “your microservices are written in Go and deployed on K8s”
The deployment Agent needs to know this background when writing deployment templates
The ops Agent also needs to know that Go and K8s are in use when troubleshooting

Although the three Profiles isolate memory at the file level, you can absolutely connect them to the same external Memory Provider — that’s the Hermes Memory Provider mechanism we’ll cover in the next article, and in particular the Mem0 enterprise-grade memory layer.

# connect all profiles to the same Mem0 instance
hermes -p code-reviewer config set memory.provider mem0
hermes -p deployer config set memory.provider mem0
hermes -p maintainer config set memory.provider mem0

Profiles provide the foundation of “independent roles,” and the Memory Provider provides the bridge for “shared knowledge.” Put the two together and you have the complete solution for multi-Agent collaboration.

6. Summary

Profiles are the most easily skipped feature in Hermes — introductory tutorials usually don’t need it, and one Agent is enough. But once you start seriously doing multi-role Agent collaboration, it becomes a foundation you can’t route around.

Its value lies in this: it gives you “persona” capability without changing your Hermes installation. One machine, one Hermes, running three completely differently positioned Agents. And each Agent’s configuration, memory, and skills are cleanly isolated from one another, with no cross-contamination.

This sets up the next article’s connection — the Memory Provider lets these personas share an enterprise-grade knowledge layer beyond their file-based memory.

A small step forward every day is a new height!

LiteLLM: A Unified LLM Gateway — From API Key Chaos to Enterprise AI Infrastructure

Sat, 16 May 2026 00:00:00 +0000

When you start using large models for real work — not the occasional API call for fun, but treating them as team infrastructure — you soon hit a wall. Not the “the models aren’t strong enough” wall, but the “I can’t keep this under control” wall: API keys scattered everywhere, no idea who spent how much, one model goes down and the whole business goes dark, and no audit trail at all for whether data ever left the intranet. What you need at that point is not a stronger model, but a gateway that can manage models. LiteLLM is that gateway.

1. The Problem: When You Start Treating AI as Infrastructure

Over the past two years, the path large models have taken into production has looked roughly like this: first individuals chatting with ChatGPT → then teams using APIs for various experiments → then AI capabilities gradually embedding into daily workflows and becoming a “standard component”, just like your database or your message queue.

The problems start at the third step. Once AI goes from a “personal toy” to “enterprise infrastructure”, you’ll find:

Scenario one: API keys scattered everywhere. The algorithm team uses OpenAI, the product team uses Claude, intranet scenarios use a local vLLM. Every team manages its own keys and writes its own calling code. One department with five project teams may have five different ways of integrating models. Ops wants unified management? Sorry — nobody can say with confidence how many keys are actually in circulation.

Scenario two: the cost black hole. No idea which team spent how many tokens. At month-end the OpenAI bill is one big lump: you can’t tell which project ran batch offline jobs on GPT-4, and which team used a flagship model for simple translation. Budget control can only be all-or-nothing; there’s no way to be precise down to team, project or environment.

Scenario three: a chain of single points of failure. OpenAI goes down and every service depending on it grinds to a halt. A local vLLM instance restarts and the running jobs all fail together. There is no fallback mechanism at all — not because nobody thought of it, but because “the cost of changing code to add a Fallback is too high”.

Scenario four: compliance audit blind spots. There’s no unified log of who called which model and what data they sent. Whether sensitive data was sent out to a public model API is completely untraceable after the fact — not because you don’t want to control it, but because you can’t.

These four scenarios point to the same root cause: the model-calling layer lacks a unified gateway. Just as in a microservices architecture you would never let every service manage its own database connections and message queues — you add a layer in between. The same goes for large models: you need a unified access layer between your business and the models. LiteLLM is that access layer.

2. What LiteLLM Is

LiteLLM is an open-source LLM gateway (GitHub 20K+ stars), maintained by the BerriAI team. Its core idea is simple: unify all model APIs into the OpenAI-compatible format — one endpoint, one interface protocol, calling 100+ kinds of models.

It has two usage modes:

SDK mode: a pure Python library, pip install litellm, suited to local scripts and small tools.
Proxy mode: a standalone HTTP service, the standard way to run it in production.

This article focuses on Proxy mode — that’s the way enterprises use it. Proxy mode not only provides a unified interface, but also a full set of gateway capabilities: routing, rate limiting, multi-tenant budgets, audit logs, Fallback and more.

Why the “OpenAI-compatible format”? It’s the de facto industry standard today. The vast majority of AI tools — Hermes Agent, OpenClaw, Cline, OpenHands — support connecting to models through an OpenAI-compatible interface. LiteLLM wraps all your models (whether the backend is OpenAI, Claude, a local vLLM or Azure) into the OpenAI-compatible format. On the business side you only change base_url and api_key; you don’t touch application code.

3. Hands On: Deploying a LiteLLM Gateway Quickly

Let’s start with a minimal version that unifies the models, then add features step by step.

3.1 Minimal Working Configuration

config.yaml:

model_list:
  - model_name: fast
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

  - model_name: smart
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: local-qwen
    litellm_params:
      model: openai/qwen2.5-14b-instruct
      api_base: http://vllm-internal:8000/v1
      api_key: EMPTY

general_settings:
  master_key: sk-master-xxxxxx

litellm_settings:
  drop_params: true
  cache: true

Start it with a single Docker command:

docker run -d --name litellm -p 4000:4000 \
    -v $PWD/config.yaml:/app/config.yaml \
    -e OPENAI_API_KEY=$OPENAI_API_KEY \
    ghcr.io/berriai/litellm:v1.85.0 \
    --config /app/config.yaml --num_workers 4

Verify it:

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-master-xxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"model": "fast", "messages": [{"role": "user", "content": "Who are you?"}]}'

You’ll get back a normal OpenAI-format response. To the business side, this is an “OpenAI service” — except that behind it, it may point at any model at all.

3.2 Connecting a Local vLLM

The most common intranet setup: local models as the mainstay, public models as a supplement. Add one entry to model_list:

  - model_name: local-qwen
    litellm_params:
      model: openai/qwen2.5-14b-instruct
      api_base: http://your-gpu-server:8000/v1
      api_key: EMPTY

model: openai/xxx means: call it using the OpenAI-compatible protocol. api_base points at your vLLM instance. For any tool that supports an OpenAI-compatible interface — Hermes, OpenClaw, Cline — just change base_url to http://litellm:4000.

3.3 Key Security Settings

A few details that are easy to overlook:

Pin the image version (v1.85.0 rather than latest). In March 2026 there was an incident in which malicious code was injected into LiteLLM’s PyPI package; since then, pinning the version and verifying image signatures is strongly recommended.
Salt Key: LITELLM_SALT_KEY is used to encrypt the provider API keys in the database. Never change it once deployed — change it and all the historical encrypted data becomes undecryptable.
Master Key is only for administrative operations, not for business calls. Use a Virtual Key for everyday calls.

4. Core Features: Routing and Fallback

Providing a unified interface alone isn’t enough. The gateway’s real value lies in traffic control.

4.1 Model Aliases — the Business Side Doesn’t Notice Backend Changes

Give a model a “business name” (fast, smart, local); several backends can hang under the same business name:

model_list:
  - model_name: fast
    litellm_params:
      model: openai/gpt-4o-mini
  - model_name: fast
    litellm_params:
      model: azure/gpt-4o-mini

router_settings:
  routing_strategy: simple-shuffle

Your business code calls model="fast"; whether that actually goes to OpenAI or Azure behind the scenes is decided by the gateway. Ops can switch the underlying model without the business changing a single line of code.

4.2 Fallback — a Dead Model Doesn’t Affect the Business

This is a standard gateway capability:

router_settings:
  fallbacks:
    - fast: ["smart", "local-qwen"]
  context_window_fallbacks:
    - fast: ["smart"]
  content_policy_fallbacks:
    - smart: ["fast"]

Three Fallback scenarios:

The model is down (fallbacks): when OpenAI is unavailable, traffic automatically goes to Claude → local model. Users notice nothing.
Context limit exceeded (context_window_fallbacks): when gpt-4o-mini runs out of context, automatically upgrade to gpt-4o.
Content policy refusal (content_policy_fallbacks): when a model refuses to answer for compliance reasons, retry with another model.

Fallback strategy advice for intranet DevOps scenarios: in data-sensitive cases, fall back only to local models and never go to the public internet; in non-sensitive cases, public models can back each other up. Use model grouping and routing strategy for fine-grained control — for example, configure the local-* model group to fall back only within the group.

4.3 Choosing a Routing Strategy

Strategy	How it works	When to use it
`simple-shuffle`	Simple round-robin (automatically weighted when there are rpm/tpm limits)	Small to medium scale, good enough
`least-busy`	Routes to the backend with the lowest current load (requires Redis)	High concurrency, many replicas
`usage-based-routing-v2`	Spreads pressure by TPM usage rate	Multi-instance deployments
`latency-based-routing`	Selects by response latency	Latency-sensitive services

For a small team, simple-shuffle is enough. When you need to scale vLLM instances out horizontally, move up to least-busy.

5. Enterprise Features: Multi-Tenancy and Budget Control

This may be the most “valuable” part of LiteLLM for DevOps teams — managing AI resources as infrastructure.

5.1 Production Deployment: Add Postgres

The minimal configuration has no database and can only be driven by config.yaml. In production you need Postgres to store virtual keys, team configuration and spend logs:

general_settings:
  master_key: sk-master-xxxx
  database_url: postgresql://litellm:password@postgres:5432/litellm

A Docker Compose reference:

services:
  postgres:
    image: postgres:18-alpine
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 5s
      retries: 5

  litellm:
    image: ghcr.io/berriai/litellm:v1.85.0
    depends_on:
      postgres:
        condition: service_healthy
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
    environment:
      DATABASE_URL: postgresql://litellm:${DB_PASSWORD}@postgres:5432/litellm
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
      LITELLM_SALT_KEY: ${LITELLM_SALT_KEY}
      OPENAI_API_KEY: ${OPENAI_API_KEY}
    command: --config /app/config.yaml --num_workers 4

volumes:
  pgdata:

With a database in place, all of LiteLLM’s operation history is recorded automatically in the LiteLLM_SpendLogs table, and from then on API calls have a complete “ledger”.

5.2 The Admin Panel — Day-to-Day Ops Without curl

Once the database is configured, LiteLLM’s built-in web admin panel is activated. Visit http://litellm:4000/ui and log in with the Master Key, and you’ll see a complete admin interface. Here you can:

Create and manage Teams, setting a per-team budget cap and a whitelist of available models
Generate, revoke and edit Virtual Keys — bind them to a Team, set a spending limit, configure rate limits
View the live request stream, per-model QPS and P95 latency
Browse spend reports and break spending down by team, Key and model
Search call logs to quickly locate errors

This is what ops looks like when it follows engineering practice. Day-to-day operations — issuing a Key for a new project, adjusting a team budget, revoking a leaked Key — are all done with a few clicks in the UI. No digging through docs to type curl commands.

The API examples below are kept for two purposes: first, to show the full meaning of every parameter (the UI and the API correspond one-to-one), and second, to make it easy for you to automate operations in CI/CD or ops scripts.

5.3 Team — Resource Isolation by Team

Creating a Team in the admin panel takes only a name, a budget and the allowed models. The API parameters behind it are as follows:

# Create the backend team
curl http://litellm:4000/team/new \
  -H "Authorization: Bearer sk-master-xxxx" \
  -d '{
    "team_alias": "backend-team",
    "max_budget": 500.0,
    "budget_duration": "30d",
    "models": ["fast", "smart", "local-qwen"]
  }'

# Create the product team (cheap models only)
curl http://litellm:4000/team/new \
  -H "Authorization: Bearer sk-master-xxxx" \
  -d '{
    "team_alias": "product-team",
    "max_budget": 100.0,
    "budget_duration": "30d",
    "models": ["fast", "local-qwen"]
  }'

One budget cycle is 30 days; requests are automatically rejected once the budget is exceeded, so there are no end-of-month reconciliation shocks.

5.4 Virtual Key — Fine-Grained Control by Environment and by Service

In the admin panel, open a Team’s detail page and click “Create Key” to generate a Virtual Key — just fill in an alias, budget, rate limits and a model whitelist. The corresponding API parameters are as follows:

# Production: permissive
curl http://litellm:4000/key/generate \
  -H "Authorization: Bearer sk-master-xxxx" \
  -d '{
    "key_alias": "backend-prod",
    "team_id": "team-id-xxx",
    "max_budget": 400.0,
    "budget_duration": "30d",
    "tpm_limit": 200000,
    "rpm_limit": 1000,
    "models": ["fast", "smart", "local-qwen"]
  }'

# Development: rate-limited, to prevent misuse
curl http://litellm:4000/key/generate \
  -H "Authorization: Bearer sk-master-xxxx" \
  -d '{
    "key_alias": "backend-dev",
    "team_id": "team-id-xxx",
    "max_budget": 100.0,
    "budget_duration": "30d",
    "tpm_limit": 20000,
    "rpm_limit": 100,
    "models": ["fast", "local-qwen"]
  }'

5.5 Using It from the Business Side — Completely Transparent

Once you have a Virtual Key, the business side simply uses it as an OpenAI key:

from openai import OpenAI

client = OpenAI(
    base_url="http://litellm-internal:4000/v1",
    api_key="sk-xxxx-backend-prod-virtual-key"
)

response = client.chat.completions.create(
    model="fast",
    messages=[{"role": "user", "content": "Write a unit test for me"}]
)

From the business side it’s completely invisible — they neither know nor care whether the backend is OpenAI, Claude or a local vLLM. They only need to know “same place, same Key, same format”.

6. Cost Tracking and Observability

Once LiteLLM is connected to Postgres, every API call is recorded in the LiteLLM_SpendLogs table. The core fields: which Team, which Key, which model, input/output tokens, the amount spent (US dollars), and a timestamp.

6.1 The Admin Panel — Check Spending in the UI Day to Day

The admin panel (http://litellm:4000/ui) comes with a Usage page, so checking spending day to day doesn’t require leaving the browser:

Spend: view team spending trends by day/week/month, and see at a glance how much each Key has spent
Usage: request volume, token consumption and average latency per model
Logs: detailed logs for every call — request content, response, cost, latency — with filtering and search
Teams / Keys: budget utilization per team, so you can see directly which one is about to exceed its budget

Checking spending, reconciling, tracking down “why did this Key spend so much today” — the UI handles all of it. This is the way that matches engineering instincts: first give people something to look at, then talk about advanced tricks.

6.2 Going Further: Query the Database Directly

When the UI’s aggregation dimensions aren’t enough — say you want to group by project tag (metadata), export CSV for finance, or feed an internal data platform — you can query the LiteLLM_SpendLogs table directly:

-- Team spend ranking for the last 7 days
SELECT team_id, SUM(spend) AS total_spend, COUNT(*) AS num_requests
FROM "LiteLLM_SpendLogs"
WHERE "startTime" > NOW() - INTERVAL '7 days'
GROUP BY team_id
ORDER BY total_spend DESC;

-- Daily spend trend for a given Key
SELECT DATE("startTime") AS day, SUM(spend) AS daily_spend
FROM "LiteLLM_SpendLogs"
WHERE api_key = 'sk-xxxx-backend-prod'
GROUP BY day
ORDER BY day DESC;

For DevOps, SQL is flexible for aggregating by any dimension and for join analysis. But day-to-day ops doesn’t need it — the UI is enough.

Beyond that, if you already have a monitoring stack such as Grafana, LiteLLM can also expose Prometheus metrics, which can be wired into your existing dashboards.

6.3 Alerting Strategy Advice

The same thinking as for any other infrastructure you operate — key metrics must have alerts:

A single team’s daily spend exceeds 60% of its budget → notify the team lead
A Key’s RPM suddenly jumps 5x → risk alert (could be a code bug or a leaked Key)
A backend model’s error rate stays above 10% for 5 minutes → ops alert
A local vLLM instance becomes unavailable → alert immediately (the intranet model is a core dependency)

7. A Kubernetes Production Deployment Reference

If your team uses K8s, here’s a reference structure:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm
spec:
  replicas: 2
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
    spec:
      containers:
        - name: litellm
          image: ghcr.io/berriai/litellm:v1.85.0
          args:
            - "--config=/etc/litellm/config.yaml"
            - "--num_workers=4"
            - "--port=4000"
          ports:
            - containerPort: 4000
          envFrom:
            - secretRef:
                name: litellm-secrets
          volumeMounts:
            - name: config
              mountPath: /etc/litellm
              readOnly: true
          livenessProbe:
            httpGet:
              path: /health/liveliness
              port: 4000
          readinessProbe:
            httpGet:
              path: /health/readiness
              port: 4000
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: 2
              memory: 2Gi
      volumes:
        - name: config
          configMap:
            name: litellm-config

A few key points:

Multiple replicas: LiteLLM itself is stateless (state lives in Postgres), so multiple replicas natively support high availability. --num_workers 4 is the number of workers inside each replica.
Secret management: the Master Key, Salt Key and every provider’s API Key all come from K8s Secrets — never hard-code them into the ConfigMap.
Health checks: /health/liveliness and /health/readiness are LiteLLM’s own standard endpoints.
Redis: in multi-replica scenarios, add Redis (router_settings.redis_host) so that replicas share routing state.

8. Lessons from the Trenches

Pitfalls that are easy to hit during an actual deployment:

1. The side effects of drop_params. Once enabled, parameters that the target model doesn’t support are silently dropped. In multi-model scenarios this is genuinely convenient — you pass reasoning_effort=high to GPT-4o, and it’s automatically dropped when the request routes to Claude, without an error. But during debugging, turn it off first, or you’ll find yourself asking “I clearly passed the parameter, so why isn’t it taking effect?”

2. Postgres connection bloat. num_workers=4 + 2 replicas = 8 database connection pools. If your Postgres is configured with a low max_connections, you may run out of connections. The solution is to put PgBouncer in front, or lower database_connection_pool_limit.

3. A Fallback may be more expensive. The fallback model may cost more than the primary — for example, gpt-4o-mini goes down and traffic falls back to gpt-4o, a 10x price difference. When configuring a Fallback chain, confirm the price of each model and prefer ones in the same price range.

4. The Salt Key cannot be changed. Generate LITELLM_SALT_KEY before deployment, and never change it afterwards. Once changed, every encrypted Provider Key in the database becomes undecryptable, and you need to wipe the database and rebuild it.

5. Don’t mix configuration sources. After store_model_in_db: true, model configuration is loaded from the database. At that point, don’t modify config.yaml and add or remove models through the API at the same time — the two configuration sources overwrite each other, and debugging that is painful. Pick one approach and stick with it to the end.

6. Don’t expose local models to the public internet. LiteLLM can mix local vLLM and public models on one gateway. Take care to use model.info.access_groups for access control — make sure call chains carrying sensitive data only route to local models.

9. Wrapping Up: Back to “Infrastructure” Thinking

Going from large-model APIs to a unified gateway, the essence of that leap is a switch in thinking — from “using AI tools” to “operating AI infrastructure”. Just as you wouldn’t hard-code a database connection string into every microservice, you shouldn’t let every team manage its own API Keys, maintain its own Fallback logic and do its own reconciliation.

The problems LiteLLM solves are exactly the ones you inevitably run into on the way from “one person calling an API” to “a team using AI”: a unified entry point, traffic control, cost accounting, security auditing. It doesn’t produce stronger AI, but it makes your AI infrastructure as manageable, measurable and auditable as your database or your message queue.

The deployment cost isn’t high — one Docker Compose (Postgres + LiteLLM) plus the vLLM instance you already have, and it’s running on your intranet in half an hour. Worth a try.

A small step forward every day is a new height!

Local LLM in Practice (Part 6): Hermes Agent — a Self-Evolving AI Agent with Memory and Learning

Sat, 09 May 2026 00:00:00 +0000

When we talked about OpenClaw last time, I mentioned that its biggest weakness is its memory system — you have to teach the Agent all over again every time, and it can’t remember your preferences or your project context. This isn’t a problem unique to OpenClaw; it’s a common failing of most AI Agents today: models keep getting stronger, yet every time you chat, the Agent acts like it’s meeting you for the first time. Hermes Agent was built precisely to tackle this problem. Its core selling point isn’t “a stronger AI” but “an AI that understands you better the more you use it”.

1. What Is Hermes Agent

Hermes Agent is developed by the Nous Research team, released in February 2026, and open-sourced under MIT. Nous Research has a solid reputation in the open-source large-model scene — the Hermes series of models they released (fine-tuned from Llama) stand out in tool calling and instruction following. Hermes Agent is their foray into the Agent space, extending “tool-calling ability at the model level” into “a persistent Agent system with memory”.

As of June 2026, Hermes Agent has tens of thousands of GitHub stars. That doesn’t match OpenClaw’s 190,000, but its focus is narrower — memory and self-learning.

2. The Three-Layer Memory Model

The most central design of Hermes Agent is its memory architecture — the “tripartite memory model”:

Procedural memory (SKILL.md): the operational skills the Agent has mastered. Things like “how to create a Git branch”, “how to deploy a service on K8s”, “how to analyze the performance bottleneck of a piece of code”. These are all reusable operational knowledge.

Episodic memory (MEMORY.md): the historical experience of the Agent’s interactions with the user. Things like “last time you refactored UserService because you found its time complexity was O(n²)”, “you prefer ciphertext comparison over direct string comparison for security checks”. These are experiences automatically extracted and distilled from one conversation after another.

Semantic memory (USER.md): the user profile. Things like “you are a backend developer, mainly using Go and Python”, “your code style preference is concise naming and thorough unit tests”, “you don’t like JWT and prefer to use Session”.

The division of labor between the three layers is clear: procedural memory is “I know how to do it”, episodic memory is “I remember what you’ve done”, and semantic memory is “I know who you are”.

3. Hands On: Installation and First Use

Install:

pip install hermes-agent

Initialize (create your personal Agent):

hermes init

This command walks you through configuring a model (OpenAI-compatible interfaces are supported, so you can point it at a local vLLM/Ollama), setting a working directory, and creating the initial SKILL.md / MEMORY.md / USER.md files.

Connect to the local vLLM from the first article:

hermes init --provider openai_compatible \
  --base-url http://your-gpu-server:8000/v1 \
  --model qwen2.5-14b-instruct \
  --api-key not-needed

First interaction:

hermes chat "map out the current project's directory structure and tell me what each module does"

After the interaction ends, check ~/.hermes/MEMORY.md — you should be able to see that the Agent has already automatically extracted the key information from this conversation.

4. Active Learning: Not Just “Remembering” but “Learning on Its Own”

The three-layer memory taxonomy isn’t anything fresh — plenty of systems have a similar layering of memory. The real highlight of Hermes Agent is its active learning mechanism.

Most Agents’ memory is passive: it remembers whatever you tell it. If you don’t tell it, it doesn’t know. OpenClaw works this way — its memory files are maintained by hand.

Hermes Agent does it differently. Every time an interaction with you ends, it asynchronously reviews the conversation in the background, automatically extracting reusable knowledge from it:

对话内容：
你: "帮我把这段数据处理的时间复杂度从 O(n²) 优化到 O(n log n)"
Hermes: [完成优化，使用哈希表替代了嵌套循环]

后台学习：
Hermes 从这次交互中自动提取：
→ added to MEMORY.md: "the user cares about algorithmic performance and asked for O(n log n) complexity in the last data-processing task"
→ if a similar pattern appears repeatedly, it is auto-generated into SKILL.md: "performance optimization preference - prefer hash tables over nested loops"

By the time the next, completely different task comes along, the Agent’s behavior has already been shaped by the earlier experience:

新对话（完全不同的任务）：
你: "帮我写一个数据去重的方法"
Hermes: [生成使用 Set 的 O(n) 方案，而不是简单的 O(n²) 双重循环]
        [because it learned your performance preference from previous experience]

That’s the value of active memory: the Agent doesn’t just answer questions — it comes to understand you better with every interaction.

5. Automatic Skill Creation

The extension of active learning is automatic skill creation. Once you and the Agent have solved a complex problem together, the Agent can distill the whole process into a reusable skill:

You teach the Agent a workflow of “code review → run tests → check coverage → submit a PR”
The Agent automatically packages that workflow into a Skill
Next time you say “review this PR using the same process as last time”, the Agent runs the whole workflow automatically

Hermes Agent’s skills follow the agentskills.io open standard. This isn’t a standard you can only use on Hermes — in theory any Agent framework that supports the standard can load these skills. Skills are shareable across platforms.

6. Other Capabilities

Parallel sub-Agents. Hermes Agent can spawn isolated sub-Agents to handle tasks in parallel. Sub-Agents consume almost none of the shared context window between them, which means you can push several independent tasks forward in parallel without your token consumption rising.

Multi-platform messaging gateway. Supports Telegram, Discord, Slack, WhatsApp, CLI and other platforms. It doesn’t connect as many platforms as OpenClaw’s 50+, but the coverage is already enough for everyday use.

Scheduled automation. A built-in cron scheduler supports unattended tasks such as overnight backups, daily reports and scheduled inspections.

MLOps support. Hermes Agent can generate tool-call traces for model fine-tuning and reinforcement learning training. If you plan to train your own Agent model, Hermes can provide the training data. Most users won’t need this capability for now, but it embodies an idea — “an Agent can improve itself”: once enough data has accumulated, you can use it to train a model that understands you better.

7. Comparison with OpenClaw

The previous article covered OpenClaw in detail; here is a direct comparison:

Dimension	OpenClaw	Hermes Agent
Core positioning	Local Agent gateway (execution)	Self-evolving Agent (memory + learning)
Memory system	Passive (files maintained by hand)	Active (tripartite model + automatic extraction)
Learning ability	No automatic learning	Self-learning loop + automatic skill creation
Skill ecosystem	13,000+ community skills on ClawHub	agentskills.io open standard, skills generated automatically
Messaging platforms	50+	6+ (Telegram/Discord/Slack/WhatsApp/CLI)
Development language	TypeScript (Node.js)	Python
Enterprise investment	Higher (needs a server)	Lower (a $5/month VPS is enough)
Strengths	Broad: many platforms, many Agents, many skills	Deep: strong memory, self-learning, better with use

The two tools are almost complementary in positioning. OpenClaw is strong in “breadth” — many platforms, many Agents, an enterprise-grade gateway. Hermes is strong in “depth” — self-evolving memory, getting smarter with use, low-cost and lightweight.

This isn’t a question of “which is better”. It’s whether you want a gateway that can connect to everything but has an average memory, or an assistant that lives only in the terminal but understands you better and better.

8. Why Memory Is the Agent’s Moat

Finally, I want to expand on a question that’s easy to overlook: why is the memory system the Agent’s real moat?

You might think an Agent’s core competitiveness is model capability — whoever has the stronger model has the better Agent. But model capability is a commodity. Today your Agent uses GPT-5; tomorrow Hermes supports it too, and the day after OpenClaw connects to it as well. The gap at the model level will only get smaller.

But memory doesn’t commoditize. After 100 interactions with your Hermes, it has remembered your preferences, your project structure, your code style, your frequently used commands — and this Hermes becomes irreplaceable to you. Even if another Agent connects to a stronger model, you won’t switch, because switching means teaching an apprentice from scratch all over again.

It’s like a good work partner. After three months on a project together, he knows when you need a code review, what kind of commit message format you like, and which kind of Bug you hate most. Swap him for someone else — maybe more capable — and the rapport has to be rebuilt.

Memory is the rapport between an Agent and its user.

This is also why, although Hermes wasn’t as hot as OpenClaw when it launched, many OpenClaw users end up noticing Hermes after using it heavily — because once they get used to “having an Agent do things for me”, they naturally ask: can this Agent remember everything I’ve done? And Hermes happens to be the one that pushes the single dimension of memory to its extreme.

9. Summary

From OpenCode to Cline to OpenHands to OpenClaw to Hermes Agent, six articles cover the complete chain from “using AI to write code” to “letting AI remember you”:

Tool	Problem it solves
OpenCode	Write code in the terminal without vendor lock-in
Cline	Program autonomously in the IDE with Plan/Act mode
OpenHands	Let AI complete development tasks autonomously in a sandbox
OpenClaw	Let AI do work beyond programming and chain tools together
Hermes Agent	Let AI remember your preferences and get smarter with use

All six tools are open source and all support local deployment. As a DevOps engineer building AI infrastructure on an intranet, you don’t need to adopt every one of them — but you do need to know what problem each one solves and when to use which. My own take: figure out each tool’s positioning first, then choose based on your team’s actual pain points, rather than chasing stars blindly.

A small step forward every day is a new height!

Local LLMs in Practice (Part 5): OpenClaw — A Local Agent Gateway That Gives LLMs "Digital Hands and Feet"

Sat, 02 May 2026 00:00:00 +0000

The previous three articles all covered agents in the programming domain — OpenCode writes code, Cline executes tasks inside the IDE, and OpenHands autonomously completes the whole development workflow. But an LLM’s abilities don’t stop at programming. If you want an LLM to manage files for you, drive a browser, run scheduled inspections of your services, or even respond to your instructions at any moment inside the messaging apps you already use (WeChat / Feishu / DingTalk) — what you need is an agent gateway, not a programming tool. That’s exactly what OpenClaw does.

1. The Story of 190,000 Stars in Three Months

In January 2026, an open-source project called ClawdBot suddenly went viral — it gained 60,000 GitHub stars in 72 hours. It was briefly renamed Moltbot, and finally settled on the name OpenClaw. In three months, its star count passed 190,000.

That pace is rare among open-source projects. Why did it take off? Because it solves a very direct need: it lets your LLM stop just chatting and actually get things done. And it does so running locally, without sending your data to the cloud, and usable straight from WeChat / Feishu / DingTalk.

OpenClaw’s author is Peter Steinberger, an independent developer from Austria (and the founder of PSPDFKit). The project is written in TypeScript (Node.js), and its core design idea is the “agent gateway” — it doesn’t provide AI capability itself, but builds a bridge between the AI models you already have and your digital life.

2. What OpenClaw Is

OpenClaw’s official positioning is “a platform-agnostic, local-first AI agent”. In plain terms: it’s an agent gateway that runs on your own device, can connect to all kinds of LLMs, link up all kinds of tools and platforms, and execute tasks for you automatically.

Three things are key to understanding it:

First, it doesn’t provide AI capability itself. OpenClaw is not a model; it doesn’t train and it doesn’t infer. What it does is connect to your LLM (whether a self-hosted Ollama model or the Claude or GPT API) and then give that model “hands and feet” — the ability to work with the file system, run commands, drive a browser and call APIs.

Second, it’s local-first. All your data lives on your own device and never passes through any third-party server. Deploy OpenClaw on an intranet server, hook it up to a local LLM, and the whole system is completely offline.

Third, it connects to many platforms. OpenClaw supports more than 50 messaging platforms — WeChat, Feishu, DingTalk, Telegram, Discord, WhatsApp, Slack, iMessage, Google Chat… You don’t need to open a dedicated app; you can drive your agent from the chat software you already use.

3. Hands On: Deployment and Configuration

Installation:

curl -fsSL https://openclaw.ai/install.sh | bash

After installation, run the setup wizard, which walks you through configuring your model API key and messaging platforms. You can also edit the configuration file by hand to connect a local Ollama:

# ~/.openclaw/config.yaml example
models:
  - provider: ollama
    model: qwen2.5:14b
    base_url: http://localhost:11434

gateways:
  - platform: telegram
    token: YOUR_BOT_TOKEN
    chat_id: YOUR_CHAT_ID

Your first automated task — in Telegram, tell the agent:

每隔一小时检查 https://example.com 能不能访问，
如果 HTTP 状态码不是 200，就发消息提醒我。

The agent writes a cron job on its own and starts the inspection.

4. Core Capabilities

Giving the model “digital hands and feet”.

With a traditional AI chat tool, you say something and it answers. OpenClaw is different — it’s an agent that lives on your device and can actually do these things:

File system operations: read and write files, organize directories, search content
Browser control: open web pages, take screenshots, scrape content
System command execution: run scripts, manage processes, schedule jobs
Account management: manage API keys, passwords and account information
External API calls: integrate with any HTTP API and chain different systems together

For example, you can send the agent a message in WeChat:

"帮我看看那台服务器的磁盘使用情况，超过 80% 就发告警"

The agent logs into the server over SSH, runs df -h, and if disk usage is above 80% it automatically posts an alert to the group chat. You don’t have to write a single script for any of this — you describe the requirement in natural language and the agent carries it out itself.

A skill marketplace (ClawHub).

OpenClaw has a community skill marketplace called ClawHub, with 13,000+ community-contributed skills. You don’t have to start from scratch — install one with a click and the agent gains a new ability. Weather lookups, stock alerts, GitHub project management, Jira ticket operations… all of these have ready-made skill plugins.

Multi-agent gateway architecture.

OpenClaw can manage several independent agent instances at the same time, each with its own workspace, memory files and configuration. In an enterprise setting, members of different teams can use their own separate agents on different messaging channels without interfering with one another.

The technical architecture has three layers:

Gateway layer: a central router that connects to all kinds of messaging channels and external platforms
Agent Runtime layer: manages multiple independent agent instances
Skill plugin layer: extends the agent’s abilities through the skill ecosystem

Scheduled automation.

It has a built-in cron scheduler. You can have the agent run inspections daily, generate a report weekly or check service status once an hour — completely unattended.

5. What Using It Looks Like

How you use OpenClaw is a bit unusual — it has no traditional “IDE plugin” or “desktop client”. It works like this:

You deploy OpenClaw on a server (your own PC, a Raspberry Pi, an intranet server)
You configure the AI model to connect to (a local Ollama or a cloud API)
You configure the messaging platforms to connect to (WeChat / Feishu / Telegram and so on)
And then you can interact with the agent inside those messaging platforms

What does that feel like? Like having an AI assistant that’s online around the clock. It lives on a server, you can reach it through any chat app, and you can put it to work for you.

6. The Memory System’s Design

Here we have to touch on a topic that’s often used for comparison: OpenClaw’s memory system.

OpenClaw’s memory is file-system based — each agent has a set of SOUL.md, AGENTS.md, USER.md and MEMORY.md files that store state and context. The design is simple and direct, but it comes with limitations:

Memory is passive. You have to hand-write AGENTS.md to tell the agent how it should behave. The agent doesn’t learn lessons from its interactions on its own.
Cross-session context is unstable. A preference you gave the agent last week may be “forgotten” this week — because maintaining the memory files is essentially manual.

This is the point OpenClaw gets criticized for most, and it’s why the Hermes Agent we’ll cover next has been able to rise on its back.

7. What It’s Suited To

Given its characteristics, OpenClaw fits the following scenarios better:

You need an automation assistant that’s online around the clock. Scheduled inspections, anomaly alerts, data synchronization — these don’t need a good memory, just the ability to execute.
You need the same agent across several messaging platforms. Feishu at the office, WeChat when you’re out, Telegram at home — the same agent, reached from different platforms.
You need to chain several tools and systems together. GitHub → Jenkins → Slack → Feishu, with the agent as the glue in the middle.
Data can’t go to the cloud. Local deployment + local model + local execution, fully offline.

Where it doesn’t fit so well: if you want an agent that understands you better the more you use it and remembers your preferences and working habits, OpenClaw’s passive memory model will disappoint you. That’s also why many OpenClaw users end up paying attention to the Hermes Agent.

8. Summary

If you line up the four tools we’ve covered and compare their form factors:

Tool	Interaction form	Positioning
OpenCode	Terminal TUI	A coding assistant in the terminal
Cline	IDE plugin	A coding agent inside the IDE
OpenHands	Web GUI + SDK	An autonomous engineer in a sandbox
OpenClaw	Messaging platforms	A local, all-purpose execution gateway

OpenClaw is not a programming tool; it’s a gateway that gives an LLM “hands and feet”. Its value lies in its breadth — many platforms, many skills, many agents. But its memory system is passive: what you haven’t told it, it won’t remember.

Is there a tool that can make up for that shortcoming — one that’s not just “you do, I say”, but that “learns from interactions on its own”? Next article: the Hermes Agent.

A small step forward every day is a new height!

Tang Moying

GitLab Predefined Variables in Practice (Part 2): The Pipeline "Orchestration Signals" — Trigger Source, Parent-Child Pipelines, Runner and Environment

1. Who Triggered the Pipeline: CI_PIPELINE_SOURCE

2. Scheduled Jobs Need Names Too: CI_PIPELINE_SCHEDULE_DESCRIPTION

3. Passing Context Between Parent and Child Pipelines: CI_PARENT_PIPELINE_ID and CI_UPSTREAM_*

4. Which Machine Am I Running On: CI_RUNNER_* and GITLAB_*

5. Which Environment Did It Reach: CI_ENVIRONMENT_* and Deploy Freezes

6. Parallel Splitting and Job Metadata: CI_JOB_* and CI_NODE_*

Summary

GitLab Predefined Variables in Practice (Part 1): The Pipeline's "Identity" — Branch, Tag, Commit, and MR

1. Deciding the Deployment Target by Branch: CI_COMMIT_BRANCH and Its Friends

2. Tagging Artifacts as “Immutable”: CI_COMMIT_SHA and CI_COMMIT_TAG

3. Distinguishing “an Ordinary Push” from “a Merge Request”: CI_PIPELINE_SOURCE and the MR Variable Family

4. Digging Information Out of Commit Messages: the CI_COMMIT_MESSAGE Family

5. Which Project Am I Running In: the CI_PROJECT_* Family

Summary

MCP vs. CLI: The Two Paths for Agents to Call Tools, and Who Is Eroding Whom?

1. MCP’s Ideal: One Standard to Connect Everything

2. MCP’s Reality: The Token Bill Is Much Steeper Than You Think

3. The Return of the CLI: Why Models “Already Know” the Command Line

4. But the CLI Is Not Perfect Either

5. How to Choose: A Decision Table

6. Summary

An Agent's "Code of Conduct" — the Essence of Skill and the Common Misconceptions

1. What a Skill Is: Custom Processes and Principles

2. “The More Skills the Better” — The Most Common Misconception

2.1 The Context Blows Up

2.2 Skills May Contradict Each Other

2.3 The Agent’s “Difficulty Choosing”

3. How to Write a Good Skill

4. Summary

The Hands and Feet of an Agent: Tool Capability Boundaries and Their Evolution

1. What a Tool Is: Function Calling Isn’t “Calling”, It’s “Translating”

2. The Evolution of Tools: From Reading and Writing Files to Manipulating the World

2.1 The First Generation: Reading and Writing Files

2.2 The Second Generation: Web Requests and API Calls

2.3 Standardizing Tools: How MCP Took Tools From “Built by the Vendor” to “Built Together by the Ecosystem”

2.4 The Third Generation: Computer Use — Letting an Agent Drive the Desktop

2.5 The Fourth Generation: Browser Use — A New Shape for Web Automation

3. How the Four Phases Relate: Not Replacement, but Expansion of Capability

4. More Tools Is Not Better

5. Summary

Hermes Memory Provider and Mem0: An Overlooked Memory Abstraction Layer That Lets Agents Share Knowledge

1. “Wait, Hermes Has This Feature Too?”

2. Inside the Memory Provider Mechanism

2.1 Design Philosophy: Built-in Memory and External Providers Coexist

2.2 The Nine Providers at a Glance

2.3 The Limitation: Only One Provider Can Be Enabled at a Time

3. Mem0 in Depth: Memory Infrastructure for Enterprise Autonomous Agents

3.1 How Mem0 Processes Memory

3.2 Multi-Tenant Isolation: A Native Capability for Enterprise Scenarios

3.3 Self-Hosting: No Data Leaves the Intranet

4. Hands On: Wiring Mem0 into Hermes

4.1 Deploying Mem0

4.2 Configuring Hermes to Use Mem0

4.3 Verifying: Check Whether Memory Is Syncing

4.4 Multi-Agent Verification

5. Why Enterprises Should Care

5.1 The Watershed from Single-Agent to Multi-Agent

5.2 Auditing and Compliance

5.3 Switch Freely, Without Lock-In

6. The Complete Architecture: LiteLLM + Mem0 + Hermes

7. Summary

Hermes Profile — Multiple Agent Personas, One Set of Tools Does It All

1. The Problem: One Hermes Isn’t Enough

2. What Is a Profile

3. Hands On: Creating Your First Persona

3.1 Creating a New Profile

3.2 Switching Profiles

3.3 Viewing All Profiles

3.4 Deleting a Profile

4. Isolation and Sharing Between Profiles

5. Why Profiles Are the Foundation for Multi-Agent Collaboration

6. Summary

LiteLLM: A Unified LLM Gateway — From API Key Chaos to Enterprise AI Infrastructure

1. The Problem: When You Start Treating AI as Infrastructure

2. What LiteLLM Is

3. Hands On: Deploying a LiteLLM Gateway Quickly

3.1 Minimal Working Configuration

3.2 Connecting a Local vLLM