6.5
When the model is upgraded, delete模型升級了,就要刪模型升级了,就要删
This chapter assumes you have used an AI coding agent and that you have written rules or steps for it. Nothing else.
讀這一章,你需要用過 AI 寫程式的 agent,而且替它寫過規則或步驟。不需要別的。
读这一章,你需要用过一个写代码的 AI agent,并且给它写过规则或步骤。别的都不需要。
The Knowledge經驗分享经验分享
In one sentence
When the model is upgraded, rerun your evals first. Then delete the rules, checks and steps that the new model no longer needs, because scaffolding built for an old model may slow the new one.
一句話
模型升級時,先重跑你的評測。然後刪掉新模型不再需要的規則、檢查和步驟,因為為舊模型搭的鷹架,可能會拖慢新模型。
一句话
模型升级了,先重跑你的评测。然后把新模型已经不需要的规则、检查和步骤删掉,因为为旧模型搭的脚手架,可能会拖慢新模型。
For leaders
- The decision: schedule a review of the setup after each model change, and decide who reruns the evals and who may delete a rule.
- The risk: rules and steps built for an old model may slow the new one. I have not measured this on an upgrade; it is a worry, not a result. Hard limits against damage, the record and the evals are not on the list to cut.
- The cost of doing it right: a rerun of your fixed tasks, more than once, plus one person's time to remove one piece at a time. Mostly money and time; I have no numbers.
- Ask the team: "If the model changed tomorrow, which rules would we test removing first, and which evals would tell us?"
給主管
- 要做的決定:每次換模型之後,排一次對設定的檢視,並決定誰重跑評測、誰可以刪規則。
- 風險:為舊模型搭的規則和步驟,可能會拖慢新模型。我沒有在一次升級上量過這件事;它是個擔心,不是結果。防止損害的硬限制、紀錄和評測,不在要砍的清單上。
- 做對的成本:重跑你的固定任務,而且不只一次,再加上一個人的時間,一次移除一塊。大多是錢和時間;我沒有數字。
- 可以問團隊:「如果模型明天換了,我們會先測試拿掉哪些規則?哪些評測能告訴我們答案?」
写给管理者
- 要拍板的事:每次换模型之后,安排一次对配置的复查,并定下谁来重跑评测、谁有权删规则。
- 风险:为旧模型搭的规则和步骤,可能会拖慢新模型。我没有在一次升级上测过这一点;这是担心,不是结果。防止破坏的硬性限制、记录和评测,不在要砍的清单上。
- 做对的代价:把你的固定任务重跑,不止一遍,再加一个人的时间,一次删一块。主要是钱和时间;我没有数字。
- 可以问团队:“如果明天换了模型,我们会先试着删掉哪些规则?哪些评测能告诉我们结果?”
The problem
Think of the setup around an agent as scaffolding, the frame builders put up around a building while they work. Every rule, check and written-out step in it was put there for a reason. Each one covers something the model of that day got wrong, or did badly, or forgot.
Then the model is upgraded. The reasons were about the old model. Some of them are gone. Nobody told your setup.
This is a real worry, though I have not measured it on an upgrade. The first version of my own harness had levels and three job-title agents. The three agents were taken out after a few days of use, on the evidence in Use subagents sparingly, which has the cost numbers and their limits. The levels went for a different reason (What I cut, and what the data says). Both had seemed reasonable on the day. What I take from it is a question, not a result: structure built when the model was weaker may stop earning its cost when the model improves, and it keeps costing until someone tests it and removes it. The rest of this chapter is how to test it.
Dead scaffolding costs in three ways.
- Time and money. Every extra step is paid for at every task.
- Room. A long rule file uses part of the context window on every turn (Context is a budget).
- Confusion. A rule written for an old model's habit can pull a new model the wrong way. I have not tested how often this happens. It is a risk, not a measured effect.
問題
把 agent 周圍的設定,想成鷹架(scaffolding),也就是蓋房子時,工人在建築周圍搭的架子。裡面每一條規則、每個檢查、每一個寫出來的步驟,都是有原因才放進去的。每一個,都在補當時的模型做錯、做不好或忘記的事。
然後模型升級了。那些原因是關於舊模型的。有些已經不在了。沒有人通知你的設定。
這是個真實的擔心,雖然我沒有在一次升級上量過。我自己的 harness 的第一版,有等級,還有三個有職稱的 agent。三個 agent 用了幾天就拿掉了,依據是 慎用子 agent 裡的證據,那一章有成本數字和它們的限制。等級被拿掉的原因不同(我砍掉了什麼,數據怎麼說)。兩者在當時都看起來合理。我從中帶走的是一個問題,不是一個結果:模型比較弱時搭的結構,在模型進步後,可能不再值回它的成本,而且在有人測試、移除它之前,它會一直花錢。這一章其餘的部分,講怎麼測。
沒用的鷹架,花的成本有三種。
- 時間和金錢。每多一步,每個任務都要付一次。
- 空間。一份長的規則檔,每一輪都占用上下文視窗的一部分(上下文是一份預算)。
- 混亂。為舊模型的習慣寫的規則,可能把新模型拉向錯的方向。這有多常發生,我沒有測過。它是風險,不是量到的效果。
问题
把 agent 周围的配置,想成脚手架,也就是工人盖楼时在楼外搭起来的架子。里面的每条规则、每项检查、每一步写明的步骤,当初放进去都有理由。每一个,都是在补当时那个模型做错的、做得差的或忘掉的东西。
然后模型升级了。那些理由说的是旧模型。其中一些已经不成立了。没人通知你的配置。
这是个实实在在的担心,虽然我没有在一次升级上测过。我自己的 harness 的第一版,有分级,还有三个按岗位分工的 agent。用了几天后,那三个 agent 被拿掉了,依据是慎用子 agent里的证据,那一章有成本数字和它们的局限。分级被拿掉,则是另一个原因(我砍掉了什么,数据怎么说)。两样东西在当时都显得合理。我从中得到的是一个问题,不是一个结果:模型还弱的时候搭起来的结构,在模型变强之后,可能不再值它的成本,而且在有人去测它、去掉它之前,它会一直花钱。本章其余部分,讲的就是怎么测。
没用的脚手架,在三个方面花钱。
- 时间和钱。每多一步,每个任务都要为它付一次。
- 空间。一份很长的规则文件,每一轮都占着上下文窗口的一部分(上下文是一份预算)。
- 混乱。为旧模型的某个习惯写的规则,可能把新模型往错的方向拉。这种情况多常发生,我没测过。这是风险,不是测出来的效果。
How it works
The rule is simple: do not delete by feeling, and do not keep by habit. Run the evals. The picture shows the order.
- Over time I added steps, checks, reminders and wording, each to cover something the old model got wrong.
- The model is upgraded. Run the same fixed tasks again, with and without each piece.
- Pieces the evals show are no longer needed can go (dashed). Pieces that still help stay. Hard limits do not depend on the model's skill, so they stay.
1. Before you upgrade, have something to rerun
This is why fixed tasks come first. If you have no set of fixed tasks with a script to score them, an upgrade is a change you cannot judge. Keep the set and its old results. They are the baseline.
2. Rerun with only the model changed
Run the same tasks on the new model with the same setup. Compare with the baseline. Check that the model is the only thing that moved: a new model can tokenize text differently, change the default amount of thinking, and miss the prompt cache at first, and each of these changes cost without changing quality. Record them next to the numbers. If the provider publishes notes on moving from the old model to the new one, read them before the rerun, so you know what to watch. Two things can happen.
- It got worse somewhere. A rule that held with the old model may not hold with the new one. Better to know this first. A model upgrade is a change like any other and can break things.
- It got better, or the same. Then the gap to look at is the cost: time, usage, turns.
Because runs differ (One run proves nothing), run each more than once, or you will read noise as a result.
3. What does not go, whatever the evals say
Some things should not be on the list at all, and it is better to know that before you start removing.
- Hard limits against damage. A refusal to touch production, a guard on the rule files, a person approving what cannot be undone. These do not depend on how clever the model is; they protect against what happens if it is wrong or fooled. A better model is not a reason to remove them (Some things you never hand over).
- The record. A better model still needs a log, a report and a person to read them (Record what happened).
- The evals. They are how you decide everything else.
Layers move at different speeds. Text (rules and steps) changes first and is easiest to cut. Enforcement in code changes more slowly, and by the evals. A platform change sits underneath all of it.
4. Take out one piece at a time
Now ask of each piece of scaffolding: does the new model still need it? You find out by removing it and running the tasks again. One piece per round, or you will not know which one mattered.
For a piece like an independent check, the experiment is direct. Run the tasks with the check on and with it off, and compare. If the evals show no difference, the check is not earning its cost for those tasks (Use subagents sparingly has the same reasoning for the second agent). If it still catches things, keep it.
5. What I would look at first
These are the parts that I expect to be the first candidates in my own setup. I list them as things to test, not as results; I have not rerun them after a model upgrade.
| Part | Why it may change | What I would do |
|---|---|---|
| Written-out playbooks for kinds of work | A newer model may need steps spelled out less | Shorten them. Keep the lists of what counts as done: they say what you want, not how to get it (Manage tasks, not chats) |
| Independent checks | A model may check itself more reliably | Compare no check and one check. Only if the evals show no difference between them does the default become no check |
| Reminders and workarounds | They cover a habit the old model had | Test with them off (Turn your way of working into skills says the same of "gotchas") |
| Rules moved into code because the model ignored them | The new model may follow them | Switch the code rule off where a breach can be undone, and count (A rule in a prompt is a request) |
| Which model each role uses | New models come out | Change the one setting, and run the tasks |
| Wording tuned for the old model | Emphasis written to be heard by one model may be too strong for another | Rewrite it plainly and run the tasks. I have not tested this |
Almost every chapter of this book ends with a section called "When to delete it". That section is the list for this step. After an upgrade I go through those sections in order, and for each one ask whether its condition now holds.
6. Write down what you deleted, and when you would put it back
When I take something out, I write one line: what, why, and what evidence would bring it back. This is what keeps the setup from becoming a guess. When the next model is worse at something, you know which line to look at. (The same habit runs through What I cut, and what the data says.)
Test your own setup
Three questions:
- If the model were upgraded tomorrow, what would you rerun to find out whether your setup still works?
- Which rule or step do you keep only because you added it once?
- Which one would hurt if it were removed, and how do you know?
A test for today. Pick one rule or step in your setup that you suspect is dead weight. Choose three tasks where it should matter. Copy the rule file first, so you can put the rule back. Run the tasks with and without it, more than once, and compare pass and usage. If you cannot tell the two apart, you have learned something. If it does matter, you have learned what the rule is worth.
原理
規則很簡單:不要憑感覺刪,也不要憑習慣留。跑評測。圖顯示順序。
- 隨著時間,我加了步驟、檢查、提醒和措辭,每一項都是為了補舊模型會犯的錯。
- 模型升級了。把同一組固定任務再跑一次,每一項有它和沒有它各跑。
- eval 顯示已經不需要的可以拿掉(虛線)。還有幫助的留下。硬性限制不靠模型的本事,所以留著。
1. 升級之前,先有東西可以重跑
這就是 固定的任務 排在前面的原因。如果你沒有一組固定任務和給它們打分的腳本,升級就是一個你沒辦法評判的改動。把這組任務和它舊的結果留著。它們是基準。
2. 只改模型,重跑
在新模型上,用同樣的設定跑同樣的任務。和基準比較。確認只有模型變了:新模型可能用不同的方式把文字切成 token、改變預設的思考量、一開始又沒命中 prompt 快取,這每一項都會改變成本,卻不改變品質。把它們和數字一起記下來。如果供應商發布了從舊模型換到新模型的說明,在重跑之前先讀,這樣你就知道要留意什麼。有兩種情況可能發生。
- 某處變差了。舊模型下守得住的規則,在新模型下可能守不住。先知道比較好。模型升級和任何其他改動一樣,可能弄壞東西。
- 變好了,或一樣。那麼要看的差距就是成本:時間、用量、輪數。
因為各次運行不同(一次運行證明不了什麼),每個都要跑不只一次,不然你會把雜訊讀成結果。
3. 不管評測怎麼說,都不能拿掉的
有些東西根本不該在清單上,開始移除之前先知道比較好。
- 防止損害的硬限制。拒絕碰正式環境、保護規則檔的防護、由人批准無法復原的事。這些不取決於模型有多聰明;它們保護的,是模型錯了或被騙了會怎樣。更好的模型不是拿掉它們的理由(有些東西根本不給)。
- 紀錄。更好的模型仍然需要日誌、報告,和讀它們的人(記下發生了什麼,不是 agent 說了什麼)。
- 評測。其他一切,都靠它們來決定。
各層的變化速度不同。文字(規則和步驟)最先變,也最容易砍。程式碼裡的執行,變得比較慢,而且要靠評測。平台的改變,在這一切底下。
4. 一次拿掉一塊
現在對每一塊鷹架問:新模型還需要它嗎?你要靠拿掉它、再跑一次任務才知道。一輪一塊,不然你不知道是哪一塊有影響。
對像獨立檢查這樣的一塊,實驗很直接。開著檢查跑任務,關掉檢查再跑,然後比較。如果評測顯示沒有差別,這個檢查在這些任務上就沒有值回它的成本(慎用子 agent 對第二個 agent 有同樣的推理)。如果它還是抓得到東西,就留著。
5. 我會先看什麼
這些是我預期,在我自己的設定裡會最先成為候選的部分。我把它們列成要測試的東西,不是結果;模型升級之後,我還沒有重跑過它們。
| 部分 | 為什麼可能變 | 我會怎麼做 |
|---|---|---|
| 為各種工作寫出來的手冊 | 較新的模型,可能不需要把步驟寫得那麼細 | 縮短它們。留著「什麼才算做完」的清單:它們說的是你要什麼,不是怎麼得到(管任務,不管對話) |
| 獨立檢查 | 模型可能更可靠地檢查自己 | 比較沒有檢查和有一次檢查。只有評測顯示兩者沒有差別,預設才變成不檢查 |
| 提醒和變通辦法 | 它們補的是舊模型的習慣 | 關掉它們來測試(把團隊的做法寫成 skill 對「gotchas」也這麼說) |
| 因為模型不聽,而搬進程式碼的規則 | 新模型可能會遵守 | 在違反了還能復原的地方,把程式碼規則關掉,並計數(提示裡的規則只是請求) |
| 每個角色用哪個模型 | 新模型會出來 | 改那一個設定,然後跑任務 |
| 為舊模型調過的措辭 | 為了讓一個模型聽見而寫的強調,對另一個可能太重 | 改寫得平實一點,然後跑任務。我沒有測過這個 |
這本書幾乎每一章,都以一節「什麼時候可以刪掉」結尾。那一節就是這一步的清單。升級之後,我依序走過這些節,對每一節問它的條件現在成不成立。
6. 寫下你刪了什麼,以及什麼時候會放回去
拿掉東西時,我寫一行:什麼、為什麼、什麼證據會讓它回來。這讓設定不會變成猜測。下一個模型在某件事上比較差時,你知道該看哪一行。(同樣的習慣貫穿 我砍掉了什麼,數據怎麼說。)
測你自己的設定
三個問題:
- 如果模型明天升級,你會重跑什麼,來知道你的設定還行不行?
- 哪一條規則或步驟,你留著只是因為你曾經加過它?
- 哪一個拿掉會痛,你怎麼知道?
今天就能做的測試。挑你設定裡一條你懷疑是廢物的規則或步驟。選三個它應該有影響的任務。先複製規則檔,這樣你能把規則放回去。有它和沒它各跑任務,而且不只一次,比較通過和用量。如果你分不出兩者,你就學到了東西。如果它確實有影響,你就學到了這條規則值多少。
原理
规则很简单:不凭感觉删,也不凭习惯留。跑评测。图里画出了顺序。
- 随着时间,我加了步骤、检查、提醒和措辞,每一项都是为了补旧模型会犯的错。
- 模型升级了。把同一组固定任务再跑一遍,每一项有它和没有它各跑一次。
- 评测显示已经不需要的可以拿掉(虚线)。还有帮助的留下。硬性限制不靠模型的本事,所以留着。
1. 升级之前,先准备好能重跑的东西
这就是固定任务排在前面的原因。如果你没有一组固定任务,也没有给它们打分的脚本,那么升级就是一个你没法评判的改动。把这组任务和它以前的结果留着。它们是基线。
2. 只换模型,重跑
在新模型上,用同样的配置跑同样的任务。和基线比较。确认只有模型变了:新模型对文本的分词方式可能不同,默认的思考量可能变,一开始还可能没命中 prompt 缓存,这些都会改变成本,却不改变质量。把它们和数字记在一起。如果提供方发布了从旧模型迁移到新模型的说明,重跑之前先读,这样你就知道该留意什么。会出现两种情况。
- 某个地方变差了。旧模型下管用的规则,换了新模型可能就不管用了。先知道这一点更好。模型升级和其他改动一样,也可能把东西弄坏。
- 变好了,或者一样。那么要看的差距就是成本:时间、用量、轮数。
因为每次运行都不一样(一次运行证明不了什么),每个都要跑不止一次,否则你会把噪声当成结果。
3. 不管评测怎么说,都不能删的
有些东西,根本不该上清单,开始删之前先清楚这一点更好。
- 防止破坏的硬性限制。拒绝碰生产环境、守护规则文件、不可撤销的事要由人审批。这些不取决于模型有多聪明;它们防的是模型出错或被骗时会发生什么。模型变好了,不是去掉它们的理由(有些东西根本不给)。
- 记录。更好的模型也需要日志、报告,以及读它们的人(记下发生了什么)。
- 评测。其他一切,都靠它来决定。
各层变化的速度不一样。文本(规则和步骤)变得最快,也最容易砍。用代码做的强制执行变得慢一些,而且要靠评测来定。平台的变化在这一切的底下。
4. 一次只拿掉一块
现在对每一块脚手架问:新模型还需要它吗?要知道答案,就拿掉它,再把任务重跑一遍。一轮只动一块,否则你不知道是哪一块起了作用。
对独立检查这类东西,实验是直接的。开着检查和关掉检查各跑一遍任务,然后比较。如果评测显示没有差别,这项检查在这些任务上就不值它的成本(慎用子 agent对第二个 agent 也是同样的道理)。如果它仍然能抓出问题,就留着。
5. 我会先看什么
这几块,是我预期在自己配置里最先被考虑的候选。我把它们列成要测的东西,不是结果;模型升级之后,我还没有重跑过它们。
| 部分 | 为什么可能变 | 我会怎么做 |
|---|---|---|
| 针对各类工作写明的步骤手册 | 更新的模型可能不需要把步骤写得那么细 | 把它们缩短。保留“什么算做完”的清单:它们说的是你要什么,不是怎么得到(管任务,不管对话) |
| 独立检查 | 模型可能更可靠地检查自己 | 比较不检查和检查一次。只有当评测显示两者没有差别,默认才变成不检查 |
| 提醒和变通办法 | 它们补的是旧模型的某个习惯 | 关掉它们再测(把团队的做法写成 skill对“坑”也是这么说的) |
| 因为模型不听而挪进代码的规则 | 新模型可能会遵守 | 在违反了也能撤销的地方,把代码里的规则关掉,并计数(提示里的规则只是请求) |
| 每个角色用哪个模型 | 新模型会不断出来 | 只改这一项设置,再跑任务 |
| 为旧模型调过的措辞 | 为了让某个模型听见而写的强调,对另一个模型可能太重 | 用平实的话重写,再跑任务。我没有测过这一点 |
本书几乎每一章的结尾,都有一节叫“什么时候可以删掉”。这一节就是这一步要用的清单。升级之后,我按顺序过一遍这些节,对每一节问它的条件现在成不成立。
6. 写下你删了什么,什么时候会放回去
我拿掉一样东西时,会写一行:什么、为什么,以及什么证据会让它回来。这能防止配置变成瞎猜。下一个模型在某件事上更差时,你就知道该看哪一行。(同样的习惯贯穿我砍掉了什么,数据怎么说。)
测你自己的配置
三个问题:
- 如果明天模型升级了,你会重跑什么,来看你的配置还管不管用?
- 哪条规则或哪一步,你留着只是因为曾经加过?
- 哪一个删了会出问题?你怎么知道?
今天就能做的一个测试。挑你配置里一条你怀疑是累赘的规则或步骤。选三个它应该起作用的任务。先把规则文件复制一份,这样你能把规则放回去。有它和没它各跑一遍任务,不止一次,比较通过情况和用量。如果你分不出两者的差别,你就学到了东西。如果它确实有影响,你就知道了这条规则值多少。
Costs and traps
Removing something is a change, and it can break things. Remove one piece at a time, and run the evals after each.
The evals are the limit. You can only say that a piece is not needed for the tasks you tested. If the tasks are few, say so and be careful with anything that guards against damage.
A rerun costs money. A full rerun after every model release is not always affordable. Rerun when you change model family, when the version number changes, or when you notice a change in behaviour. The cache matters here: cached reads cost about a tenth of the base input price on Anthropic's API (Models and cost), but the first runs on a new model start cold, and I have not checked how Copilot bills any of this.
New models can be worse at something. An upgrade is not always better at every task. The evals are for this too.
Scaffolding can hide a weak spot. A check that always caught the same mistake may be the thing that kept you from noticing it. When you remove it, look at the failed runs.
代價和陷阱
拿掉東西也是改動,也可能弄壞東西。一次拿掉一塊,每次之後都跑評測。
評測就是極限。你只能說,某一塊在你測過的任務上不需要。如果任務很少,就說出來,並且對任何防止損害的東西要小心。
重跑要花錢。每次模型發布之後都完整重跑,不一定負擔得起。換模型系列、版本號變了,或你注意到行為變了的時候再重跑。快取在這裡有影響:在 Anthropic 的 API 上,快取讀取大約是基本輸入價格的十分之一(選模型:算的是每次成功的成本),但新模型上最先的幾次運行是冷啟動,而這一切 Copilot 怎麼計費,我沒查過。
新模型可能在某件事上比較差。升級不一定在每個任務上都更好。評測也是為了這個。
鷹架可能藏著弱點。一個總是抓到同一種錯誤的檢查,可能正是讓你沒注意到那種錯誤的東西。拿掉它的時候,去看失敗的運行。
代价和坑
拿掉东西也是一种改动,也可能把东西弄坏。一次只拿掉一块,每拿掉一块就跑一次评测。
评测就是边界。你只能说某一块对你测过的那些任务不需要。如果任务很少,就这么说,并且对任何防止破坏的东西要格外小心。
重跑要花钱。每次发布新模型都完整重跑一遍,不一定负担得起。换了模型系列、版本号变了,或者你注意到行为变了,就重跑。缓存在这里有影响:在 Anthropic 的 API 上,命中缓存的读取约是基础输入价格的十分之一(选模型:看每次成功的成本),但新模型上最初几次运行是冷启动,我也没有查过 Copilot 对这些怎么计费。
新模型可能在某件事上更差。升级不一定在每个任务上都更好。评测也是为这个准备的。
脚手架可能掩盖弱点。一项总能抓住同一个错误的检查,可能正是让你没注意到这个错误的东西。拿掉它的时候,去看失败的运行。
When to delete it Industry practice
This chapter is itself a procedure for deleting, so what is there to delete? The step-by-step rerun. If a platform tells you, on your own tasks, what an upgrade changed, or runs your evals on each release, I would use that. The habit stays: nothing in the setup is permanent, and every piece has to earn its place against the model you have now.
什麼時候可以刪掉 業界做法
這一章本身就是一套刪東西的程序,那有什麼可刪的?一步一步重跑的做法。如果平台在你自己的任務上告訴你升級改變了什麼,或在每次發布時替你跑評測,我會用那個。習慣會留著:設定裡沒有東西是永久的,每一塊都要對照你現在有的模型,證明自己該在那裡。
什么时候可以删掉 业界做法
这一章本身就是一套删东西的流程,那还有什么可删的?就是一步一步重跑的那套做法。如果平台能在你自己的任务上告诉你一次升级改变了什么,或者每次发布都替你跑评测,我会用它。习惯留下来:配置里没有什么是永久的,每一块都得对照你现在用的这个模型,证明自己该留下。
Try it on your own setup
No harness needed. Open your agent in your own repository and paste:
List every rule, step and check in this repository's agent instruction, skill and hook files. Sort each into one of: hard limit against damage, record or log, workaround for a habit of an older model, written-out playbook, or emphasis in the wording. For each workaround, playbook or emphasis, say from git history what failure it was added for, or "no reason found". Then name the five I should test removing first after a model upgrade, and which of my tasks should be rerun to check. Do not change any files.在你自己的環境試試
不用裝 harness。在你自己的倉庫裡打開 agent,貼上:
列出這個倉庫的 agent 指令檔、skill 和 hook 檔案裡,每一條規則、步驟和檢查。把每一個分到其中一類:防止損害的硬限制、紀錄或日誌、補舊模型習慣的變通辦法、寫出來的手冊、措辭裡的強調。對每個變通辦法、手冊或強調,從 git 歷史說明它是為了什麼失敗而加的,或寫「找不到原因」。然後指出,模型升級之後,我該最先測試拿掉的五個,以及我該重跑哪些任務來檢查。不要改任何檔案。在你自己的环境里试试
不需要 harness。在你自己的仓库里打开 agent,粘贴:
列出这个仓库的 agent 指令、skill 和 hook 文件里的每一条规则、步骤和检查。把每一项归入以下之一:防止破坏的硬性限制、记录或日志、针对旧模型某个习惯的变通办法、写明的步骤手册、措辞里的强调。对每个变通办法、步骤手册或强调,根据 git 历史说明它是为了什么失败而加的,或者写“没找到原因”。然后指出,模型升级后我应该先试着删哪五项,以及该重跑我的哪些任务来检查。不要修改任何文件。Go deeper
延伸閱讀
延伸阅读
You can stop here: that is the whole idea. The next part shows where it runs, so you can look inside.讀到這裡就夠了,這就是整個想法。下一部分告訴你它在哪裡運行,可以進去看看。读到这里就够了,核心思路就是这些。下一部分告诉你它在哪里跑,可以进去看看。
In my practice (Copilot Harness)實作实战
Copilot Harness is my open-source harness for VS Code Copilot.
Repository: https://github.com/Smars-Bin-Hu/copilot-agentic-harness
Get started: clone it, open it in VS Code, and type /harness-rookie-get-me-ready in Copilot Chat. You need Python 3.9 or later and git.
Copilot Harness 是我開源的 VS Code Copilot harness。
倉庫: https://github.com/Smars-Bin-Hu/copilot-agentic-harness
開始:clone 下來,用 VS Code 打開,在 Copilot Chat 裡輸入 /harness-rookie-get-me-ready。需要 Python 3.9 以上和 git。
Copilot Harness 是我开源的 VS Code Copilot harness。
仓库: https://github.com/Smars-Bin-Hu/copilot-agentic-harness
上手:把仓库 clone 下来,用 VS Code 打开,在 Copilot Chat 里输入 /harness-rookie-get-me-ready。需要 Python 3.9 及以上版本和 git。
Explore it in the harness
Copilot Harness keeps a list of what it expects to change first after a model upgrade, and a list of what it chose not to build and when it would reconsider. The harness can walk you through both. After the setup above, type:
/harness-rookie-wondering I have read the chapter "When the model is upgraded, delete". Teach me what this harness expects to change after a model upgrade (topic 15) and what it will never remove. For each part, open the real files with me and show how I would test whether it is still needed.It teaches one small piece at a time, stops to ask if you are following, and opens the files with you.
在 harness 裡探索
Copilot Harness 有一份清單,列出它預期模型升級後最先會變的東西,另有一份清單,列出它選擇不做的東西,以及什麼時候會重新考慮。harness 可以帶你走過這兩份。照上面的說明準備好之後,輸入:
/harness-rookie-wondering 我讀完了「模型升級了,就要刪」這一章。請教我這個 harness 預期模型升級後哪些東西會變(主題 15),以及哪些東西它永遠不會拿掉。對每一部分,帶我打開真實的檔案,並示範我該怎麼測試它是不是還需要。它一次教一小塊,停下來確認你跟得上,然後和你一起打開檔案。
在 harness 里探索
Copilot Harness 有一份清单,列着它预期在模型升级后最先会变的东西,还有一份清单,列着它选择不做的东西,以及什么时候会重新考虑。harness 能带你把这两份都过一遍。完成上面的配置后,输入:
/harness-rookie-wondering 我读过“模型升级了,就要删”这一章。请教我这个 harness 预期在模型升级后哪些东西会变(主题 15),以及它永远不会删什么。对每一部分,和我一起打开真实的文件,给我看怎么测试它是否还需要。它一次只教一小块,会停下来问你跟不跟得上,并和你一起打开文件。