> Harness EngineeringHarness EngineeringHarness Engineering The engineering outside the model模型之外的工程模型之外的工程

8.3

Is the map right? A blind audit地圖對不對:盲抽查地图对不对:盲抽查

This chapter follows The toolkit. It is step three of the road. The ideas of recall and precision are explained in plain words here; you need no statistics. The method is not tied to code maps: it fits any list a model or a script produces that you can check against a source.

這一章接在 Toolkit:用腳本畫出程式碼的地圖 後面。它是路線的第三步。召回率和精確率的意思,這裡用白話解釋;你不需要懂統計。這個方法不限於程式碼地圖:任何由模型或腳本產生、而你能拿原始資料來檢查的清單,都適用。

这一章接在Toolkit:用脚本画出代码的地图后面,是路线的第三步。召回率和精确率的概念,这里用大白话讲,不需要统计学。这个方法不限于代码地图:凡是模型或脚本产出、又能拿来源核对的清单,都适用。

The Knowledge經驗分享经验分享

In one sentence

No map of code is complete, but you can measure how accurate it is and where it is blind. Take a sample of objects, have an agent read them blind, check every difference at its line, and compare with the map.

一句話

沒有程式碼的地圖是完整的,但你可以量它有多準,以及它在哪裡看不見。抽一批物件的樣本,讓 agent 盲讀,在每一處差異的那一行去檢查,再和地圖比較。

一句话

没有哪张代码地图是完整的,但你可以衡量它有多准、盲区在哪。抽一个对象样本,让 agent 盲读,每一处差异都回到对应的行去核对,再和地图比较。

For leaders

  • The decision: require that the map of the code is measured before anyone learns from it, and set the pass bar. I use 0.95, a line I chose and did not derive.
  • The risk: a map can find the ten uses you knew and still miss a third of the rest, and every note built on it inherits the hole. A pass on a sample of 20 to 40 objects is a floor for starting, not a proof.
  • The cost of doing it right: an agent reads 20 to 40 sampled objects, once per toolkit change, and a person or script checks every difference at its line.
  • Ask the team: "How many uses does our map miss or invent, and how was that measured by a reader that never saw the map?"

給主管

  • 要做的決定:要求在任何人拿程式碼地圖去學之前,先把地圖量過,並定下合格線。我用 0.95,這是我選的線,不是推導出來的。
  • 風險:地圖可以找到你知道的那十個使用,卻漏掉其餘的三分之一,而建在它上面的每份筆記,都繼承這個洞。在 20 到 40 個物件的樣本上通過,是開始的底線,不是證明。
  • 做對的成本:每次 toolkit 改動,一個 agent 讀 20 到 40 個抽樣的物件,一次;再由人或腳本在每一處差異的那一行去檢查。
  • 可以問團隊:「我們的地圖漏掉或編造了多少使用?這是被一個從沒看過地圖的讀者怎麼量出來的?」

写给管理者

  • 要拍板的事:要求在任何人拿地图去学之前,先衡量这张代码地图,并定下及格线。我用 0.95,这条线是我选的,不是推导出来的。
  • 风险:一张地图可能找到了你知道的那十处使用,却漏掉其余的三分之一,而建在它上面的每条笔记都继承这个洞。20 到 40 个对象的样本通过,只是开始的底线,不是证明。
  • 做对的代价:每次 toolkit 改动,由一个 agent 读 20 到 40 个抽样的对象,再由人或脚本把每处差异回到对应的行去核对。
  • 可以问团队:“我们的地图漏了或编造了多少处使用?这个数是怎么测的,测的时候那个读者看过地图吗?”

The problem

You built a map (the toolkit). The five or ten known uses are all found. Can you trust it?

Not yet. Those known uses prove the toolkit can find a use. They say nothing about how many it misses, or whether some of what it found is not there. A map that finds the 10 uses you knew and misses a third of the rest looks perfect on that test.

If you learn from a map that misses a third of the uses, every card built on it inherits the hole. So measure first. The audit is the measuring.

問題

你建了一張地圖(toolkit)。五、六個已知的使用全找到了。你能信它嗎?

還不能。那些已知的使用只證明 toolkit 找得到一個使用。它們沒說漏掉多少,也沒說找到的有沒有不存在的。一張找到你知道的那 10 個使用、卻漏掉其餘三分之一的地圖,在那個測試上看起來是完美的。

如果你從一張漏掉三分之一使用的地圖去學,建在它上面的每張卡片都繼承這個洞。所以先量。抽查就是在量。

问题

你建好了一张地图(toolkit)。五处或十处已知的使用都找到了。你能信它吗?

还不能。那些已知的使用只能证明 toolkit 能找到使用。它们说明不了漏掉多少,也说明不了找到的东西里有没有不存在的。一张地图找到了你知道的那 10 处使用,却漏了其余的三分之一,在这个测试上看起来完美无缺。

如果你从一张漏了三分之一使用的地图去学,建在它上面的每张卡片都继承这个洞。所以先衡量。审计就是这个衡量。

How it works

The idea: ask an independent reader, then compare

The map is one opinion about what the code says. Get a second opinion that does not copy the first. An agent reads the code of some objects and writes down every use it sees. Then we compare the two lists.

For that second opinion to be worth anything, it must be blind. The agent writes down every use it finds before it looks at the map. A reading made after looking at the map copies the map. The two lists then agree because one was copied from the other, and the audit measures nothing. The same reasoning is behind an independent reviewer (a model checking a model): the checker must not see the thing it is checking.

What gets sampled

A script picks the sample: a few objects of each type (for example 5 per type, so about 20 to 40 in all). It hands the agent only the file and line range of each object, not its uses. The sample is picked at random with a seed (a number that fixes the random choice), so a second audit with another seed picks different objects.

Reading blind

For each object the agent reads its lines and writes one record per use: what it uses, which kind of use, and the line number. If an object uses nothing, it writes "none". Two special cases:

  • A name built at run time: the agent writes what is fixed and marks it dynamic. It is not scored.
  • A short name that could mean several objects: the agent writes it in full, with the database or package it resolves to. Otherwise the score cannot tell if the map picked the right one.

Recall and precision, in plain words

Think of a library with a card catalogue. The shelves are the code. The catalogue is the map.

  • Recall asks: of the books on the shelves, how many are in the catalogue? Low recall means the map missed things.
  • Precision asks: of the entries in the catalogue, how many are really on the shelves? Low precision means the map made things up.

Each is a fraction. An example to show the arithmetic, not a result: suppose the code has 20 uses in the sample, and the map lists 22 entries: 19 of the 20, plus 3 that are not real. Recall is 19 out of 20, which is 0.95. Precision is 19 out of 22, which is about 0.86. The "20" is not known for certain either: it is the blind reader's list after every difference was checked (next section), so these numbers are estimates of the truth.

only the reader map missed it both have it agreement only the map in the code? recall: share of found uses on the map the reader's list = left + middle precision: share of map uses that are real
Two lists of uses for the same objects. Every difference is looked up again at its line, because the reader can be wrong too.

You need both. A map can have perfect recall by listing everything, true or not. It can have perfect precision by listing three safe uses. Only the pair tells you the truth. And they are scored per type, because a toolkit can be good at one kind of object and poor at another, and an average hides that.

The reader can be wrong too

This is the part I want to be plain about. The blind reader is a model. It can misread, miss a use, or invent one. If we scored raw differences, we would blame the map for the reader's mistakes.

So after scoring, every difference is checked again at its line:

  • In the map but not in the reader's list. Look in the file. If the use is there, the reader missed it: add it to the reader's list and write a note. If it is not there, it is a false use in the map.
  • In the reader's list but not in the map. Look at the line again. If the reader was wrong, correct the list and write a note. If the use is there, it is a real miss of the map.

Then score again. Only what is left counts against the map. The notes about the reader's own mistakes stay in the report. They tell you how far a reading can be trusted, which is itself useful.

The bar, and what to do with the differences

The bar I use is 0.95 for both recall and precision, in every type. This is a line I chose; I did not derive it. Below it, the audit fails for that type. The line should depend on what a mistake costs: a missed use leaves a card without a relation, a made-up use adds a relation that is not there, and an object on shared data deserves a higher bar. I did not weigh recall and precision differently. With 20 to 40 objects, 0.95 and 0.90 cannot be told apart; the audit shows large gaps, not small ones.

A failure is not bad news. The differences are a list of real examples of what the toolkit gets wrong, each with a file and a line. Look for what they have in common: one form of call, one kind of file. Those are the examples to fix the extractor with. Then audit again with another seed, so you do not tune the toolkit to one lucky sample.

What no map sees

Some uses are out of reach of any static map, however good:

  • SQL built as a string at run time and then executed;
  • calls driven by a row in a configuration table;
  • a package or class chosen by a variable;
  • synonyms and linked servers the toolkit was not taught.

These are marked, not guessed. A name that resolves to something outside the repository is marked outside; one that is built at run time is marked dynamic. Each becomes an entry in the requests list (8.5), so that a person can answer it with a read-only query against the real system.

What the report looks like

The audit ends in one report: the pass or fail, the score per type, every miss and false use with its file and line, the patterns, and the readings that were corrected.

The audit report template: result, score per type, misses, false uses, patterns, corrected readings, what was not counted.
# Map audit

Date: <YYYY-MM-DD> · Map at: <repo>@<short commit>, ... · Toolkit: <name> (signature <signature from `toolkits` of kb audit>) · Sample: `kb audit --per-type <n> --seed <s>` · Bar: <0.95>

Result: **pass** | **fail**

## Score per type

| Type | Objects | Uses read | Uses in the map | Both | Recall | Precision | Pass |
| --- | --- | --- | --- | --- | --- | --- | --- |

## The map's misses (in the code, not in the map)

| Object | Use | Path:line | Why the toolkit misses it |
| --- | --- | --- | --- |

## The map's false uses (in the map, not in the code)

| Object | Use | Path | Why the toolkit sees it |
| --- | --- | --- | --- |

## Patterns

- <one line per form the toolkit gets wrong, with how common it looks in the code>

## Names the map could not place (`unresolved` of `kb map`, most used)

- <name: used by n; what it is>

## Readings corrected after checking the line

- <object, use: what was misread. Or "none">

## Not counted

- Dynamic code: <ids>

Test your own setup

  1. Take your map. Pick 5 objects of each type at random.
  2. In a fresh chat, without showing the map, ask an agent to list every use in each, with line numbers.
  3. Compare. Check each difference at its line.
  4. Compute recall and precision per type, as above.
  5. Repeat with another random sample and see if the numbers move.

原理

想法:問一個獨立的讀者,再比較

地圖是對程式碼說了什麼的一個意見。要拿到不抄第一個的第二個意見。一個 agent 讀某些物件的程式碼,寫下它看到的每個使用。然後我們比較兩份清單。

這第二個意見要有價值,它必須是盲的。agent 在看地圖之前,就寫下它找到的每個使用。看過地圖之後做的解讀,會抄地圖。兩份清單一致,是因為一份抄自另一份,抽查就什麼也量不到。獨立 reviewer 背後也是同樣的道理(模型能檢查模型嗎):檢查者不能看到它檢查的東西。

抽什麼樣本

一個腳本挑出樣本:每種類型挑幾個物件(例如每種 5 個,所以全部約 20 到 40 個)。它只把每個物件的檔案和行的範圍交給 agent,不給它的使用。樣本用一個種子(固定隨機選擇的一個數字)隨機挑,所以用另一個種子的第二次抽查,會挑出不同的物件。

盲讀

對每個物件,agent 讀它的那幾行,每個使用寫一筆紀錄:它用了什麼、哪一種使用,以及行號。如果一個物件什麼都沒用,就寫「none」。兩個特殊情況:

  • 執行時才組出來的名字:agent 寫下固定的部分,並標成動態。它不計分。
  • 可能指幾個物件的短名字:agent 寫完整,附上它解析到的資料庫或套件。不然計分的時候,看不出地圖有沒有選對。

召回率和精確率,用白話說

想像一個有卡片目錄的圖書館。書架是程式碼。目錄是地圖。

  • 召回率問:書架上的書,有幾本在目錄裡?召回率低,表示地圖漏掉了東西。
  • 精確率問:目錄裡的條目,有幾條真的在書架上?精確率低,表示地圖編造了東西。

兩者都是分數。舉個例子來看算術,不是結果:假設樣本裡的程式碼有 20 個使用,地圖列了 22 條:20 個裡的 19 個,加上 3 個不是真的。召回率是 20 個裡的 19 個,也就是 0.95。精確率是 22 條裡的 19 條,大約 0.86。這個「20」也不是確定知道的:它是盲讀者的清單,在每一處差異都檢查過之後(下一節),所以這些數字是對真相的估計。

只有盲讀的人有 地圖漏了 兩邊都有 吻合 只有地圖有 程式碼裡有嗎? 召回:讀的人找到的「用」,地圖有多少 讀的人的清單 = 左 + 中 精度:地圖裡的「用」,有多少是真的
同一批物件的兩份「用」清單。每個差異都回到原行再看一次,因為讀的人也可能錯。

兩個都需要。一張地圖把所有東西都列出來,不管真假,召回率就可以是滿分。它只列三個保險的使用,精確率就可以是滿分。只有兩個放在一起,才說出真相。而且它們是每種類型分開計分的,因為 toolkit 可能對一種物件很好、對另一種很差,平均會把這點藏起來。

讀者也可能錯

這是我想說清楚的部分。盲讀者是一個模型。它可能讀錯、漏掉一個使用,或編造一個。如果我們對原始的差異計分,我們就會把讀者的錯怪到地圖頭上。

所以計分之後,每一處差異都在它的那一行再檢查一次:

  • 在地圖裡,但不在讀者的清單裡。去檔案裡看。如果使用在那裡,是讀者漏掉了:把它加進讀者的清單,並寫一則備註。如果不在那裡,它就是地圖裡一個假的使用。
  • 在讀者的清單裡,但不在地圖裡。再看一次那一行。如果讀者錯了,修正清單並寫一則備註。如果使用在那裡,那就是地圖真正漏掉的。

然後再計一次分。只有剩下的,才算在地圖頭上。關於讀者自己犯的錯的備註,留在報告裡。它們告訴你一份解讀能信到什麼程度,這本身就有用。

合格線,以及差異怎麼處理

我用的線,是每種類型的召回率和精確率都 0.95。這是我選的線;不是推導出來的。低於它,那種類型的抽查就失敗。這條線應該取決於出錯的代價:漏掉的使用,讓一張卡片少了一個關係;編造的使用,多加了一個不存在的關係;碰到共用資料的物件,值得更高的線。我沒有給召回率和精確率不同的權重。樣本是 20 到 40 個物件時,0.95 和 0.90 分不出來;抽查看得出大的差距,看不出小的。

失敗不是壞消息。這些差異是一份真實例子的清單,說明 toolkit 哪裡做錯了,每個都有檔案和行號。找出它們的共同點:一種呼叫的寫法、一種檔案。這些就是修抽取器用的例子。然後用另一個種子再抽查一次,這樣你就不會只對一個幸運的樣本調校 toolkit。

沒有地圖看得到的

有些使用,任何靜態地圖都碰不到,不管它多好:

  • 執行時組成字串、再執行的 SQL;
  • 由設定表裡的一列驅動的呼叫;
  • 由變數選擇的套件或類別;
  • toolkit 沒被教過的同義詞(synonym)和連結的伺服器。

這些要標出來,不要猜。解析到倉庫之外的名字,標成外部;執行時才組出來的,標成動態。每一個都變成請求清單裡的一個條目(8.5),讓人可以用對真實系統的唯讀查詢來回答它。

報告長什麼樣

抽查以一份報告結束:通過或失敗、每種類型的分數、每個漏掉和假的使用及其檔案和行號、規律,以及被修正過的解讀。

抽查報告的範本:結果、每種類型的分數、漏掉的、假的使用、規律、被修正的解讀,以及沒有計入的部分。
# Map audit

Date: <YYYY-MM-DD> · Map at: <repo>@<short commit>, ... · Toolkit: <name> (signature <signature from `toolkits` of kb audit>) · Sample: `kb audit --per-type <n> --seed <s>` · Bar: <0.95>

Result: **pass** | **fail**

## Score per type

| Type | Objects | Uses read | Uses in the map | Both | Recall | Precision | Pass |
| --- | --- | --- | --- | --- | --- | --- | --- |

## The map's misses (in the code, not in the map)

| Object | Use | Path:line | Why the toolkit misses it |
| --- | --- | --- | --- |

## The map's false uses (in the map, not in the code)

| Object | Use | Path | Why the toolkit sees it |
| --- | --- | --- | --- |

## Patterns

- <one line per form the toolkit gets wrong, with how common it looks in the code>

## Names the map could not place (`unresolved` of `kb map`, most used)

- <name: used by n; what it is>

## Readings corrected after checking the line

- <object, use: what was misread. Or "none">

## Not counted

- Dynamic code: <ids>

測你自己的設定

  1. 拿你的地圖。每種類型隨機挑 5 個物件。
  2. 在全新的對話裡,不給它看地圖,請 agent 列出每個物件裡的每個使用,附上行號。
  3. 比較。每一處差異都在它的那一行檢查。
  4. 如上面那樣,每種類型計算召回率和精確率。
  5. 用另一個隨機樣本再做一次,看數字會不會動。

原理

思路:请一个独立的读者,再比较

地图是对代码内容的一种看法。再取一个不抄第一个的看法。一个 agent 读几个对象的代码,写下它看到的每一处使用。然后我们比较这两份清单。

这第二个看法要有价值,就必须是盲的。agent 要在看地图之前,写下它找到的每一处使用。看了地图之后做的阅读,会抄地图。两份清单一致,是因为一份抄了另一份,审计就什么也没衡量。独立 reviewer 背后是同样的道理(模型能检查模型吗):检查者不能看到它要检查的东西。

抽什么样本

由脚本来抽样:每种类型抽几个对象(比如每种 5 个,一共约 20 到 40 个)。它只把每个对象的文件和行范围交给 agent,不给它使用关系。样本是用种子(固定随机选择的一个数)随机抽的,所以换一个种子做第二次审计,抽到的对象不同。

盲读

对每个对象,agent 读它的那几行,每处使用写一条记录:用到了什么、哪一类使用、行号。如果一个对象什么都没用到,就写“none”。有两种特殊情况:

  • 运行时拼出来的名字:agent 把固定的部分写下来,标成动态。它不计分。
  • 可能指向多个对象的短名字:agent 要写全,带上它解析到的数据库或包。否则,评分时就分不清地图选的对不对。

召回率和精确率,用大白话讲

想象一座有卡片目录的图书馆。书架是代码。目录是地图。

  • 召回率问的是:书架上的书,有多少在目录里?召回率低,说明地图漏了东西。
  • 精确率问的是:目录里的条目,有多少真的在书架上?精确率低,说明地图编造了东西。

两个都是分数。举个例子来演示算术,它不是结果:假设样本里的代码有 20 处使用,地图列了 22 条:20 处里的 19 处,加上 3 条不真实的。召回率是 20 里的 19,也就是 0.95。精确率是 22 里的 19,约 0.86。这个“20”也不是确切知道的:它是盲读者的清单在每一处差异核对完之后的结果(见下一节),所以这些数字都是对真实情况的估计。

只有盲读者有 地图漏了 两边都有 吻合 只有地图有 代码里有吗? 召回率:读者找到的使用,地图有多少 读者的清单 = 左 + 中 精确率:地图里的使用,有多少是真的
同一批对象的两份使用清单。每处差异都回到对应的行再看一次,因为读者也可能错。

两个你都需要。一张地图把什么都列上,召回率就可以是满分,不管真假。它只列三处保险的使用,精确率就可以是满分。只有这一对数字合起来,才说明真实情况。而且它们要按类型分别计分,因为一个 toolkit 可能对一种对象做得好、对另一种做得差,取平均会把这一点藏起来。

读者也可能错

这一段我想说得直白一些。盲读者是一个模型。它可能读错、漏掉一处使用,或者编造一处。如果我们按原始差异计分,就会把读者的错误算到地图头上。

所以计分之后,每一处差异都回到对应的行再核对一次:

  • 在地图里,但不在读者的清单里。去文件里看。如果那处使用确实在,就是读者漏了:把它加进读者的清单,并写一条笔记。如果不在,那就是地图里的一处虚假使用。
  • 在读者的清单里,但不在地图里。再看一遍那一行。如果读者错了,改正清单并写一条笔记。如果那处使用确实在,那就是地图真的漏了。

然后重新计分。只有剩下的才算在地图头上。关于读者自己的错误的那些笔记留在报告里。它们告诉你一次阅读能信到什么程度,这本身就有用。

及格线,以及差异怎么处理

我用的及格线是:每种类型的召回率和精确率都要 0.95。这条线是我选的,不是推导出来的。低于它,这种类型的审计就不通过。线该定在哪,要看出错的代价:漏掉一处使用,会让卡片少一条关系;编造一处使用,会多出一条不存在的关系;碰到共用数据的对象,值得更高的线。我没有对召回率和精确率区别对待。只有 20 到 40 个对象时,0.95 和 0.90 分不出来;审计显示的是大的差距,不是小的。

不通过不是坏消息。这些差异就是 toolkit 出错的真实例子的清单,每条都带文件和行号。找找它们的共同点:同一种调用写法,同一种文件。这些就是修抽取器用的例子。修完再换一个种子审计一次,免得你把 toolkit 调得只适应某一个碰巧的样本。

任何地图都看不到的

有些使用,任何静态地图都够不着,不管它多好:

  • 运行时拼成字符串再执行的 SQL;
  • 由配置表中的一行驱动的调用;
  • 由变量选择的包或类;
  • toolkit 没教过的同义词和链接服务器。

这些会被标出来,不去猜。解析到仓库之外的名字,标成外部;运行时拼出来的,标成动态。每一个都成为请求清单(8.5)里的一条,让人可以用针对真实系统的只读查询来回答它。

报告长什么样

审计以一份报告收尾:通过还是不通过、每种类型的分数、每一处漏掉的和虚假的使用(带文件和行号)、各种规律,以及被纠正的阅读。

审计报告模板:结果、每种类型的分数、漏掉的、虚假的、规律、被纠正的阅读、没有计入的部分。
# Map audit

Date: <YYYY-MM-DD> · Map at: <repo>@<short commit>, ... · Toolkit: <name> (signature <signature from `toolkits` of kb audit>) · Sample: `kb audit --per-type <n> --seed <s>` · Bar: <0.95>

Result: **pass** | **fail**

## Score per type

| Type | Objects | Uses read | Uses in the map | Both | Recall | Precision | Pass |
| --- | --- | --- | --- | --- | --- | --- | --- |

## The map's misses (in the code, not in the map)

| Object | Use | Path:line | Why the toolkit misses it |
| --- | --- | --- | --- |

## The map's false uses (in the map, not in the code)

| Object | Use | Path | Why the toolkit sees it |
| --- | --- | --- | --- |

## Patterns

- <one line per form the toolkit gets wrong, with how common it looks in the code>

## Names the map could not place (`unresolved` of `kb map`, most used)

- <name: used by n; what it is>

## Readings corrected after checking the line

- <object, use: what was misread. Or "none">

## Not counted

- Dynamic code: <ids>

在你自己的环境里测一测

  1. 拿你的地图。每种类型随机挑 5 个对象。
  2. 在新对话里,不给 agent 看地图,让它列出每个对象里的每一处使用,带行号。
  3. 比较。每一处差异都回到对应的行核对。
  4. 按上面的办法,按类型算出召回率和精确率。
  5. 换一个随机样本再来一遍,看看数字有没有变。

Costs and traps

  • A sample is a sample. The audit cannot see types or forms that are not in it. If a rare form matters, take more objects per type, or pick them by hand.
  • The audit reads about 20 to 40 objects. That is real tokens, spent once per toolkit change.
  • The blind rule is easy to break. If the agent opens the map "just to check", the audit is spoiled. Hold the line.
  • Passing 0.95 is not safety. It is a statement about the sample. I treat it as a floor for starting to learn, not as a proof.
  • The reader and the toolkit may share a blind spot. They are both built by models, so they could miss the same odd form. Checking every difference at its line helps, but cannot catch a use both overlooked. Two cheap ways to reduce it: use a different model or prompt for the reader, and have a person read a few objects by hand as an anchor.

代價和陷阱

  • 樣本就是樣本。抽查看不到不在樣本裡的類型或寫法。如果某個少見的寫法很重要,每種類型多拿一些物件,或者用手挑。
  • 抽查要讀約 20 到 40 個物件。那是真的 token,每次 toolkit 改動花一次。
  • 盲讀的規則很容易破。如果 agent「只是為了確認」打開了地圖,抽查就毀了。守住那條線。
  • 過了 0.95 不等於安全。它只是對樣本的一個說法。我把它當成開始學習的底線,不是證明。
  • 讀者和 toolkit 可能有共同的盲點。它們都是模型建的,所以可能漏掉同一個奇怪的寫法。每一處差異都在它的那一行檢查,有幫助,但抓不到兩者都忽略的使用。兩個便宜的減少辦法:讀者用不同的模型或 prompt,並讓人手動讀幾個物件當作錨點。

代价和坑

  • 样本只是样本。审计看不到样本里没有的类型或写法。如果某种罕见的写法很重要,就每种类型多抽一些对象,或者手工挑。
  • 审计要读大约 20 到 40 个对象。这是真的 token,每次 toolkit 改动花一次。
  • 盲读的规则很容易被打破。如果 agent “只是核对一下”就打开了地图,审计就废了。守住这条线。
  • 过了 0.95 不等于安全。它说的是样本。我把它当作开始学习的底线,不当作证明。
  • 读者和 toolkit 可能有同一个盲区。它们都是模型造的,所以可能漏掉同一种古怪的写法。每处差异都回到对应的行核对,有帮助,但抓不到两边都忽略的使用。有两个便宜的办法能减轻:给读者换一个模型或 prompt,再让人手工读几个对象做个基准。

When to delete it

If the map comes from a tool whose accuracy has been measured by someone else, on code like yours, skip the audit. Otherwise rerun it whenever the toolkit changes.

什麼時候可以刪掉

如果地圖來自一個準確度已經被別人量過的工具,而且是在和你相似的程式碼上,就跳過抽查。否則,每次 toolkit 改動就重跑一次。

什么时候可以删掉

如果地图来自一个工具,而它的准确度已经有别人在和你类似的代码上测过,就跳过审计。否则,每次 toolkit 改动都重新跑一遍。

Go deeper

延伸閱讀

延伸阅读

You can stop here: that is the whole idea. The next part shows where it runs, so you can look inside.讀到這裡就夠了,這就是整個想法。下一部分告訴你它在哪裡運行,可以進去看看。读到这里就够了,核心思路就是这些。下一部分告诉你它在哪里跑,可以进去看看。

In my practice (Copilot Harness)實作实战

Copilot Harness is my open-source harness for VS Code Copilot.

Repository: https://github.com/Smars-Bin-Hu/copilot-agentic-harness

Get started: clone it, open it in VS Code, and type /harness-rookie-get-me-ready in Copilot Chat. You need Python 3.9 or later and git.

Copilot Harness 是我開源的 VS Code Copilot harness。

倉庫: https://github.com/Smars-Bin-Hu/copilot-agentic-harness

開始:clone 下來,用 VS Code 打開,在 Copilot Chat 裡輸入 /harness-rookie-get-me-ready。需要 Python 3.9 以上和 git。

Copilot Harness 是我开源的 VS Code Copilot harness。

仓库: https://github.com/Smars-Bin-Hu/copilot-agentic-harness

上手:把仓库 clone 下来,用 VS Code 打开,在 Copilot Chat 里输入 /harness-rookie-get-me-ready。需要 Python 3.9 及以上版本和 git。

Explore it in the harness

Copilot Harness has an audit that samples, scores per type and lists every difference with its line. After the setup above, type:

/harness-rookie-wondering I have read the chapter "Is the map right? A blind audit". Teach me topic 16, part 16.3: how the audit samples objects, why the reading must be blind, and how a score is worked out. Open the audit skill with me and walk through one example of checking a difference at its line.

It teaches one small piece at a time, stops to ask if you are following, and opens the files with you.

在 harness 裡探索

Copilot Harness 有一個抽查,會抽樣、每種類型計分,並列出每一處差異及其行號。照上面的說明準備好之後,輸入:

/harness-rookie-wondering 我讀完了「地圖對不對:盲抽查」這一章。請教我主題 16 的 16.3:抽查怎麼抽物件、為什麼解讀必須是盲的,以及分數是怎麼算出來的。和我一起打開抽查的 skill,走一遍在某一行檢查一處差異的例子。

它一次教一小塊,停下來確認你跟得上,然後和你一起打開檔案。

在 harness 里探索

Copilot Harness 里有一套审计:抽样、按类型计分,并列出每一处差异及其行号。按上面的说明准备好之后,输入:

/harness-rookie-wondering 我读完了“地图对不对:盲抽查”这一章。请教我主题 16 的 16.3:审计是怎么抽对象的、为什么阅读必须是盲的、分数是怎么算出来的。带我打开审计的 skill,并走一遍“回到对应的行核对一处差异”的例子。

它一次只教一小块,停下来确认你跟上了,再和你一起打开文件。