Persistent bug: score lead contradiction that cannot be corrected

[GuangHui68]

Over the past year of using KataGo for game analysis, I have consistently encountered a serious issue: significant and persistent errors in score lead estimation during complex multi-group capturing fights. This problem was previously mentioned in issues #1182 and #1183, where it was initially mistaken for a model-related issue. I now believe this is a fundamental flaw in the MCTS logic feedback mechanism within the main program. I have repeatedly encountered clear contradictions in score lead between consecutive moves, and crucially, neither increasing computation time nor forcing deeper analysis or re-analysis can correct these errors.
我使用katago分析棋谱的这一年来，发现katago在多块棋对杀时，存在明显的目差计算错误问题。以前在1182#、1183#中就提到过，当时认识不深，误以为是model的问题
现在我认为，这个BUG的本质，在于主程序的MCTS逻辑反馈，设置不当。我多次遇到，紧接的前后2手，目差存在明显错误，无论是扩大计算量，还是手动深入分析、重新分析，都无法纠正。
For example, in this game from 1986 between Song Xuelin 7p and Liu Xiaoguang 8p, analyzed with the latest b11c768 model, at White 170 the score lead is displayed as +18.1 — clearly an exaggerated and unreasonable value. Looking back, at White 168, KataGo had already identified a move with a +11.8 point lead as an available option. This suggests that the score estimation is not merely noisy, but systematically diverges. It can be said that the bug was already present at White 168, and the error may have originated as early as Black 167, or possibly even earlier.
比如这盘棋，1986年第20届围棋全国团体赛男子组第5轮，宋雪林7段vs刘小光八段，这是使用最新的 b11c768分析的。图中，白170的目差是+18.1！，这么明显的错误，程序熟视无睹。再往前翻，其实白168，katago已经有一个+11.8目的选择。可以说白168的时候，已经有了明确的bug，那么错误最迟可能是黑167的目差判断错误，当然也有可能更早。

+ Another example: here at White 172, the score lead is +12 points; immediately following Black 173, it shows +9.6. Such a clear contradiction between consecutive moves should not be ignored, yet the main program fails to detect it.

If such contradictions occurred 10 moves apart, it would be easier to dismiss, but when they occur on consecutive moves, it is clearly unreasonable.
比如这里的白172，+12目；紧接着黑173，+9.6目。这么明显的BUG，主程序同样没有发现。
如果说，目差前后冲突是相隔10手产生的，难以纠正还好推搪，但是这么前后紧邻的，也太不合理了吧？

My visit count is not small — 20k here. Later I will provide examples showing that even with 120k visits, this bug cannot be fixed.

Additionally, I previously used the b28c512 model, and at these same two moves (172 and 173), the score leads showed contradictory values of -20+ points, which were also clearly incorrect.
我这个visits，也不算小了啊，20k。后面会给出例子，证明哪怕是120k，这种bug也是修不好的。
顺带说一下，以前用 b28c512 model，这172、173两步，是前后的-20目+，显然也是错的
Someone told me that using 10 million visits would fix the problem. That's laughable. I have previously used 1 million visits on the b40c768 model, and the bug was still there (not on this particular game, but on a Fan Xiping handicap-three-stones game). So I cannot understand what this bug really is.

But at the very least, the main program seems to lack a proper feedback mechanism between consecutive moves — even adjacent moves can be completely inconsistent. In a situation like this, where there is only one move possible, shouldn't the score lead be 0?

I cannot believe that the author would consider this bug unimportant. If the score lead cannot be calculated correctly (and the errors I have shown are very large), then using KataGo for game analysis is clearly unreliable.
某人告我，你使用1000万visits，就不会出问题了。这不是笑话么，还别说，我曾经在 b40c768 model下，使用过100万的visits，结果bug依旧（不是这局棋，是范西屏的让三子局）。所以我无法理解这个bug到底是怎么回事？
但是至少，主程序应该欠缺前后反馈机制，甚至前后两手都可以脱节。这种只此一手的情况下，难道不该目差为0么？
我不相信， 作者会认为这个bug不重要。如果目差都算不准（我举例的可是极大的偏差），那用它去分析棋谱，显然就是不可信的。

19860406 第20届全国团体赛第5轮 宋雪林vs.zip

[GuangHui68]

This is a case I encountered in earlier research: Game 5 of the 41st Korean Myungin (Honor) Tournament, 2013-12-15, Lee Sedol vs Choi Cheolhan. The analysis was done using the b28c512 model. At Black 223, the score lead is +4.7; at White 224, it is +6.0. This was after three rounds of deep analysis at 120k visits each. Initially, both values were above +12 points, but after further analysis they could not be reduced any further.
这个是以前研究时的案例，20131215 41届韩国名人战第5局 李世石vs崔哲瀚，用的是 b28c512。黑223，+4.7；白224，+6.0。这还是经过了120k的3次深入分析后的结果，最初两个目差都是12目+。但是继续下去，再也减少不了了。

+ Immediately following, there is another bug. Moreover, the score calculation in this position is very simple — there is no complex fighting at all. At White 232, the score lead is +6.6; at Black 233, it is +7.6. 接下去又是一个bug，而且现在的目差计算非常简单，根本不存在什么对杀。白232，+6.6目，黑233，+7.6目

20131215 41届韩国名人战第5局 李世石vs崔哲瀚.zip

[GuangHui68]

This is a second type of bug, slightly harder to correct. At Black 109, the score lead is +5.6. The cause appears to be an earlier error at Black 105, which was calculated as -10.8 and later corrected. Still, it's only four moves apart — not far, right?

这是第二类bug，纠正难度稍微大些。黑109，+5.6目；原因是前面的黑105，-10.8目计算有误，后面纠正了前面的错误。不过当前才隔了4手，也不远吧？

- This is a third type of bug. At White 264, recapturing the 18 stones is the only move — why is it shown as -2.0 points?

这是第三类bug。白264，接回18子只此一手，怎么是 -2.0目呢？

[GuangHui68]

As a comparison, I used StarGo's paid analysis mode on the same Liu Xiaoguang game. The results are as follows:
At White 170, StarGo also showed +6.6 points. It seems that even other AIs have misevaluations here. (I couldn't afford a very high computation setting, but 69k visits should be enough, right?)
作为对比，我使用星阵的付费模式，对刘小光的那局棋进行了分析，结果如下
白170，星阵也显示 +6.6目？看来这里AI都有误算啊（我付不起太高的计算量，不过6万也够了吧？）

+ However, at White 172 and Black 173, the score leads were completely normal ( just only 20k visits ). 但是白172、黑173，目差完全正常，而且只需2万不到的计算量。

[lightvector]

Thanks for the report. I haven't looked in detail at these bugs, but at least some of these bugs seem to be unrelated to KataGo. For example

This is a third type of bug. At White 264, recapturing the 18 stones is the only move — why is it shown as -2.0 points?

This one likely has nothing to do with KataGo - this is presumably the Katrain gui, which I don't personally work on or maintain. KataGo is simply outputting particular scores, and it's Katrain presumably that is the one choosing to report it as -2.0 relative to some other score. KataGo is only responsible for reporting the underlying raw numbers, all issues relating to whether they are shown as + or - relative to some other number or by how much are going to be the choices of various GUIs and not KataGo's issue.

For the rest of what you mention, if other AIs are showing issues too then it simply might just be a complex situation.

If such contradictions occurred 10 moves apart, it would be easier to dismiss, but when they occur on consecutive moves, it is clearly unreasonable.

Ah I can understand why you'd think that. However, if there is a major error (which there always can be, as the bots far from perfect) 10 moves apart, then as you go move by move the winrate and score cannot stay constant between each consecutive move. If it was consistent on every move, then it would be consistent across every 10 moves too. There will be usually some point at which the error is realized, and that single move may show a sharp change.

Perhaps you are expecting every error to be "smooth" where it gets always realized only gradually, so that over 10 moves there is a major error but no single move shows a big jump? That can happen too, but it's actually somewhat less common compared to most of the error happening on just one or two of the moves in between. Think about it - when there are very large errors, they are often caused by misreading tactics in some fight, or by some blind spot, , rather than by subtle whole-board judgment or subtle direction of play - the latter kinds of misjudgments tend to lead to more gradual things but they also tend to swing the score far less. And when it's a misread tactic or blind spot, it is often closer to all-or-nothing, either you solve it or you don't. There's not as much room for the judgment to change only gradually. Because it is all-or-nothing, there will often be a point at which the bot changes from not seeing something to seeing it. So, logically, this leads to the phenomenon that a large error over 10 or 20 moves can often be traced to large errors occurring over just one or two or three moves, rather than errors that are realized only gradually, and so if you accept that contradictions may happen 10 or 20 moves apart due to imperfections in the bot, you can also expect that similarly large contradictions must also often happen over single moves.

[lightvector]

If you find some of the choices of any particular GUI to confuse the matter, you might be interested in trying alternative GUIs maintained by other community members like https://github.com/wimi321/lizzieyzy-next? For example I think lizzieyzy tends by default to show you simply the absolute estimated score on moves (e.g. black is ahead by X), rather than making particular choices for you about how to show it as a + or - relative to some other moves, and then you can judge the differences for yourself and at least get a clearer picture of what KataGo's misevaluations are. When the score is volatile and the differences are rapidly changing and confusing, you might prefer a GUI that simply shows the more raw absolute number.

[GuangHui68]

If you find some of the choices of any particular GUI to confuse the matter, you might be interested in trying alternative GUIs maintained by other community members like……

KaTrain is the officially recommended GUI for KataGo, isn't it? Lizzie has been discontinued for years. Also, a GUI is only responsible for displaying KataGo's output in a human-readable format — it cannot determine the score lead value itself. The values I'm using here are, of course, the absolute score leads, and the positions I showed are near the endgame, where ambiguity is minimal. This should make debugging easier. At move 264, there is clearly no other move — the score lead must be ZERO.

I also understand that the second type of bug may be difficult to fix. But for the first type — the obvious score lead errors on adjacent moves — and the third type, would they be relatively easier to address? Even fixing just one of them would already be a good step forward. This bug has been around for several years now, hasn't it?

katrain 不是katago官方优先推荐的么？lizzie已经停止开发多年了。另外，GUI只是负责把katago的输出界面化给人而已，这个具体目差多少，katrain自己肯定是不能确定的。我这里选的，当然是绝对目差，而且我举例的，本来就是官子快结束的时候。就是说，是歧义最少的时候，这时候的误差，更加容易 debug。这264，除了目差0，显然不存在其他选择。
我也知道，第2类bug，要修复难度比较大。但是第1类的紧邻的这种目差显而易见的错误，还有第3类，是不是更加容易些？哪怕先修复其中一个，也是一个很好的进步啊，这个bug，都好几年了吧。

[GuangHui68]

I have also tested other GUIs, such as Sabaki. Among all the GUIs I have used, I think KaTrain's analysis interface is the best. Its only drawback is that the move numbers are not displayed directly on the stones, but the changes in score lead are very clear at a glance, making it very suitable for identifying which moves are brilliant and which are mistakes in a game. This is something that other GUIs do not offer.

我测试过其他的GUI，比如 Sabaki。在我用过的这些GUI中，我认为，katrain的分析界面是最好的，它的唯一缺点就是谱着的手数没有直接显示在棋子上，但是目差的增减是一目了然，非常适合确定一盘棋中，哪些是妙手，哪些是恶手。
这点是其他GUI所不具备的。

[lightvector]

At move 264, there is clearly no other move — the score lead must be ZERO.

No, this still sounds like the GUI's issue, not KataGo's issue.

KataGo does NOT report score differences. KataGo reports the absolute score - e.g. in this position Black is winning by 5, in this other position White is winning by 2. It's entirely the GUI that chooses to take one score and another score and report their difference, and KataGo has nothing to do with the GUI's choice of how to compute that difference.

I have no idea what katrain really is doing exactly in this case. But I'm guessing it runs a quick search at the root with fewer visits and gets score X, and then runs a different updated/deeper/newer search on a particular move with possibly a different number of visits, and gets score X-2. That's fine - KataGo is allowed to change its mind, if you search 10k visits and it gets one score, and then you search an additional 10k visits and it reevaluates the position after the same move and realizes something deeper, and changes its mind, that's a thing that can happen.

But then, it's entirely Katrain's choice to remember the old value of X (even though KataGo already changed its mind) and compare it to the new value of X-2 and claim that that difference is -2, and that the difference is due to the move, rather than to the fact that the search was done later or with a different number of visits or some other difference that caused KataGo to re-evaluate. That's not KataGo's problem, KataGo simply always reports its latest absolute estimate of the position it's asked about, including if it changed its mind, and it's the GUI that chooses to subtract it against some other possibly stale or different value from before that and give you a strange result.

the changes in score lead are very clear at a glance

It seems like this case of -2.0 indicates the opposite - that actually it's not very clear and in at least one case the GUI is picking a choice of what two numbers to subtract, one of them presumably "stale", that produces a result that you don't understand.

This is why as I mentioned if you're trying to understand exactly what KataGo is doing in a volatile situation and exactly when it's changing its mind and why, it may be better to do it with a GUI that gives you a more true view of exactly what KataGo is actually reporting, rather than complicating it by showing you doing additional processing on what KataGo is showing you by taking certain differences between multiple outputs at different times. The GUI might be doing something that seems clear when the position is simple and KataGo is confident about everything, but actually it's hiding the complexity from you in a way that breaks down when the situation becomes more complex and can't accurately represent to you the real data when KataGo needs to change its mind.

[lightvector]

KaTrain is the officially recommended GUI for KataGo

KaTrain is one possible GUI that people use that is popular, but it's not an "official" GUI, and I have never worked on its code. I prefer LizzieYzy myself (the original Lizzie is dead yes but LizzieYzyNext is updated and maintained recently by someone), or actually even https://github.com/lightvector/katsura which is a relatively simple minimal GUI I wrote that has just the features I want for myself.

It sounds like what you'd like is a feature request for KaTrain to re-evaluate its cached records on past moves based on future evaluations prior to taking their difference - e.g. if it asks KataGo about a position and obtains the scores of moves as A,B,C,... such that the value of the position is X ~= max(A,B,C) and then it next evaluates A some more and KataGo changes its mind to say that the score is a worse A', then rather than report A'-X as the value of move A, Katrain should first update its estimate of position to be X' ~= max(A',B,C) and then report A'-X' as the value of the move, taking into account how KataGo's newest evaluation of the move affects the judgment of the original position instead of subtracting it against an old value.

[GuangHui68]

KaTrain is one possible GUI that people use that is popular, but it's not an "official" GUI, and I have never worked on its code. I prefer LizzieYzy myself (the original Lizzie is dead yes but LizzieYzyNext is updated and maintained recently by someone), or actually even https://github.com/lightvector/katsura which is a relatively simple minimal GUI I wrote that has just the features I want for myself.

Okay, I will immediately submit a report to KaTrain and will include your suggestions along with it.

Regarding the first bug: does the issue lie within KataGo itself? That is, in a situation where there is only one move available, KataGo should ideally give consistent score lead evaluations for the preceding two positions — for example, White 172 and Black 173 in this case.

[GuangHui68]

Also, regarding your katsura — I couldn't get the engine to work. The entire program folder has no README, no gtp logs, and the error messages are all garbled codes.
I asked an AI assistant about this app. It mentioned that this app might conflict with Docker. Is that true?

另外，你这个 katsura，我搞不定 engine 啊。没有logs输出，对话框中全部乱码……
我问了AI，这个app，与 docker 冲突？

[GuangHui68]

天啊，折腾了2个小时，在AI帮助下，总算把你这个app跑通了，简直是把所有的坑，都踩了一遍，给测试奖金吧，呵呵……
1、在Windows下运行你这个程序，一定要安装 wsl 的 Linux 环境，只有 docker，是不行的。命令如下：
wsl --install Ubuntu-24.04 ；接着，安装完成后，设置 账号、密码，再执行
wsl --set-default Ubuntu-24.04
上述两步是关键！

然后，路径一定要用Linux格式，比如这里的 cd /mnt/g/Katago/katsura/katago1171 ；注意，最后必须是分号。然后执行命令是
./katago.exe gtp -model ……，各位注意，必须是 ./katago.exe，否则报错。 最后的 cfg文件，各位使用 katago推荐的就行，需要各种参数的那个。
你这个版本的优点，是可以随时从任意一步开始分析，确实比 katrain 强啊

[GuangHui68]

嗯，在你这个app下，白172、黑173都是正常的目差。但是，要实现这个结果，需要300K+的visits，吓人啊……
右边文字是原先 katrain的注释，当时只用了35k

[GuangHui68]

另外这个GUI有个明显的bug，你看，我现在选了一个变化图，这时理应已经下了的子，自动不再显示步数（而不是我要手动去选，如下图所示）；然而现在还显示着，这样就坏事了

[sbbdms]

1、关于目差不是半目的整数倍：
KataGo内的数据输出，包括像胜率、目差这类，都是基于统计意义下的数值，其中经过加权平均等处理，所以目差不是半目整数倍的情况是常见结果。
当然，去做人工的四舍五入就能得到绝艺讲解那样的结果，显然这难度不大，且KataGo作者认为没有必要。

2、关于胜率和目差的冲突：
上面已经提到KataGo内数据输出是经过加权平均处理的结果，那么应该很容易能想到类似于以下的情况：某个盘面下，KataGo搜索了以下选点：
（1）选点A，胜率90%，目差+1目，搜索权重为9.0
（2）选点B，胜率5%，目差-100目，搜索权重为1.0
那么它们加权平均的结果就是胜率81.5%，目差-9.1目。很显然这里的胜率和目差的结果是冲突的，但很显然这也是一个非常容易能够被理解的结果。

这类结果尤其容易且不限于出现在以下情况：
（1）在几条大龙互相对杀的这类高不确定度的盘面下。此时胜率可能非常震荡，但加权平均之后有可能就会在50%附近。而目差的绝对值却往往会相差很大，加权下来出现一方优十几目几十目都是很容易出现的结果
（2）像上面的例子这样计算量严重不足的时候

3、关于KataGo内胜率和目差的优先级：
KataGo的默认设置里，胜率的优先级是远高于目差的。如果你想放弃基于胜率给出的结果，硬要以目差作为主要指标的话，那你可以调整配置文件里winLossUtilityFactor、staticScoreUtilityFactor、dynamicScoreUtilityFactor这三项。其中第一项是胜率的效应（默认为1.00），第二项是目差的效应（默认为0.10、0.30，这里我也不拓展解释它们俩的区别了）。
但是如果这样设置，计算出来的结果往往比默认结果还要差得多。原因无它，棋局的胜负肯定远比目差更重要。

4、关于你提到的人工修正KataGo的输出结果
KataGo使用的神经网络是一个黑箱，它是经过几千万自对弈棋谱下用一个月甚至几个月，按照特定的优化器自我训练而来的，而不是靠人工调参的。
所以：
（1）不要再去想着人工修正某个局面下的目差输出这种解决方案了，这点从程序架构上来说就不现实。除非你想让围棋AI回到像以前《手谈》那样的水平，那时候就是纯人工调参的结果，很容易就能人工调整程序在某个局面下的评估输出。
按现在KataGo的程序架构，如果你想硬要“修复”神经网络在某个局面下的输出，那就只能把这个局面在训练过程里产生几万倍的权重，在这种条件下去让神经网络进行训练。但是这样就会产生我接下来要提到的第二个问题：
（2）神经网络的优劣判定，是以【对任何局面输入】下的整体表现（常见的评估标准即是以elo分）去做评估的。如果你硬要以上面这种方式去“修复”某个特定局面下的输出，那么训练出来的权重在其它局面下的输出几乎肯定会变差。
显然这是得不偿失的行为，也就是为什么你在KataGo连开了好几个issue描述基本相同的内容，也不可能得到你想要的解决结果的原因。