openmic.social is an uncensored community. You may encounter strong language, controversial opinions, and mature or NSFW material. You must be 18+ to browse. Illegal content is prohibited and removed on sight — please report it. By continuing, you accept that you may see content you personally disagree with.
Although this looks like a clever approach, a kind of stochastic key, I do not see how this guarantees to distinguish text written by big babble machines versus humans. Humans also have a certain pattern of writing, a given distribution of how some words are more likely to appear than others. How can one tell them really apart?
As an indicator, yeah, might be usable. But I wouldn’t read too much into it before seeing results of a study that runs actual tests.
It’s not about the variation of the words, it’s about the variation of the words from the model baseline.
Like if your word choice was almost the exact same as Claude’s normally, maybe you just talked to them a lot and picked up their phrases like it’s not nothing.
But if you managed to be almost exactly like Claude and yet varied the possible words exactly according to a hidden entropy key, they’d know it was actually Claude with the SymthID-Text watermarking applied, as no human would end up falling into that statistical bucket.
Yeah, still, I wouldn’t claim “as no human would end up falling into that”, given that it may not be that unlikely to find at least one human who displays similar writing the more humans you involve.
Until a formal analysis is presented and an experimental study is published, which covers the most important influencing factors, the reliability of this concept is limited.
No, it is actually statistically impossible for a human to replicate this on sufficiently long runs of text.
This is not about replicating writing like a model. This is basically about guessing which words to pick from the list of suitable words based on a rule that you don’t know (because the key is secret).
To reduce this to the simplest possible example, imagine you are writing a “text” from just two letters: “a” and “b”. Let’s say for convenience that the text is supposed to be random. So the text would look something like “ababaaabbababbbabababaabbbabaababbaaabbabaabbaaaaabaaabbbaabaabababbabbbbbbbbabbabaabbbbbbbaabbabaab”
(generated with
'''.join(random.choice(['a', 'b']) for i in range(0, 50)))The watermarking works as follows: the model owner holds a key, and then uses that key to influence the random choices between “a” and “b” somehow, in a context-dependent way. The actual algorithm is quite complicated, but for simplicity let’s just say we have a secret pattern which biases the random choice towards it. In order to see the exaggerated results, let’s say the secret key is “aaaabbbb” (of course this is a bad secret key, once again just an example), and that the bias is strong (let’s say 80%). So this would mean that the first four letters in our text are more likely to be “a”, the next four letters are more likely to be “b”, then the next four letters are more likely to be “a”, and so on.
Then the text would look something like “aaaabaabaabaabbbabaaaabbaaaababbbaaaabbbabaabbbbaaabaabbaaaaabbbabababbaaaaabbbbaaabbbbbaaaaababaaba”.
(generated with
''.join(random.choice(['a', 'b'] + ([key[i % len(key)]] * 3)) for i in range(0, 100)))You can see visually that the secret key has affected the text. Of course in this example even if you didn’t know the secret key you could probably figure it out, in reality the algorithm is way more complicated than that, relying on cryptography, so you wouldn’t be able to know the secret key or see that the string has been biased at all.
If the text is long enough, and you know the secret key, you can guarantee that the text was generated with it. In our examples, the letters in the text match our key 77% of the time. The probability of an actual random algorithm generating a text like that is already very low, despite the base entropy being only 100 bits. If my math is correct, for our example the p-value is 2.7 * 10⁻⁸, or about 0.00000027%. I would bet a hungy that the text was generated by our watermarking algorithm, with odds like these!
Of course we did exaggerate the bias and our base algorithm was random. In reality the bias is smaller, the algorithm for determining the likelihoods of possible next tokens is very complicated (it’s the LLM itself), and the algorithm for determining which token to bias is also way more complicated (involving cryptography and real secret keys). That said, hopefully it should help you understand why, for sufficiently long texts, this fingerprinting is just not possible to be replicated by humans.
I do not have the time to work through every part of the example, but imo the main claim is still overstated. Showing that a result would be extremely unlikely under a particular null model is not the same as showing that it is “statistically impossible” for a human to produce. It also does not give a guarantee how the text was written. A tiny p-value is still a probability under assumptions and not a proof of provenance.
Furthermore, a human does not even need to know the secret key. By pure chance a human written text can display an unusually high alignment with the detector’s secret partitioning.
The published watermark work, which is also cited by the article, appears to be much more careful about this (based on a quick skim). It reports false positive/negative rates, thresholds, length requirements, and more. Those can be very strong results, provided the assumed conditions apply. They do not turn a detector into an infallible test. Moreover longer text only helps if the assumptions and watermark signal actually remain intact, which can fail in general.
In such controlled settings, sure, I do not have much issues there. But the claims of “a human cannot replicate this” or “we can guarantee the text was generated with the watermark” are much stronger than the statistics, and especially the cited literature, actually appear to support.
I thought the article explained that pretty reasonably on a scale of probability and weight. The longer the text, the more reliable the scoring.
But it does not show a sufficient formal proof and no experimental validation. Many important questions to evaluate the concept are left unanswered, which limits the interpretability and condenses it to “just trust me, bro, it’s a good idea, because I say so”.
I’m not sure we’ve read the same article. There are literally interactive demonstrations within the page to demonstrate how the concept works.
Interactive demonstrations are not the same as a formal proof or experimental validation. So we shouldn’t attribute more to this technique than the available evidence can really support.
I found some time to quickly skim through the sources they have listed. And from that it became pretty clear that this is not realiable in detecting LLM generated versus human output in general. Under very tight assumptions specific error rates were reported that appeared rather low. However, these assumptions do not hold in general, even with more text if no relevant signal remains. There is currently no scientifically validated general purpose way of reliably detection.
More importantly in the context of Claude, the production watermarking scheme is undisclosed. Therefore, the cited experiments on known watermarking schemes can neither establish how reliably text generated by Claude can be detected, nor how reliably the technique described in the article removes the actual watermark.
It can be treated as an indicator at best, but not as validated proof.
Gish Gallop… if you’re going to start questioning whether the technique clearly demonstrated has validity, then you need to specifically state what your objections are, as opposed to vague statements. For emphasis, the demonstration isn’t on AI detecting, but rather AI watermarking. You wouldn’t use this tool to check if text was written by AI, but rather if the text was written by one singular LLM vs literally everything else.