Found a humanizer that actually bypassed pangram, copyleaks and gptzero - but not t9xt

A casual Tuesday morning turned in to an unexpected encounter - found a tool endschool.today - that maps iterations against Pangram, and finds the best combination that flags as human. I tried several texts and to my surprise they came back as 100% human with medium confidence. While this does highlight how unreliable AI detection models are in general for deterministic purposes, it also reveals a fundamental problem - humanizers.

Why are humanizers a problem? For the longest time, it was AI or Human. And the boundary was clear. You could almost sniff the text and say, this is AI. More recently, tools using creative restructuring have blurred the lines. Because the output layer is separated from the underlying logical layer of the model, the patterns became less detectable. Then came the model response, most detectors rushed to add more data from humanizers in to their training set. This worked well for a bit, but turns out there was no single 'humanized' pattern. New patterns keep popping up, making it difficult to detect at first - but eventually they end up on the radar. It's this covert assimilation of human text patterns that makes humanizers so problematic. While some humans might say they are great at spotting them, it still takes up time - and at scale, it's simply not possible. AI Labs are scanning millions of pages an hour for new data indexing, imagine all that shitty content being added to the index and shared authoritatively because a search index company says it exists. I would argue the future of models I dependent on filtering out AI text. Whether it's safety, so that AI models don't hide instructions for their peers in plain forms or to prevent model drift because the new training data is skewed towards unsubstantiated synthetic content.

How do AI detectors adapt to humanizers? I suspect (as detection companies usually do not share amongst themselves) a new adapter is trained on newly launched humanizers every week/month and added on top of existing model + adapters. This is great, but exposed a deeper problem - the more humanized content you added, the more blurry the line between humanized and human text became. As humanizers borrowed more patterns from human text, it became really difficult to identify authenticity. (some were surprisingly good). Another problem that arises from adding more content to the negative set is content pollution, each additional training set has the potential to overlap on previous training data - this creates a major problem even with locked weights. Reconciling adapters is not an easy task, especially across such treacherous text. Let me throw in an example, last month I checked an extract from Techcrunch on a leading tool - it came back 100% human. Another extract from the same author this week came back as AI (minus one sentence). Now we don't know what tools writers use on tc, so it's very hard to answer whether it is or isn't AI (or assisted), the writing did seem a little bit mechanical - but that's been their overall editorial style for the past many years. Now many tools like Grammarly have been around for over 15 years, so I would suspect that those tools would also raise these flags. But there is an alternate explanation, which is that a recently trained adapter skewed their detection output. (the more likely scenario)

Why the humanizer did not come back as Human on our model? Idk for sure. But I suspect from looking at the data, because it's similar to other humanizers we have added to our training set. Every week we filter from millions of scanned pages to add new data to our set, identify by author (AI model) and find new text patterns to study. This doesn't make us foolproof, but it does give us an edge. We identified this as a problem early on and designed our system around it, rather than using raw texts on training we use a proprietary adaptive patterns approach (which pretty much no one does to the best of my knowledge). This doesn't mean we don't use texts, but we take it a few reasoning layers further.

Does this mean these tools have stopped working? Not necessarily, they might find the tool and add it to their dataset as soon as today. But it doesn't stop more from coming up. For example, we use an internal humanizer we developed to estimate gaps, and pretty much no provider has flagged it as AI yet. 'In addition', we are also in talks with other model companies to license our tech. This would make it incredibly hard for humanizers to thrive.

Here's an example text that flagged as human but is 100% AI (and humanized):
The outputs generated by language models will vary depending on implementation, training data and the governance around them.
Organisations that establish processes for accountability, provenance, explainability and more stand to benefit the most in the long-term from these technologies while minimising risks associated with them such as overconfident false facts, error compounding and provenance obfuscation.
Artificial Intelligence is changing the way we work across industries.

For transparency: every tool has its weaknesses - this doesn't mean these tools are bad or that you should not use them. Nor would I call any particular tool from these bad or unreliable. Normally posts like this are picked up by their teams, so let's try again next week...