This news item from a few weeks ago revealed that when users created a “public link” of their Claude chat history, that chat history ended up in Google’s search queries.

Aren’t these urls supposed to be more or less undiscoverable by virtue of the long string of random characters that they contain?

How did Google discover those urls?

Did these people publish the urls somewhere publically themselves?

Or did Google “fish” them out of their gmail inbox or something like that?

  • KurtVonnegut@mander.xyzOP
    2·
    2 days ago

    Ah ok, looks like I didn’t read the article carefully enough.

    That still leaves two scenarios though:

    1. Claude fucked up, and gave Google a way to index public links of people. After this news broke, they asked Google (and other search engines) to not show those urls anymore.
    2. People fucked up and/or fell victim to Google’s the data hunger, which let Google to index their public links, after which Claude asked Google not to index such links.

    I guess, in the end, I’m mostly wondering if the long random strings of public links are generally “brute-force-discovered” by crawlers like Google, or whether “discovery” of such links requires some kind of breach.

    • SnoopSqueak@lemmy.today
      5·
      2 days ago

      Claude fucked up, and gave Google a way to index public links of people.

      Anthropic fucked up, Claude is not a person.

      After this news broke, they asked Google (and other search engines) to not show those urls anymore.

      Correct, that seems to be the case.

      People fucked up and/or fell victim to Google’s the data hunger, which let Google to index their public links

      Kind of. The user fuck up would be not realizing that Anthropic had no protections from search engines, and that by clicking “share” they were creating publicly accessible links.

      after which Claude asked Google not to index such links.

      Anthropic seems to have updated the domain those links are generated on so that crawlers do not index public chats, yes.

      I guess, in the end, I’m mostly wondering if the long random strings of public links are generally “brute-force-discovered” by crawlers like Google, or whether “discovery” of such links requires some kind of breach.

      I am also curious. I looked it up. https://developers.google.com/search/docs/fundamentals/how-search-works

      The first stage is finding out what pages exist on the web. There isn’t a central registry of all web pages, so Google must constantly look for new and updated pages and add them to its list of known pages. This process is called “URL discovery”. Some pages are known because Google has already visited them. Other pages are discovered when Google extracts a link from a known page to a new page: for example, a hub page, such as a category page, links to a new blog post.

      If I had to guess based on this, Google had probably indexed the domain Claude or its frontend run on and automatically scooped up the public chats as they became available.

      It is also possible that any Claude chat shared will still be accessible to web crawlers, if they just ignore the flags telling them not to scrape, crawl, access, index, etc.