• digdilem@lemmy.ml
    link
    fedilink
    English
    arrow-up
    9
    ·
    edit-2
    2 months ago

    In my experience, the AI bots are absolutely not honoring robots.txt - and there are literally hundreds of unique ones. Everyone and their dog has unleashed AI/LLM harvesters over the past year without much thought to the impact to low bandwidth sites.

    Many of them aren’t even identifying themselves as AI bots, but faking human user-agents.