Navigating the Digital Frontier: The Struggles of a Volunteer-Run LGBT History Archive

Preserving Our Past in a Digital Age

In a world where social values are evolving, the importance of remembering the struggles of past generations cannot be overstated. Jonathan Harborne, the visionary behind the LGBT History Project, founded this essential online encyclopedia fifteen years ago to document the rich tapestry of LGBTQ+ experiences. With over fifty million views since its inception in 2011, the site has become a beacon of hope and resilience, even gaining recognition from the British Library as a vital part of history.

The Unseen Threats

However, the integrity of this valuable resource faced a severe threat when a barrage of artificial intelligence crawlers began scraping its content. Harborne’s recent efforts to modernize the website—transitioning from an outdated platform to a new AWS server—uncovered an alarming reality. Instead of the expected enhancements in speed and security, the site was brought to its knees by automated traffic.

Upon analyzing server logs with the help of Claude Code, Harborne discovered that Meta’s crawler alone had made approximately 26,000 requests in a single day, extracting around 12GB of data, including sensitive editing histories and login pages. This influx of bot traffic not only slowed the site but also doubled operational costs, forcing Harborne to pay out of pocket for the very infrastructure needed to support the site.

The Fight for Digital Rights

Harborne’s struggles are not unique. As independent website operators grapple with an incessant wave of bot traffic, many find themselves at the mercy of a few colossal corporations that dominate the internet landscape. Reports suggest that bot traffic now constitutes about 30% of all web activity, raising alarming questions about the future of digital content.

Catherine Flick, a professor specializing in ethics and technology, highlights the profound changes in web hosting dynamics. She notes the challenges independent operators now face, stating, “You just can’t really get away with doing that anymore,” referring to the simpler days of running a web server from home. The implications are far-reaching, affecting the open-access ethos that the internet was initially built upon.

The Technical Battleground

Faced with this digital onslaught, Harborne found himself in a steep learning curve, mastering tools like Cloudflare to configure blocking rules and distinguish human users from bots. This technical battle underscores a deeper issue: the lack of accountability among major tech companies. Harborne expressed frustration over the difficulty of reaching out to giants like Meta and Alibaba, lamenting the absence of a regulatory framework to protect independent content creators from such invasive practices.

The Broader Implications for Digital Ecosystems

Harborne’s experience serves as a cautionary tale about the relationship between AI companies and the very sources of information they rely on. He argues that as these companies harvest data from small websites without regard for their sustainability, they risk destroying the ecosystems that make their products viable. “It’s like a virus that’s killing its own host,” he warns, emphasizing the precarious balance between innovation and the survival of independent digital platforms.

A Call for Ethical Practices in AI

As discussions around the ethics of AI continue to gain traction, experts like Carissa Veliz, an AI ethicist from the University of Oxford, argue that prioritizing profit over human rights leads to a digital landscape reminiscent of the Wild West. The lack of effective policing on the internet leaves volunteer-run initiatives vulnerable, forcing them to fend for themselves against an overwhelming tide of corporate interests.

Despite these challenges, Harborne remains committed to preserving the LGBT History Project as long as he can sustain it financially. His determination to implement stricter controls on bot access reflects a growing need among independent creators to protect their digital legacies. In an era where technology is rapidly evolving, the lessons learned from Harborne’s fight resonate powerfully within Miami’s vibrant, yet equally challenging, digital and cultural landscape.

Editorial note: This article was created by A Bit Lavish Miami’s Magazine as an original editorial reinterpretation based on publicly available reporting. Original source: fastcompany.com. Read the original article here: https://www.fastcompany.com/91589354/ai-crawlers-hammered-a-volunteer-run-lgbt-history-archive-2.
Images are used for editorial reference with source credit. If an image requires correction or removal, please contact A Bit Lavish.