datasette-ip-rate-limit 0.1a0

2026-05-14 · Source: Simon Willison's Weblog · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Intermediate, quick

Summary

The datasette-ip-rate-limit 0.1a0 plugin was released on May 14, 2026, to address issues with poorly-behaved crawlers overwhelming the datasette.io site. Developed using Codex (GPT-5.5 xhigh), this configurable plugin enables rate limiting of Datasette requests based on client IP addresses. Its production configuration, demonstrated on datasette.io, includes parameters such as a "Fly-Client-IP" header, a maximum of 10,000 keys, and exemptions for paths like "/static/*" and "/-/turnstile*". Specific rules can be defined, such as limiting requests to "/global-power-plants/*" and "/legislators/*" to 60 requests within a 60-second window, with a 20-second block duration upon exceeding the limit.

Key takeaway

For site administrators experiencing crawler-induced load issues on Datasette instances, implementing datasette-ip-rate-limit 0.1a0 offers a direct solution. You should review your site's traffic patterns to identify vulnerable paths and configure specific rate-limiting rules, including appropriate `window_seconds` and `max_requests` values, to mitigate abuse without impacting legitimate users.

Key insights

A new Datasette plugin rate limits requests by IP address, configurable to protect specific site areas.

Principles

• Rate limiting protects against abusive traffic.
• AI can assist in plugin development.

Method

The plugin identifies client IPs via a specified header, applies rules based on paths, and blocks IPs exceeding defined request limits within a time window.

In practice

• Configure `max_keys` for IP tracking capacity.
• Define `exempt_paths` for critical services.
• Set `window_seconds` and `max_requests` per rule.

Topics

• datasette-ip-rate-limit
• Datasette
• IP Rate Limiting
• Web Crawlers
• Plugin Development

Code references

• datasette/datasette-ip-rate-limit
• simonw/datasette.io

Best for: CTO, VP of Engineering/Data, Software Engineer, DevOps Engineer, AI Engineer

Related on AIssential

• 2026 NLP Data Collection Guide: How Proxy IPs Help Improve Large-scale Collection Efficiency — Effective NLP data collection for AI models requires overcoming anti-scraping measures and ensuring stable, scalable access.
• Millions of AI agents imperiled by critical vulnerability in open source package — A critical Starlette vulnerability (CVE-2026-48710) allows authentication bypass and data theft in AI agent systems.
• datasette-tailscale 0.1a0 — An experimental Datasette plugin integrates with Tailscale, enabling secure, private network access to Datasette instances via a sidecar.
• datasette-llm-limits 0.1a0 — The datasette-llm-limits plugin enables granular, periodic spending limits for LLM usage within Datasette.
• v2.1.76 — GitHub imposes secondary rate limits, requiring users to wait or sign in to continue operations.
• Sorry for the outages: Bot spam is pushing our servers to the limit — Bot spam, particularly from Perplexity, is causing severe website outages and service disruptions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Simon Willison's Weblog.