پلگ ان ٹیگ: crawler blocking

Block AI scrapers and unwanted crawlers with bot rules, robots.txt and noai meta controls, per-page settings, and basic rate limiting.