Crawler Policy

2. General position

Ordinary indexing by reputable search engines is permitted where it complies with robots.txt, respects reasonable crawling norms, does not impose excessive load, does not misrepresent identity, and does not extract substantial content for prohibited uses.

Automated access that is abusive, deceptive, excessive, unauthorised, technically burdensome, security-relevant, or inconsistent with this policy may be blocked, rate-limited, logged, or otherwise restricted.

Failure to block a crawler does not constitute permission, waiver, licence, consent, or acceptance of the crawler’s own terms.

3. Identification required

Automated agents should identify themselves accurately through a meaningful user-agent string and, where reasonably possible, provide a contact URL or email address.

Automated agents must not impersonate browsers, search engines, academic researchers, commercial entities, public authorities, or other agents. Misleading user-agent strings may be treated as abusive access.

Agents that use rotating user-agent strings, concealed infrastructure, proxy networks, compromised hosts, residential proxy services, or other evasion techniques may be blocked or otherwise restricted without notice.

4. Rate limits and server load

Automated requests must be conservative. Crawlers should avoid burst traffic, aggressive parallelism, repeated requests for the same resource, unnecessary fetching of static assets, avoidable cache bypassing, and requests that create disproportionate server load.

Requests that degrade availability, consume disproportionate resources, trigger security controls, or interfere with ordinary access may be blocked, rate-limited, logged, or otherwise restricted.

Any published robots.txt crawl-delay or disallow instruction should be respected where technically applicable.

5. Permitted automated access

Subject to this policy, the following automated access is generally permitted:

• ordinary indexing of public HTML pages by reputable general-purpose search engines;
• link checking or citation checking at a conservative request rate;
• access to robots.txt, sitemap.xml, ai.txt, and llms.txt for compliance purposes;
• limited archival or academic access where it is lawful, proportionate, non-commercial, and technically conservative;
• security monitoring by authorised infrastructure or hosting providers.

Permission for ordinary indexing does not imply permission for bulk extraction, mirrored republication, commercial dataset creation, generative-AI training, or substitute services.

6. Prohibited automated uses

Unless prior written permission has been obtained, the following automated uses are prohibited, subject to any use that cannot lawfully be restricted:

• scraping substantial parts of the website for republication, mirroring, or systematic archiving;
• bulk extraction for commercial datasets, research datasets, or proprietary databases;
• collection for training, fine-tuning, benchmarking, evaluating, or augmenting machine-learning or generative-AI systems;
• creating substitute services, summaries, search products, archives, or derivative databases that materially reproduce or substitute for the website;
• circumventing technical access controls, rate limits, IP blocks, ASN blocks, user-agent blocks, or other access restrictions;
• using deceptive technical means to evade published access restrictions or blocking mechanisms;
• probing, scanning, vulnerability testing, or attempting unauthorised access;
• using rotating IP addresses, cloud-hosting relays, residential proxies, compromised hosts, or other infrastructure primarily to evade access controls;
• harvesting email addresses, contact details, metadata, publication lists, or page structures for unsolicited marketing, profiling, enrichment, or lead generation;
• submitting forms, generating artificial traffic, or testing server behaviour without permission.

7. AI crawlers and model training

Automated collection of website content for training, fine-tuning, benchmarking, evaluating, augmenting, or operating artificial intelligence systems is not permitted unless expressly authorised in writing, permitted by a specific licence attached to a particular work, or otherwise permitted by applicable law.

This restriction applies to foundation models, large language models, embedding systems, retrieval-augmented generation systems, AI search systems, synthetic-data pipelines, evaluation datasets, benchmark datasets, and downstream systems that incorporate extracted website content.

Some individual papers may be available through external repositories under specific licences. Where a paper carries a separate licence, that licence governs use of that paper. It does not grant permission to crawl, scrape, mirror, mine, or extract other parts of this website.

8. Search indexing

Reputable search engines may index public HTML pages where they comply with robots.txt, reasonable crawling practices, and this policy. Search snippets, ordinary search indexing, and linking to public pages are permitted.

Search indexing permission is not permission for bulk dataset creation, generative-AI training, commercial content extraction, mirrored republication, or substitute-content services.

Where a search engine operates both a search index and AI-training, answer-generation, or retrieval-augmented products, permission for search indexing should not be treated as permission for those separate uses.

9. Research, library, and archival use

Bona fide academic, library, policy, or archival projects may request permission for specific forms of automated access. Permission should be obtained in advance where the intended access is substantial, repeated, dataset-oriented, technically burdensome, non-public, or connected with AI systems.

Requests should identify:

• the requesting person, institution, or organisation;
• the purpose of collection;
• the scope of requested material;
• the proposed technical method and request rate;
• the retention period;
• whether the material will be redistributed, published, or shared;
• whether the material will be used in connection with AI systems;
• the safeguards proposed to prevent misuse or onward extraction.

10. Dataset use and republication

Website content must not be incorporated into bulk datasets, commercial content databases, model-training corpora, vector databases, evaluation datasets, synthetic-data pipelines, or retrieval systems without prior written permission, unless a specific work’s licence expressly permits that use or applicable law otherwise permits it.

Public availability of a page does not by itself imply permission to reproduce, republish, mirror, rehost, sell, enrich, mine, or redistribute its content at scale.

11. Static files and machine-readable policies

The following files are provided to communicate automated-access preferences and support crawler compliance:

• robots.txt;
• sitemap.xml;
• ai.txt;
• llms.txt.

These files may be updated from time to time. Automated agents should check the current versions before accessing the website.

These machine-readable files supplement this policy. They do not displace the Terms of Use, this policy, any licence applicable to a particular work, or applicable law.

Where a machine-readable instruction is more restrictive than the general permission stated in this policy, automated agents should treat the machine-readable restriction as the applicable access instruction for that resource unless written permission states otherwise.

12. Logging and enforcement

The website may use server logs, visitor records, robot logs, blocked- request logs, 404 logs, GeoIP data, ASN data, organisation names, user-agent analysis, request-path analysis, request-frequency analysis, referrer data, and other technical indicators to identify and restrict automated access.

Access may be restricted by IP address, network, ASN, organisation, user-agent, request pattern, country, referrer, or other relevant technical characteristic.

Blocking or otherwise restricting a request does not imply any conclusion about the requester beyond operational, security, legal, or policy enforcement considerations.

13. Data protection

Automated-access records may contain personal data, including IP addresses and user-agent strings. The handling of such data is described in the Privacy Notice.

These records may be processed for purposes including security, abuse prevention, diagnostics, operational integrity, enforcement of this policy, and protection of legal rights.

14. Permission requests

Requests for automated access, research crawling, archival crawling, dataset use, AI-related use, or other non-standard access should be sent to:

Peter Kahl
Lex et Ratio
Email: peter.kahl@lexetratio.com

A request should include enough detail to assess the proposed purpose, scope, method, request volume, retention period, downstream use, and safeguards.

15. No waiver

Failure to detect, block, or object to an automated agent does not constitute permission, licence, waiver, consent, acquiescence, or acceptance of that agent’s terms.

Temporary technical access to a resource does not by itself imply permission to use that resource for a prohibited purpose.

16. Changes to this policy

This Crawler Policy may be amended from time to time. The current version will be published on this page.

Automated agents should comply with the version in force at the time of access.