Crawler Policy

2. General position

Ordinary indexing by reputable search engines is permitted where it complies with robots.txt, respects reasonable crawling norms, does not impose excessive load, does not misrepresent identity, and does not extract substantial content for prohibited uses.

Automated access that is abusive, deceptive, excessive, unauthorised, technically burdensome, security-relevant, or inconsistent with this policy may be blocked, rate-limited, logged, or served a 404 response.

Failure to block a crawler does not constitute permission, waiver, licence, consent, or acceptance of the crawler’s own terms.

3. Identification required

Automated agents should identify themselves accurately through a meaningful user-agent string and, where possible, provide a contact URL or email address.

Automated agents must not impersonate browsers, search engines, academic researchers, commercial entities, public authorities, or other agents. Misleading user-agent strings may be treated as abusive access.

Agents that use rotating user-agent strings, concealed infrastructure, proxy networks, compromised hosts, residential proxy services, or other evasion techniques may be blocked without notice.

4. Rate limits and server load

Automated requests must be conservative. Crawlers should avoid burst traffic, aggressive parallelism, repeated requests for the same resource, unnecessary fetching of static assets, avoidable cache bypassing, and requests that create disproportionate server load.

Requests that degrade availability, consume disproportionate resources, trigger security controls, or interfere with ordinary access may be blocked, rate-limited, logged, or served a 404 response.

Any published robots.txt crawl-delay or disallow instruction must be respected where technically applicable.

5. Permitted automated access

Subject to this policy, the following automated access is generally permitted:

• ordinary indexing of public HTML pages by reputable general-purpose search engines;
• link checking or citation checking at a conservative request rate;
• access to robots.txt, sitemap.xml, ai.txt, and llms.txt for compliance purposes;
• limited archival or academic access where it is lawful, proportionate, non-commercial, and technically conservative;
• security monitoring by authorised infrastructure or hosting providers.

Permission for ordinary indexing does not imply permission for bulk extraction, mirrored republication, commercial dataset creation, generative-AI training, or substitute services.

6. Prohibited automated uses

Unless prior written permission has been obtained, the following automated uses are prohibited:

• scraping substantial parts of the website for republication, mirroring, or archiving;
• bulk extraction for commercial datasets, research datasets, or proprietary databases;
• collection for training, fine-tuning, benchmarking, evaluating, or augmenting machine-learning or generative-AI systems;
• creating substitute services, summaries, search products, archives, or derivative databases that compete with, displace, or materially reproduce this website;
• circumventing robots.txt, ai.txt, llms.txt, rate limits, access controls, IP blocks, ASN blocks, user-agent blocks, or other technical measures;
• probing, scanning, vulnerability testing, or attempting unauthorised access;
• using rotating IP addresses, cloud-hosting relays, residential proxies, anonymisation networks, compromised hosts, or deceptive infrastructure to evade controls;
• harvesting email addresses, contact details, metadata, publication lists, or page structures for unsolicited marketing, profiling, enrichment, or lead generation;
• submitting forms, generating artificial traffic, or testing server behaviour without permission.

7. AI crawlers and model training

Automated collection of this website’s content for training, fine-tuning, benchmarking, evaluating, augmenting, or operating artificial intelligence systems is not permitted unless expressly authorised in writing or permitted by a specific licence attached to a particular work.

This restriction applies to foundation models, large language models, embedding systems, retrieval-augmented generation systems, AI search systems, synthetic-data pipelines, evaluation datasets, benchmark datasets, and downstream systems that incorporate extracted website content.

Some individual papers may be available through external repositories under specific licences. Where a paper carries a separate licence, that licence governs use of that paper. It does not grant permission to crawl, scrape, mirror, mine, or extract the rest of this website.

8. Search indexing

Reputable search engines may index public HTML pages where they comply with robots.txt, reasonable crawling norms, and this policy. Search snippets, ordinary search indexing, and linking to public pages are permitted.

Search indexing permission is not permission for bulk dataset creation, generative-AI training, commercial content extraction, mirrored republication, or substitute-content services.

Where a search engine operates both a search index and AI training, answer-generation, or retrieval-augmented product, permission for search indexing should not be treated as permission for those separate AI-related uses.

9. Research, library, and archival use

Bona fide academic, library, policy, or archival projects may request permission for specific forms of automated access. Permission should be obtained in advance where the intended access is substantial, repeated, dataset-oriented, technically burdensome, non-public, or connected with AI systems.

Requests should identify:

• the requesting person, institution, or organisation;
• the purpose of collection;
• the scope of requested material;
• the proposed technical method and request rate;
• the retention period;
• whether the material will be redistributed, published, or shared;
• whether the material will be used in connection with AI systems;
• the safeguards proposed to prevent misuse or onward extraction.

10. Dataset use and republication

Website content must not be incorporated into bulk datasets, commercial content databases, model-training corpora, vector databases, evaluation datasets, synthetic-data pipelines, or retrieval systems without prior written permission, unless a specific work’s licence expressly permits that use.

Public availability of a page does not imply permission to reproduce, republish, mirror, rehost, sell, enrich, mine, or redistribute its content at scale.

11. Static files and machine-readable policies

The following files are provided to support automated-agent compliance:

• robots.txt;
• sitemap.xml;
• ai.txt;
• llms.txt.

These files may be updated from time to time. Automated agents should check and comply with the current versions.

Where there is a conflict between a machine-readable instruction and this human-readable policy, the stricter restriction should be treated as applying unless written permission states otherwise.

12. Logging and enforcement

The website may use server logs, visitor records, robot logs, blocked- request logs, 404 logs, GeoIP data, ASN data, organisation names, user-agent analysis, request-path analysis, request-frequency analysis, referrer data, and other technical indicators to identify and restrict automated access.

Access may be blocked by IP address, network, ASN, organisation, user- agent, request pattern, country, referrer, or other technical characteristic.

Blocking a request does not imply any view about the requester beyond operational, security, legal, or policy enforcement considerations.

13. Data protection

Automated-access records may contain personal data, including IP addresses and user-agent strings. The handling of such data is described in the Privacy Notice.

The website uses these records for purposes including security, abuse prevention, diagnostics, operational integrity, enforcement of this policy, and protection of legal rights.

14. Permission requests

Requests for automated access, research crawling, archival crawling, dataset use, AI-related use, or other non-standard access should be sent to:

Peter Kahl
Lex et Ratio Ltd
Email: peter.kahl@juris.vc

A request should include enough detail to assess the proposed purpose, scope, method, request volume, retention period, downstream use, and safeguards.

15. No waiver

Failure to detect, block, or object to an automated agent does not constitute permission, licence, waiver, consent, acquiescence, or acceptance of that agent’s terms.

Temporary technical access to a resource does not imply permission to use that resource for a prohibited purpose.

16. Changes to this policy

This Crawler Policy may be amended from time to time. The current version will be published on this page.

Automated agents are expected to comply with the version in force at the time of access.