01
Fetch policy
Permitted
Whether the page may be requested at all. Robots and the acquisition signals the origin published decide it, and a scoped permit has to cover the path first.
Robots + acquisition permit
Ethical Crawler
Crawler that actually reads and respects policy.
Invented example
What it decides
02
Training policy
Review required
Whether the text may train a model. Silence is not permission, so an origin that says nothing lands here, and nothing in the pipeline moves it out on its own.
No rights evidence
03
Evidence record
Not exportable
Every verdict is stored with the reasons that produced it and a hash of the evidence it was decided against, so it can be re-run later and compared.
Training review blocks use
What I am trying to find out
- How smart can a model get on data that specifically opted in?
- A generalist wants everything. A specialist wants one domain done properly: code, or markdown, or one trade. Where the ceiling sits for either is unknown, and no corpus of strictly opted-in third-party text exists to find out on.
- Does anyone declare a policy at all?
- Six endpoints exist to say so. How many origins publish any of them, whether what they publish parses, and whether it still says the same thing on a re-read.