netweir documentation

• Getting started: install, fetch a page, write and run a spider.
• Selecting: CSS, XPath and Beautiful Soup style queries.
• Forms, logins and APIs: POST and other methods, form submission, JSON APIs.
• Crawling: spiders, requests, pipelines, exporters and the command line.
• Declarative spiders: items and rules that run in Rust.
• Self-healing: blocks, checkpoints, tracked selectors and repair proposals.
• Browser: drive Chrome for pages that need JavaScript.
• Settings: every setting and its default.

The design behind it all is in design/v0.1.md, the browser driver’s in design/browser.md, and requests other than GETs in design/requests.md.