web-reader

A small, self-hosted fetcher run by one person for non-commercial, personal research.

What it fetches

Public documents only, mainly published electricity-reliability regulations and guidance (NERC, FERC and the regional entities), so they can be read and quoted accurately. It never logs in, never bypasses a paywall, and never requests anything marked confidential or restricted (for example, CEII).

How the content is used

Kept as a private reference library with exact quotes and citations. Content is not used to train AI models and is not republished.

How it behaves

• Honours robots.txt (user-agent token: web-reader).
• One request at a time per site, at least 3 seconds apart; slower on sites that ask for it.
• Respects Retry-After and backs off on HTTP 429 and 503.
• Asks whether a file has changed (If-Modified-Since / If-None-Match) instead of downloading it again.
• Identifies itself honestly on every request.

To block or limit it

Add this to your robots.txt:

User-agent: web-reader
Disallow: /

Contact

hs8uyh6kv@mozmail.com — you will get a reply, and the crawler will be adjusted or stopped.