# Safe fetching for AI research

> **Diagram:** SafeFetcher checks the URL, resolves the host and requires every address to be public, connects pinned to the checked address, checks robots.txt and reads a size- and time-capped body. A failed step refuses the fetch and returns nothing. A redirect goes back to the first check, at most five times. What it returns goes through TextExtractor and UntrustedText before reaching a model.

`SafeFetcher` reads a public web page, feed or JSON document so the AI has real sources to work from. Run it
from a queued job or an Artisan command: it **refuses to run inside a web request**.

```php
use ByteMinus\AiBlog\Fetching\Feeds\FeedParser;
use ByteMinus\AiBlog\Fetching\SafeFetcher;
use ByteMinus\AiBlog\Fetching\TextExtractor;

$document = app(SafeFetcher::class)->fetch('https://example.com/feed.xml');

$feed = app(FeedParser::class)->parseDocument($document);   // RSS, Atom, RSS 1.0 or JSON Feed
$page = app(TextExtractor::class)->extract($document);      // readable text, title, author, date
$page->untrusted();                                         // wrap it before it goes near a model
```

On the first URL and on every redirect (at most 5) it checks that the URL is plain http(s) with no credentials,
a standard port and no internal host names; resolves the host itself and requires every address to be public;
**pins the connection to the address it checked**, so DNS rebinding can't move the request inside your network;
obeys `robots.txt`; ignores proxies and cookies; and caps the body size and total time. Anything it returns is
untrusted data: hand it to a model only through `TextExtractor` and `UntrustedText`.