AI/Blogfor Laravel

Safe fetching for AI research

Updated

SafeFetcher reads a public web page, feed or JSON document so the AI has real sources to work from. Run it from a queued job or an Artisan command: it refuses to run inside a web request.

use ByteMinus\AiBlog\Fetching\Feeds\FeedParser;
use ByteMinus\AiBlog\Fetching\SafeFetcher;
use ByteMinus\AiBlog\Fetching\TextExtractor;

$document = app(SafeFetcher::class)->fetch('https://example.com/feed.xml');

$feed = app(FeedParser::class)->parseDocument($document);   // RSS, Atom, RSS 1.0 or JSON Feed
$page = app(TextExtractor::class)->extract($document);      // readable text, title, author, date
$page->untrusted();                                         // wrap it before it goes near a model
redirect: check again, at most 5 times Check URL scheme, port, host Resolve DNS all addresses public Connect pinned to that IP robots.txt must allow Read body size and time caps Untrusted text for the model refused or failed: nothing is returned
A fetch passes five checks in order, and a redirect starts again from the first one. Any failed check stops the fetch with nothing returned.

On the first URL and on every redirect (at most 5) it checks that the URL is plain http(s) with no credentials, a standard port and no internal host names; resolves the host itself and requires every address to be public; pins the connection to the address it checked, so DNS rebinding can't move the request inside your network; obeys robots.txt; ignores proxies and cookies; and caps the body size and total time. Anything it returns is untrusted data: hand it to a model only through TextExtractor and UntrustedText.

Edit on GitHubView as Markdown