Oliver/internet_mapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
minio
internet_mapper/README.md
oliver 574a370f30 readme updates
2024-11-12 21:24:57 -07:00

502 B
Raw Permalink Blame History

Surreal Crawler

Crawls sites saving all the found links to a surrealdb database. It then proceeds to take batches of 100 uncrawled links untill the crawl budget is reached. It saves the data of each site in a minio database.

TODO

  • Domain filtering - prevent the crawler from going on alternate versions of wikipedia.
  • Conditionally save content - based on filename or file contents
  • GUI / TUI ?
  • Better asynchronous getting of the sites. Currently it all happens serially.