Oliver/internet_mapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Web crawler + storage + visualization (soon)
34 Commits 3 Branches 0 Tags 1.6 MiB
Rust 94.9%
HTML 4.7%
CSS 0.3%
JavaScript 0.1%
fd71a8bc13
Go to file
Oliver Atkinson fd71a8bc13 back to spider-rs
2024-12-12 11:04:45 -07:00
.vscode add s3 support 2024-11-12 21:03:58 -07:00
demo_site added demo site for testing elements 2024-11-12 17:50:59 -07:00
src back to spider-rs 2024-12-12 11:04:45 -07:00
.gitignore add s3 support 2024-11-12 21:03:58 -07:00
Cargo.lock back to spider-rs 2024-12-12 11:04:45 -07:00
Cargo.toml back to spider-rs 2024-12-12 11:04:45 -07:00
compose.yml add s3 support 2024-11-12 21:03:58 -07:00
README.md readme updates 2024-11-12 21:24:57 -07:00

README.md

Surreal Crawler

Crawls sites saving all the found links to a surrealdb database. It then proceeds to take batches of 100 uncrawled links untill the crawl budget is reached. It saves the data of each site in a minio database.

TODO

  • Domain filtering - prevent the crawler from going on alternate versions of wikipedia.
  • Conditionally save content - based on filename or file contents
  • GUI / TUI ?
  • Better asynchronous getting of the sites. Currently it all happens serially.