Oliver/internet_mapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Web crawler + storage + visualization (soon)
33 Commits 3 Branches 0 Tags 1.5 MiB
Rust 94.9%
HTML 4.7%
CSS 0.3%
JavaScript 0.1%
0f8a3d7215
Go to file
oliver 0f8a3d7215 using a custom parser now :)
2024-11-12 23:08:09 -07:00
.vscode add s3 support 2024-11-12 21:03:58 -07:00
demo_site added demo site for testing elements 2024-11-12 17:50:59 -07:00
src using a custom parser now :) 2024-11-12 23:08:09 -07:00
.gitignore add s3 support 2024-11-12 21:03:58 -07:00
Cargo.lock using a custom parser now :) 2024-11-12 23:08:09 -07:00
Cargo.toml using a custom parser now :) 2024-11-12 23:08:09 -07:00
compose.yml add s3 support 2024-11-12 21:03:58 -07:00
README.md readme updates 2024-11-12 21:24:57 -07:00

README.md

Surreal Crawler

Crawls sites saving all the found links to a surrealdb database. It then proceeds to take batches of 100 uncrawled links untill the crawl budget is reached. It saves the data of each site in a minio database.

TODO

  • Domain filtering - prevent the crawler from going on alternate versions of wikipedia.
  • Conditionally save content - based on filename or file contents
  • GUI / TUI ?
  • Better asynchronous getting of the sites. Currently it all happens serially.