Web crawler + storage + visualization (soon)
| .vscode | ||
| src | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| compose.yml | ||
| crawl.sh | ||
| driver.py | ||
| poetry.lock | ||
| pyproject.toml | ||
| README.md | ||
| schema.surql | ||
| test | ||
Surreal Crawler
Mapping with a budget of 1000 (crawl 1000 sites, so many more links are actually discovered), on my webiste on 8/26/2024 took 1m9s.
This is including the crawl and loading into the database and linking sites. (Locally hosted surreal db instance)
This run created 4299 site links with 23286 links between the sites. (It found my this git site which really bolsters those numbers.)