Oliver/internet_mapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
11 Commits 1 Branch 0 Tags 114 KiB
Python 64.7%
Rust 33.1%
Shell 2.2%
main
Go to file
oliver 2d2b09116e lockfile has update 🤷
2024-08-26 23:15:06 -06:00
.vscode add 2024-08-23 05:22:49 -06:00
src let me cook! 2024-08-25 15:50:59 -06:00
.gitignore add 2024-08-23 05:22:49 -06:00
Cargo.lock lockfile has update 🤷 2024-08-26 23:15:06 -06:00
Cargo.toml use custom spider dependency 2024-08-26 01:06:13 -06:00
compose.yml let me cook! 2024-08-25 15:50:59 -06:00
crawl.sh it helps when you don't reference [0] for two separate elements... 2024-08-26 00:55:36 -06:00
driver.py it helps when you don't reference [0] for two separate elements... 2024-08-26 00:55:36 -06:00
poetry.lock let me cook! 2024-08-25 15:50:59 -06:00
pyproject.toml let me cook! 2024-08-25 15:50:59 -06:00
README.md add useage 2024-08-26 01:14:10 -06:00
schema.surql update schema 2024-08-26 00:57:36 -06:00
test let me cook! 2024-08-25 15:50:59 -06:00

README.md

Surreal Crawler

Mapping with a budget of 1000 (crawl 1000 sites, so many more links are actually discovered), on my webiste on 8/26/2024 took 1m9s.

This is including the crawl and loading into the database and linking sites. (Locally hosted surreal db instance)

This run created 4299 site links with 23286 links between the sites. (It found my this git site which really bolsters those numbers.)

Install / Build

  • You will need rust to compile the crawler rustup.rs
  • You need python3 (will come installed on most linux distros) and poetry for dependancy management.
    • Install pipx, python3
    • Then: pipx install poetry
    • Then: poetry install to install the project dependancies
  • You need to install surrealdb

Use

Just run ./crawl.sh {url} and it will start crawling. You can tweak the budget inside crawl.sh if you want.

You can also prefix the command with time to benchmark the system, such as: time ./crawl.sh https://discord.com.