Oliver/internet_mapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add useage

Browse Source
This commit is contained in:
oliver 2024-08-26 01:14:10 -06:00
parent 6e95aeb154
commit 38fe7d3a59
1 changed files with 9 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
README.md
View File

@ -6,11 +6,18 @@ This is including the crawl and loading into the database and linking sites. (Lo
This run created 4299 site links with 23286 links between the sites. (It found my this git site which really bolsters those numbers.) This run created 4299 site links with 23286 links between the sites. (It found my this git site which really bolsters those numbers.)
# Use / Install ## Install / Build
* You will need rust to compile the crawler [rustup.rs](https://rustup.rs) * You will need rust to compile the crawler [rustup.rs](https://rustup.rs)
* You need python3 (will come installed on most linux distros) and poetry for dependancy management. * You need python3 (will come installed on most linux distros) and poetry for dependancy management.
* Install `pipx`, `python3` * Install `pipx`, `python3`
* Then: `pipx install poetry` * Then: `pipx install poetry`
* Then: `poetry install` to install the project dependancies * Then: `poetry install` to install the project dependancies
* You need to install [surrealdb](https://surrealdb.com)
## Use
Just run `./crawl.sh {url}` and it will start crawling. You can tweak the budget inside [crawl.sh](https://git.oliveratkinson.net/Oliver/internet_mapper/src/branch/main/crawl.sh) if you want.
You can also prefix the command with `time` to benchmark the system, such as: `time ./crawl.sh https://discord.com`.