Oliver/internet_mapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

readme updates

Browse Source
This commit is contained in:
Rushmore75 2025-03-19 15:05:32 -06:00
parent 71b7b2d7bc
commit b9c1f0b492
1 changed files with 5 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
README.md
View File

@ -7,6 +7,9 @@ Crawls sites saving all the found links to a surrealdb database. It then proceed
- [ ] Domain filtering - prevent the crawler from going on alternate versions of wikipedia. - [ ] Domain filtering - prevent the crawler from going on alternate versions of wikipedia.
- [ ] Conditionally save content - based on filename or file contents - [ ] Conditionally save content - based on filename or file contents
- [ ] GUI / TUI ? - [x] GUI / TUI ? - Graphana
- [ ] Better asynchronous getting of the sites. Currently it all happens serially.3/19/25: Took 20min to crawl 100 pages - [x] Better asynchronous getting of the sites. Currently it all happens serially.
- [ ] Allow for storing asynchronously
3/19/25: Took 20min to crawl 100 pages
This ment we stored 100 pages, 142,997 urls, and 1,425,798 links between the two. This ment we stored 100 pages, 142,997 urls, and 1,425,798 links between the two.