From bac3cd9d1d30f8ce263d49318aceb2685a6ea42d Mon Sep 17 00:00:00 2001 From: Rushmore75 Date: Wed, 19 Mar 2025 15:03:49 -0600 Subject: [PATCH] add most recent long run --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 94913df..67e2356 100644 --- a/README.md +++ b/README.md @@ -8,4 +8,5 @@ Crawls sites saving all the found links to a surrealdb database. It then proceed - [ ] Domain filtering - prevent the crawler from going on alternate versions of wikipedia. - [ ] Conditionally save content - based on filename or file contents - [ ] GUI / TUI ? -- [ ] Better asynchronous getting of the sites. Currently it all happens serially. \ No newline at end of file +- [ ] Better asynchronous getting of the sites. Currently it all happens serially.3/19/25: Took 20min to crawl 100 pages +This ment we stored 100 pages, 142,997 urls, and 1,425,798 links between the two.