diff --git a/README.md b/README.md index 7ec5c6d..f5b4aa1 100644 --- a/README.md +++ b/README.md @@ -40,14 +40,18 @@ $EDITOR Crawler.toml - [x] Allow for storing asynchronously - dropping the "links to" logic fixes this need - [x] Control crawler via config file (no recompliation needed) -3/17/25: Took >1hr to crawl 100 pages +### Feats -3/19/25: Took 20min to crawl 1000 pages +3/17/25: Took >1hr to crawl 100 pages. + +3/19/25: Took 20min to crawl 1000 pages. This ment we stored 1000 pages, 142,997 urls, and 1,425,798 links between the two. -3/20/25: Took 5min to crawl 1000 pages +3/20/25: Took 5min to crawl 1000 pages. -3/21/25: Took 3min to crawl 1000 pages +3/21/25: Took 3min to crawl 1000 pages. + +7/.../25: Downloaded just shy of 12TB of data from a remote server. # About