Commit Graph

Select branches

Hide Pull Requests

main

no-lock

spider-rs

status_codes

#1

#2

#3

#4

#6

Mono Color

• 50606bb69e It isnt quite working yet status_codes Rushmore75 2025-04-17 09:59:23 -06:00
• 5850f19cab Merge pull request 'stream_response' (#6) from stream_response into main main Oliver 2025-04-17 15:39:49 +00:00
• 2c8546e30a logging cleanup Rushmore75 2025-04-17 09:36:27 -06:00
• 4e619d0ebc logging cleanup Rushmore75 2025-04-17 09:36:13 -06:00
• 647c4cd324 work off content-type header Rushmore75 2025-04-17 09:35:57 -06:00
• 7fab961d76 no longer how this is working Rushmore75 2025-04-17 09:35:26 -06:00
• d3fff194f4 logging updates Rushmore75 2025-04-17 08:17:37 -06:00
• 3497312fd4 de-enshitified file saving logic Rushmore75 2025-04-17 08:17:29 -06:00
• 0fd76b1734 Merge pull request 'stream_response' (#4) from stream_response into main Oliver 2025-04-15 21:23:54 +00:00
• 9bfa8f9108 batch_size Rushmore75 2025-04-15 13:38:28 -06:00
• bdb1094a30 steam data to the disk Rushmore75 2025-04-15 13:07:47 -06:00
• 9aa2d9ce22 code settings Rushmore75 2025-04-15 13:06:53 -06:00
• 4b557a923c Merge pull request 'foss_storage' (#3) from foss_storage into main Oliver 2025-04-15 15:11:59 +00:00
• c08a20ac00 cleanup and more accuratly use metrics Rushmore75 2025-04-15 09:07:16 -06:00
• 94912e9125 change up how files are discovered Rushmore75 2025-04-15 09:06:57 -06:00
• a9465dda6e add instructions Rushmore75 2025-03-31 15:05:18 -06:00
• add6f00ed6 no recomp needed Rushmore75 2025-03-31 14:53:10 -06:00
• 4a433a1a77 This function sometimes throws errors, this logging should help Rushmore75 2025-03-31 14:18:37 -06:00
• 03cbcd9ae0 remove minio code Rushmore75 2025-03-31 14:18:11 -06:00
• 6fc71c7a78 add speed improvements Rushmore75 2025-03-21 12:14:29 -06:00
• 96a3ca092a :) Rushmore75 2025-03-21 12:11:05 -06:00
• b750d88d48 working filesystem storage Rushmore75 2025-03-21 11:42:43 -06:00
• 808790a7c3 file patch; Oliver 2025-03-21 07:11:51 +00:00
• 2de01b2a0e remove removed code Oliver 2025-03-21 06:48:39 +00:00
• be0fd5505b i think the files work better Oliver 2025-03-21 06:48:17 +00:00
• a23429104c dead code removal Oliver 2025-03-21 06:03:34 +00:00
• 66581cc453 getting there Oliver 2025-03-21 05:59:40 +00:00
• 7df19a480f updates Rushmore75 2025-03-20 15:11:01 -06:00
• 1b9964a938 not yet no-lock Rushmore75 2025-03-20 14:33:27 -06:00
• b9c1f0b492 readme updates Rushmore75 2025-03-19 15:05:32 -06:00
• 71b7b2d7bc it works and it is awesome Rushmore75 2025-03-19 15:04:00 -06:00
• bac3cd9d1d add most recent long run Rushmore75 2025-03-19 15:03:49 -06:00
• 1f6a0acce3 shutup spellchecker Rushmore75 2025-03-19 15:03:39 -06:00
• 53dbf53ab9 newest settings Rushmore75 2025-03-19 15:03:24 -06:00
• 0477bb26e4 viz improvements Rushmore75 2025-03-19 15:03:11 -06:00
• 6409baaffb Reducted trips to surreal by x500 Rushmore75 2025-03-19 12:41:08 -06:00
• 135a7e4957 Merge pull request 'multithreading' (#2) from multithreading into main Oliver 2025-03-19 05:00:59 +00:00
• 9aa34b3eee epic metrics Oliver 2025-03-19 04:59:50 +00:00
• de80418c00 better logging Rushmore75 2025-03-18 16:09:46 -06:00
• e3e4175f51 logging improvements Rushmore75 2025-03-18 15:25:56 -06:00
• d11e7dd27c the biggest 1 line improvement ever Rushmore75 2025-03-18 15:25:40 -06:00
• f2a3e836a0 spelling and clippy Rushmore75 2025-03-18 15:08:29 -06:00
• 3b4e6a40ce minimize vec resizing Rushmore75 2025-03-18 15:07:50 -06:00
• bd0b946245 fixed tracing Rushmore75 2025-03-18 15:02:32 -06:00
• b7540a4680 checkpoint - onto profiling Rushmore75 2025-03-18 10:53:06 -06:00
• 82929fd0fc updating for base64 Oliver Atkinson 2024-12-13 13:28:24 -07:00
• f42e770a10 moved to other repo Oliver Atkinson 2024-12-13 11:01:35 -07:00
• 611a1e923b starting on the extension Oliver Atkinson 2024-12-12 15:32:04 -07:00
• 298ad39a79 rename Oliver Atkinson 2024-12-12 14:59:54 -07:00
• 215056e493 use contains operator for better output Oliver Atkinson 2024-12-12 14:26:49 -07:00
• 22be3b2f61 updating deps Oliver Atkinson 2024-12-12 14:14:38 -07:00
• c1c8cf07bb unifed settings for testing Oliver Atkinson 2024-12-12 11:42:07 -07:00
• e4bf320ecd unifed settings for testing spider-rs Oliver Atkinson 2024-12-12 11:41:39 -07:00
• fd71a8bc13 back to spider-rs Oliver Atkinson 2024-12-12 11:04:45 -07:00
• 0f8a3d7215 using a custom parser now :) oliver 2024-11-12 23:08:09 -07:00
• 574a370f30 readme updates oliver 2024-11-12 21:24:57 -07:00
• eaa79b749e prepare get function for s3 oliver 2024-11-12 21:19:05 -07:00
• 2c28d69d55 add s3 support oliver 2024-11-12 21:03:58 -07:00
• d28d18de08 formatting and timer changes, consolidated functions oliver 2024-11-12 18:40:10 -07:00
• 8a5ac61b26 Merge pull request 'custom_engine' (#1) from custom_engine into main Oliver 2024-11-13 00:53:31 +00:00
• d9d4c56142 fits with compose.yml oliver 2024-11-12 17:51:13 -07:00
• f87a43c3a9 added demo site for testing elements oliver 2024-11-12 17:50:59 -07:00
• 7cac880f8e remove un-used function oliver 2024-11-12 17:50:28 -07:00
• 720adaa552 added support for nearly all html tags that can have a link oliver 2024-11-12 17:50:06 -07:00
• 7c32600694 this is really all that's needed oliver 2024-11-12 17:49:45 -07:00
• 399510c599 use reqwest client for epic speedup oliver 2024-11-10 20:37:00 -07:00
• ec66c4e765 remove unused import oliver 2024-11-10 20:36:39 -07:00
• a9628ee5e4 working, now onto speeding it up oliver 2024-11-10 20:24:04 -07:00
• 5404d5c3e8 it works :party: oliver 2024-11-09 23:30:57 -07:00
• fd971bafbf it works now oliver 2024-11-09 15:28:10 -07:00
• c3997b0bb7 works more, but still not all the way oliver 2024-11-09 11:30:32 -07:00
• 7826c4cec6 jank-ish fix but it sure does work make the root record (for links https://example.com/) have a record id of the url, thus preventing duplication when using upsert Oliver Atkinson 2024-10-31 15:32:37 -06:00
• 3a46dd937b updates Oliver Atkinson 2024-10-31 15:09:48 -06:00
• fbca067b1f clean up walk() Oliver Atkinson 2024-10-31 14:10:14 -06:00
• 9324160e74 crawling 🕷️ Oliver Atkinson 2024-10-07 11:14:56 -06:00
• 974bccc457 no longer using spider, just wiritng my own crawler Oliver Atkinson 2024-10-04 13:52:34 -06:00
• 2d2b09116e lockfile has update 🤷 oliver 2024-08-26 23:15:06 -06:00
• 38fe7d3a59 add useage oliver 2024-08-26 01:14:10 -06:00
• 6e95aeb154 install instructions oliver 2024-08-26 01:09:46 -06:00
• fdbc49337b use custom spider dependency oliver 2024-08-26 01:06:13 -06:00
• a507d8dfaa readmd with test results oliver 2024-08-26 01:01:11 -06:00
• 2017a08f4c update schema oliver 2024-08-26 00:57:36 -06:00
• 67d89ff9eb it helps when you don't reference [0] for two separate elements... oliver 2024-08-26 00:55:36 -06:00
• edf9bfc1f5 update needed to merge, which isn't the default action oliver 2024-08-26 00:49:04 -06:00
• 409f8c0c01 let me cook! oliver 2024-08-25 15:50:59 -06:00
• e66131b411 add oliver 2024-08-23 05:22:49 -06:00
• bfed9a6ca6 first commit oliver 2024-08-23 05:21:42 -06:00