Commit Graph

Select branches

Hide Pull Requests

foss_storage

main

status_codes

#1

#13

#16

#2

#3

#4

#6

#8

Mono Color

• 2c339a36f9 handle checking for file better main Oliver 2025-10-09 23:00:11 -06:00
• 73216f7003 fix the issue where nothing works Oliver 2025-10-09 22:35:01 -06:00
• 1e59ebd5c4 even when not downloading, update the database Oliver 2025-10-09 22:13:06 -06:00
• 52d5e101d0 bragging Oliver 2025-10-09 22:03:19 -06:00
• 5b728bacd6 close #24, make program aware of the files already on disk Oliver 2025-10-09 21:52:41 -06:00
• b0fe7f4761 close #18, format Oliver 2025-10-09 21:52:06 -06:00
• 5ade5e36df closes #11 Oliver 2025-08-08 23:39:44 -06:00
• 95b8af0356 Restart threads that prematurely ended Oliver 2025-08-08 23:35:01 -06:00
• ad8d7c606d increase csma/ca time Oliver 2025-08-08 23:34:45 -06:00
• f7a3ca8fd7 changes status_codes table 2025-08-04 23:02:05 +00:00
• f3a51065b5 remove fixme Rushmore75 2025-07-17 09:37:03 -06:00
• 343d3a7570 better logging Rushmore75 2025-07-17 09:36:37 -06:00
• e535bcc295 Merge branch 'main' of https://git.oliveratkinson.net/Oliver/internet_mapper Rushmore75 2025-07-17 08:59:32 -06:00
• a0fd81d956 better config file Rushmore75 2025-07-17 08:58:30 -06:00
• 5cbba33a09 update how the database interactions work Rushmore75 2025-07-17 08:52:47 -06:00
• 83def7ba27 close #10 Oliver 2025-07-16 16:07:37 -06:00
• 76e78cc745 better logging Rushmore75 2025-07-16 16:02:16 -06:00
• b4038b76dd fix prometheus lol Rushmore75 2025-07-16 16:02:07 -06:00
• caa523f1eb cleanup Rushmore75 2025-07-16 11:48:23 -06:00
• f7bb0eef16 turn program into batch_size parrallel downloaders Rushmore75 2025-07-16 11:47:42 -06:00
• 865f9be8c0 Merge pull request 'works 😄' (#16) from tempfiles into main Oliver 2025-07-16 02:26:14 +00:00
• 48abc73092 works 😄 Oliver 2025-07-15 20:25:44 -06:00
• 0061866976 Merge pull request 'traces and new metrics work' (#13) from better_metrics into main Oliver 2025-07-16 00:58:47 +00:00
• 9662b68b0d traces and new metrics work Oliver 2025-07-10 23:44:23 -06:00
• 6f98001d8e Merge pull request 'status_codes' (#8) from status_codes into main Oliver 2025-07-11 00:49:27 +00:00
• 4989a59ddf checkpoint foss_storage Oliver 2025-07-10 18:46:25 -06:00
• 6790061e22 helper code Rushmore75 2025-07-09 15:58:22 -06:00
• 50606bb69e It isnt quite working yet Rushmore75 2025-04-17 09:59:23 -06:00
• 5850f19cab Merge pull request 'stream_response' (#6) from stream_response into main Oliver 2025-04-17 15:39:49 +00:00
• 2c8546e30a logging cleanup Rushmore75 2025-04-17 09:36:27 -06:00
• 4e619d0ebc logging cleanup Rushmore75 2025-04-17 09:36:13 -06:00
• 647c4cd324 work off content-type header Rushmore75 2025-04-17 09:35:57 -06:00
• 7fab961d76 no longer how this is working Rushmore75 2025-04-17 09:35:26 -06:00
• d3fff194f4 logging updates Rushmore75 2025-04-17 08:17:37 -06:00
• 3497312fd4 de-enshitified file saving logic Rushmore75 2025-04-17 08:17:29 -06:00
• 0fd76b1734 Merge pull request 'stream_response' (#4) from stream_response into main Oliver 2025-04-15 21:23:54 +00:00
• 9bfa8f9108 batch_size Rushmore75 2025-04-15 13:38:28 -06:00
• bdb1094a30 steam data to the disk Rushmore75 2025-04-15 13:07:47 -06:00
• 9aa2d9ce22 code settings Rushmore75 2025-04-15 13:06:53 -06:00
• 4b557a923c Merge pull request 'foss_storage' (#3) from foss_storage into main Oliver 2025-04-15 15:11:59 +00:00
• c08a20ac00 cleanup and more accuratly use metrics Rushmore75 2025-04-15 09:07:16 -06:00
• 94912e9125 change up how files are discovered Rushmore75 2025-04-15 09:06:57 -06:00
• a9465dda6e add instructions Rushmore75 2025-03-31 15:05:18 -06:00
• add6f00ed6 no recomp needed Rushmore75 2025-03-31 14:53:10 -06:00
• 4a433a1a77 This function sometimes throws errors, this logging should help Rushmore75 2025-03-31 14:18:37 -06:00
• 03cbcd9ae0 remove minio code Rushmore75 2025-03-31 14:18:11 -06:00
• 6fc71c7a78 add speed improvements Rushmore75 2025-03-21 12:14:29 -06:00
• 96a3ca092a :) Rushmore75 2025-03-21 12:11:05 -06:00
• b750d88d48 working filesystem storage Rushmore75 2025-03-21 11:42:43 -06:00
• 808790a7c3 file patch; Oliver 2025-03-21 07:11:51 +00:00
• 2de01b2a0e remove removed code Oliver 2025-03-21 06:48:39 +00:00
• be0fd5505b i think the files work better Oliver 2025-03-21 06:48:17 +00:00
• a23429104c dead code removal Oliver 2025-03-21 06:03:34 +00:00
• 66581cc453 getting there Oliver 2025-03-21 05:59:40 +00:00
• 7df19a480f updates Rushmore75 2025-03-20 15:11:01 -06:00
• b9c1f0b492 readme updates Rushmore75 2025-03-19 15:05:32 -06:00
• 71b7b2d7bc it works and it is awesome Rushmore75 2025-03-19 15:04:00 -06:00
• bac3cd9d1d add most recent long run Rushmore75 2025-03-19 15:03:49 -06:00
• 1f6a0acce3 shutup spellchecker Rushmore75 2025-03-19 15:03:39 -06:00
• 53dbf53ab9 newest settings Rushmore75 2025-03-19 15:03:24 -06:00
• 0477bb26e4 viz improvements Rushmore75 2025-03-19 15:03:11 -06:00
• 6409baaffb Reducted trips to surreal by x500 Rushmore75 2025-03-19 12:41:08 -06:00
• 135a7e4957 Merge pull request 'multithreading' (#2) from multithreading into main Oliver 2025-03-19 05:00:59 +00:00
• 9aa34b3eee epic metrics Oliver 2025-03-19 04:59:50 +00:00
• de80418c00 better logging Rushmore75 2025-03-18 16:09:46 -06:00
• e3e4175f51 logging improvements Rushmore75 2025-03-18 15:25:56 -06:00
• d11e7dd27c the biggest 1 line improvement ever Rushmore75 2025-03-18 15:25:40 -06:00
• f2a3e836a0 spelling and clippy Rushmore75 2025-03-18 15:08:29 -06:00
• 3b4e6a40ce minimize vec resizing Rushmore75 2025-03-18 15:07:50 -06:00
• bd0b946245 fixed tracing Rushmore75 2025-03-18 15:02:32 -06:00
• b7540a4680 checkpoint - onto profiling Rushmore75 2025-03-18 10:53:06 -06:00
• 82929fd0fc updating for base64 Oliver Atkinson 2024-12-13 13:28:24 -07:00
• f42e770a10 moved to other repo Oliver Atkinson 2024-12-13 11:01:35 -07:00
• 611a1e923b starting on the extension Oliver Atkinson 2024-12-12 15:32:04 -07:00
• 298ad39a79 rename Oliver Atkinson 2024-12-12 14:59:54 -07:00
• 215056e493 use contains operator for better output Oliver Atkinson 2024-12-12 14:26:49 -07:00
• 22be3b2f61 updating deps Oliver Atkinson 2024-12-12 14:14:38 -07:00
• c1c8cf07bb unifed settings for testing Oliver Atkinson 2024-12-12 11:42:07 -07:00
• 0f8a3d7215 using a custom parser now :) oliver 2024-11-12 23:08:09 -07:00
• 574a370f30 readme updates oliver 2024-11-12 21:24:57 -07:00
• eaa79b749e prepare get function for s3 oliver 2024-11-12 21:19:05 -07:00
• 2c28d69d55 add s3 support oliver 2024-11-12 21:03:58 -07:00
• d28d18de08 formatting and timer changes, consolidated functions oliver 2024-11-12 18:40:10 -07:00
• 8a5ac61b26 Merge pull request 'custom_engine' (#1) from custom_engine into main Oliver 2024-11-13 00:53:31 +00:00
• d9d4c56142 fits with compose.yml oliver 2024-11-12 17:51:13 -07:00
• f87a43c3a9 added demo site for testing elements oliver 2024-11-12 17:50:59 -07:00
• 7cac880f8e remove un-used function oliver 2024-11-12 17:50:28 -07:00
• 720adaa552 added support for nearly all html tags that can have a link oliver 2024-11-12 17:50:06 -07:00
• 7c32600694 this is really all that's needed oliver 2024-11-12 17:49:45 -07:00
• 399510c599 use reqwest client for epic speedup oliver 2024-11-10 20:37:00 -07:00
• ec66c4e765 remove unused import oliver 2024-11-10 20:36:39 -07:00
• a9628ee5e4 working, now onto speeding it up oliver 2024-11-10 20:24:04 -07:00
• 5404d5c3e8 it works :party: oliver 2024-11-09 23:30:57 -07:00
• fd971bafbf it works now oliver 2024-11-09 15:28:10 -07:00
• c3997b0bb7 works more, but still not all the way oliver 2024-11-09 11:30:32 -07:00
• 7826c4cec6 jank-ish fix but it sure does work make the root record (for links https://example.com/) have a record id of the url, thus preventing duplication when using upsert Oliver Atkinson 2024-10-31 15:32:37 -06:00
• 3a46dd937b updates Oliver Atkinson 2024-10-31 15:09:48 -06:00
• fbca067b1f clean up walk() Oliver Atkinson 2024-10-31 14:10:14 -06:00
• 9324160e74 crawling 🕷️ Oliver Atkinson 2024-10-07 11:14:56 -06:00
• 974bccc457 no longer using spider, just wiritng my own crawler Oliver Atkinson 2024-10-04 13:52:34 -06:00