|
|
4e619d0ebc
|
logging cleanup
|
2025-04-17 09:36:13 -06:00 |
|
|
|
647c4cd324
|
work off content-type header
|
2025-04-17 09:35:57 -06:00 |
|
|
|
7fab961d76
|
no longer how this is working
|
2025-04-17 09:35:26 -06:00 |
|
|
|
d3fff194f4
|
logging updates
|
2025-04-17 08:17:37 -06:00 |
|
|
|
3497312fd4
|
de-enshitified file saving logic
|
2025-04-17 08:17:29 -06:00 |
|
|
|
bdb1094a30
|
steam data to the disk
|
2025-04-15 13:07:47 -06:00 |
|
|
|
c08a20ac00
|
cleanup and more accuratly use metrics
|
2025-04-15 09:07:16 -06:00 |
|
|
|
94912e9125
|
change up how files are discovered
|
2025-04-15 09:06:57 -06:00 |
|
|
|
add6f00ed6
|
no recomp needed
|
2025-03-31 14:53:10 -06:00 |
|
|
|
4a433a1a77
|
This function sometimes throws errors, this logging should help
|
2025-03-31 14:18:37 -06:00 |
|
|
|
96a3ca092a
|
:)
|
2025-03-21 12:11:05 -06:00 |
|
|
|
b750d88d48
|
working filesystem storage
|
2025-03-21 11:42:43 -06:00 |
|
|
|
808790a7c3
|
file patch;
|
2025-03-21 07:11:51 +00:00 |
|
|
|
2de01b2a0e
|
remove removed code
|
2025-03-21 06:48:39 +00:00 |
|
|
|
be0fd5505b
|
i think the files work better
|
2025-03-21 06:48:17 +00:00 |
|
|
|
a23429104c
|
dead code removal
|
2025-03-21 06:03:34 +00:00 |
|
|
|
66581cc453
|
getting there
|
2025-03-21 05:59:40 +00:00 |
|
|
|
7df19a480f
|
updates
|
2025-03-20 15:11:01 -06:00 |
|
|
|
71b7b2d7bc
|
it works and it is awesome
|
2025-03-19 15:04:00 -06:00 |
|
|
|
6409baaffb
|
Reducted trips to surreal by x500
|
2025-03-19 12:41:08 -06:00 |
|
|
|
9aa34b3eee
|
epic metrics
|
2025-03-19 04:59:50 +00:00 |
|
|
|
de80418c00
|
better logging
|
2025-03-18 16:09:46 -06:00 |
|
|
|
e3e4175f51
|
logging improvements
|
2025-03-18 15:25:56 -06:00 |
|
|
|
d11e7dd27c
|
the biggest 1 line improvement ever
|
2025-03-18 15:25:40 -06:00 |
|
|
|
f2a3e836a0
|
spelling and clippy
|
2025-03-18 15:08:29 -06:00 |
|
|
|
3b4e6a40ce
|
minimize vec resizing
|
2025-03-18 15:07:50 -06:00 |
|
|
|
bd0b946245
|
fixed tracing
|
2025-03-18 15:02:32 -06:00 |
|
|
|
b7540a4680
|
checkpoint - onto profiling
|
2025-03-18 10:53:06 -06:00 |
|
Oliver Atkinson
|
82929fd0fc
|
updating for base64
|
2024-12-13 13:28:24 -07:00 |
|
Oliver Atkinson
|
298ad39a79
|
rename
|
2024-12-12 14:59:54 -07:00 |
|
Oliver Atkinson
|
215056e493
|
use contains operator for better output
|
2024-12-12 14:26:49 -07:00 |
|
Oliver Atkinson
|
22be3b2f61
|
updating deps
|
2024-12-12 14:14:38 -07:00 |
|
Oliver Atkinson
|
c1c8cf07bb
|
unifed settings for testing
|
2024-12-12 11:42:07 -07:00 |
|
|
|
0f8a3d7215
|
using a custom parser now :)
|
2024-11-12 23:08:09 -07:00 |
|
|
|
eaa79b749e
|
prepare get function for s3
|
2024-11-12 21:19:05 -07:00 |
|
|
|
2c28d69d55
|
add s3 support
|
2024-11-12 21:03:58 -07:00 |
|
|
|
d28d18de08
|
formatting and timer changes, consolidated functions
|
2024-11-12 18:40:10 -07:00 |
|
|
|
7cac880f8e
|
remove un-used function
|
2024-11-12 17:50:28 -07:00 |
|
|
|
720adaa552
|
added support for nearly all html tags that can have a link
|
2024-11-12 17:50:06 -07:00 |
|
|
|
399510c599
|
use reqwest client for epic speedup
|
2024-11-10 20:37:00 -07:00 |
|
|
|
ec66c4e765
|
remove unused import
|
2024-11-10 20:36:39 -07:00 |
|
|
|
a9628ee5e4
|
working, now onto speeding it up
|
2024-11-10 20:24:04 -07:00 |
|
|
|
5404d5c3e8
|
it works :party:
|
2024-11-09 23:30:57 -07:00 |
|
|
|
fd971bafbf
|
it works now
|
2024-11-09 15:28:10 -07:00 |
|
|
|
c3997b0bb7
|
works more, but still not all the way
|
2024-11-09 11:30:32 -07:00 |
|
Oliver Atkinson
|
7826c4cec6
|
jank-ish fix but it sure does work
make the root record (for links https://example.com/) have a record id of the url, thus preventing duplication when using upsert
|
2024-10-31 15:32:37 -06:00 |
|
Oliver Atkinson
|
3a46dd937b
|
updates
|
2024-10-31 15:09:48 -06:00 |
|
Oliver Atkinson
|
fbca067b1f
|
clean up walk()
|
2024-10-31 14:10:14 -06:00 |
|
Oliver Atkinson
|
9324160e74
|
crawling 🕷️
|
2024-10-07 11:14:56 -06:00 |
|
Oliver Atkinson
|
974bccc457
|
no longer using spider, just wiritng my own crawler
|
2024-10-04 13:52:34 -06:00 |
|
