Oliver/internet_mapper
1
0
Fork 0
Code Issues 11 Pull Requests Actions Packages Projects Releases Wiki Activity

custom_engine #1

Merged
Oliver merged 16 commits from custom_engine into main 2024-11-13 00:53:31 +00:00
Conversation 0 Commits 16 Files Changed 19 +3302 -963
Oliver commented 2024-11-13 00:53:19 +00:00
Owner
Copy Link

wrote a custom scraping engine instead of using spider-rs. This makes the project more flexible for my needs

wrote a custom scraping engine instead of using spider-rs. This makes the project more flexible for my needs
Oliver added 16 commits 2024-11-13 00:53:19 +00:00
no longer using spider, just wiritng my own crawler 974bccc457
crawling 🕷️ 9324160e74
clean up walk() fbca067b1f
updates 3a46dd937b
jank-ish fix but it sure does work 7826c4cec6 
make the root record (for links https://example.com/) have a record id of the url, thus preventing duplication when using upsert
works more, but still not all the way c3997b0bb7
it works now fd971bafbf
it works :party: 5404d5c3e8
working, now onto speeding it up a9628ee5e4
remove unused import ec66c4e765
use reqwest client for epic speedup 399510c599
this is really all that's needed 7c32600694
added support for nearly all html tags that can have a link 720adaa552
remove un-used function 7cac880f8e
added demo site for testing elements f87a43c3a9
fits with compose.yml d9d4c56142
Oliver merged commit 8a5ac61b26 into main 2024-11-13 00:53:31 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Oliver
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Oliver/internet_mapper#1
No description provided.
Delete Branch "custom_engine"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?