custom_engine #1

Merged

Oliver merged 16 commits from custom_engine into main 2024-11-13 00:53:31 +00:00

Conversation 0 Commits 16 Files Changed 19 +3302 -963

Oliver commented 2024-11-13 00:53:19 +00:00

Owner

Copy Link

wrote a custom scraping engine instead of using spider-rs. This makes the project more flexible for my needs

wrote a custom scraping engine instead of using spider-rs. This makes the project more flexible for my needs

Oliver added 16 commits 2024-11-13 00:53:19 +00:00

no longer using spider, just wiritng my own crawler 974bccc457

crawling 🕷️ 9324160e74

clean up walk() fbca067b1f

updates 3a46dd937b

jank-ish fix but it sure does work ... 7826c4cec6

make the root record (for links https://example.com/) have a record id of the url, thus preventing duplication when using upsert

works more, but still not all the way c3997b0bb7

it works now fd971bafbf

it works :party: 5404d5c3e8

working, now onto speeding it up a9628ee5e4

remove unused import ec66c4e765

use reqwest client for epic speedup 399510c599

this is really all that's needed 7c32600694

added support for nearly all html tags that can have a link 720adaa552

remove un-used function 7cac880f8e

added demo site for testing elements f87a43c3a9

fits with compose.yml d9d4c56142

Oliver merged commit 8a5ac61b26 into main 2024-11-13 00:53:31 +00:00

Oliver referenced this issue from a commit 2024-11-13 00:53:33 +00:00

Merge pull request 'custom_engine' (#1) from custom_engine into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No Label

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

1 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Oliver/internet_mapper#1

No description provided.