jarink/brokenlinks/worker.go, branch mainProgram to inspects and maintains web sites.http://git.kilabit.info/jarink/atom?h=main2026-02-11T18:07:04Zbrokenlinks: fix possible panic in markAsBroken2026-02-11T18:07:04ZShulhanms@kilabit.info2026-02-11T18:05:52Zurn:sha1:e4b3383d84cd181f4a9fec1097ec61e583d6f827
If the Link does not have parentUrl, set the parent URL using the link
URL itself.
This only happened if the target URL that we will scan return an error.
brokenlinks: store the anchor or image source in link2026-02-11T18:04:40ZShulhanms@kilabit.info2026-02-11T18:04:40Zurn:sha1:9c7ee77376294e9abd70ca356e26d0ab16ad7466
In the struct Link, we add field Value that store the href from A element
or src from IMG element.
This allow us to debug any error during scan, especially joining path
and link.
brokenlinks: make link that return HTML always end with slash2026-02-11T14:45:06ZShulhanms@kilabit.info2026-02-11T03:47:42Zurn:sha1:8100b3be0730173a77f1a64f9ac6bc8862a159ac
If parent URL like "/page" return the body as HTML page, the URL should
be end with slash to make the relative links inside it works when joined
with the parent URL.
brokenlinks: skip processing "mailto:" URL2026-02-11T14:45:06ZShulhanms@kilabit.info2026-02-10T21:45:04Zurn:sha1:c92d49c41e27d50a36fd6eba0824331606627132brokenlinks: check if link has been seen before scan2026-02-10T21:38:20ZShulhanms@kilabit.info2026-02-10T21:38:20Zurn:sha1:61eb5351087be894a4bfeb71c99346b8065bb7f1
Given the following queue and its parent,
/page2.html => /index.html
/brokenPage => /index.html
/brokenPage => /page2.html
Before scanning the second "/brokenPage" on parent page "/page2.html",
check if its seen first to get the status code before we run the scan.
This allow jarink report "/brokenPage" as broken link for both pages,
not just in "/index.html".
brokenlinks: check for redirect during scan2026-02-04T20:29:49ZShulhanms@kilabit.info2026-02-04T20:29:49Zurn:sha1:d8a892eb2f28b3ef4c2625c682d255f4f616cae2
If the request redirected, use the "Location" value in the response
header as the parent URL instead of from the original link in queue.
brokenlinks: fix generating relative URL2026-02-04T15:58:10ZShulhanms@kilabit.info2026-02-04T15:10:42Zurn:sha1:fa31e0a656d03fe3744c70a1171e3831647923c9
If the parent URL end with .html or .htm, join the directory of parent
instead of the current path with the relative path.
brokenlinks: skip parsing non-HTML page2026-02-04T14:53:00ZShulhanms@kilabit.info2026-02-04T14:53:00Zurn:sha1:b36d6e1f423bc405895d1b72e9a5915c4aa74ecc
If the response Content-type return other than "text/html", skip parsing
the content and return immediately.
brokenlinks: mark the link in queue as seen with status code 02026-02-04T10:03:47ZShulhanms@kilabit.info2026-02-04T10:03:47Zurn:sha1:6b5ed409a5f11ed437586c8b046bcfc43749361d
This is to fix double URL being pushed to queue.
brokenlinks: improve fetch logging and decrease timeout to 10 seconds2026-01-21T21:14:25ZShulhanms@kilabit.info2026-01-21T21:14:25Zurn:sha1:a559b47dc217793ca7f80121b3ea86f03e47afd3
When fetching, print log after the fetch completed.
If success, print the URL along with HTTP status code.
If fail, print the URL along with the error.
The timeout now reduce to 10 seconds to prevent long delay when working
with broken website.