jarink/brokenlinks/brokenlinks_test.go, branch mainProgram to inspects and maintains web sites.http://git.kilabit.info/jarink/atom?h=main2026-02-11T18:04:40Zbrokenlinks: store the anchor or image source in link2026-02-11T18:04:40ZShulhanms@kilabit.info2026-02-11T18:04:40Zurn:sha1:9c7ee77376294e9abd70ca356e26d0ab16ad7466
In the struct Link, we add field Value that store the href from A element
or src from IMG element.
This allow us to debug any error during scan, especially joining path
and link.
brokenlinks: make link that return HTML always end with slash2026-02-11T14:45:06ZShulhanms@kilabit.info2026-02-11T03:47:42Zurn:sha1:8100b3be0730173a77f1a64f9ac6bc8862a159ac
If parent URL like "/page" return the body as HTML page, the URL should
be end with slash to make the relative links inside it works when joined
with the parent URL.
all: mark and skip the slow test2026-01-21T21:15:47ZShulhanms@kilabit.info2026-01-21T21:15:47Zurn:sha1:a33efc3992f58355eb98d7a5574df955952924b8
The TestScan_slow takes around ~11 seconds due to test include
[time.Sleep].
brokenlinks: print the progress to stderr2026-01-21T18:51:52ZShulhanms@kilabit.info2026-01-21T18:51:52Zurn:sha1:2a4376d5ddeee82d4ef38f4953453abb43e85220
Each time the scan start, new queue add, fetching start, print the
message to stderr.
This remove the verbose options for better user experience.
all: refactoring, use single struct to represent Link2026-01-21T18:39:41ZShulhanms@kilabit.info2026-01-21T18:39:41Zurn:sha1:79eaccc81b85eb92dab9cf18d52662f367903652
Previously, have [jarink.Link], [brokenlinks.Broken], and
[brokenlinks.linkQueue] to store the metadata for a link.
These changes unified them into struct [jarink.Link].
brokenlinks: refactoring the logic, simplify the code2026-01-21T17:27:18ZShulhanms@kilabit.info2026-01-21T17:27:18Zurn:sha1:26fc8bd3203dae6b4705ada227439c90129bbe36
Previously, we made the scan logic to run in multiple goroutine with
one channel to push and consume the result and another channel to push
and pop link to be processed.
The logic is a very complicated code, making it hard to read and debug.
These changes refactoring it to use single goroutine that push and pop
link from/to a slices, as queue.
brokenlinks: fix infinite loop on unknown host2025-11-20T10:12:19ZShulhanms@kilabit.info2025-07-01T19:07:43Zurn:sha1:5301c666eec35699bbb9024678bb37adc057404c
On link with invalid domain, it should break and return the error
immediately.
brokenlinks: implement caching for external URLs2025-06-27T05:19:23ZShulhanms@kilabit.info2025-06-21T08:20:01Zurn:sha1:1ca561ed0ecfa59b70a10191ac8e58cde90d126e
Any succesful fetch on external URLs, will be recorded into jarink
cache file, located in user's home cache directory.
For example, in Linux it would be `$HOME/.cache/jarink/cache.json`.
This help improve the future rescanning on the same or different target
URL, minimizing network requests.
all: add test cases for simulating slow server2025-06-18T18:06:48ZShulhanms@kilabit.info2025-06-18T18:06:48Zurn:sha1:8bc8fce1bd80b5a25c452ac5a24b1a1e3f5a4feb
The test run a server that contains three six pages that contains
various [time.Sleep] duration before returning the response.
This allow us to see how the main scan loop works, waiting
for resultq and listWaitStatus.
brokenlinks: add test cases for IgnoreStatus options2025-06-17T16:49:18ZShulhanms@kilabit.info2025-06-17T16:49:18Zurn:sha1:5ac82928702a0f0a7be0ef6e96ab04c39a7e8e9d
There are two test cases, one for invalid status code like "abc",
and one for unknown status code like "50".