Check for broken external and internal links.

fastlinkcheck checks for broken links in HTML documents. This occurs in parallel so performance is fast. Both external links and internal links are checked. Internal links are checked by verifying local files.

To get started, read the documentation.

Install

pip install fastlinkcheck

Usage

After installing fastlinkcheck, the cli command link_check is available from the command line. We can see various options with the --help flag.

link_check --help

usage: link_check [-h] [--host HOST] [--config_file CONFIG_FILE] [--pdb]
[--xtra XTRA]
path

Check for broken links recursively in path.

positional arguments:
path                  Root directory searched recursively for HTML files

optional arguments:
-h, --help            show this help message and exit
--host HOST           Host and path (without protocol) of web server
--config_file CONFIG_FILE
Location of file with urls to ignore
--pdb                 Run in pdb debugger (default: False)
--xtra XTRA           Parse for additional args (default: '')


We can search the directory _example/broken_links in this repo recursively for broken links like this:

link_check _example/broken_links


ERROR: The Following Broken Links or Paths were found:

- 'http://fastlinkcheck.com/test.html' was found in the following pages:
- /Users/hamelsmu/github/fastlinkcheck/_example/broken_links/test.html

- 'http://somecdn.com/doesntexist.html' was found in the following pages:
- /Users/hamelsmu/github/fastlinkcheck/_example/broken_links/test.html

- /Users/hamelsmu/github/fastlinkcheck/_example/broken_links/test.html

Specifying the --host parameter allows you detect links that are internal by identifying links with that host name. External links are verified by making a request to the appropriate website. On the other hand, internal links are verified by inspecting the presence and content of local files.

We must be careful when using the --host argument to only pass the host (and path, if necessary) without the protocol. For example, this is how we specify the hostname if your site's url is http://fastlinkcheck.com/test.html:

link_check _example/broken_links --host fastlinkcheck.com


ERROR: The Following Broken Links or Paths were found:

- 'http://somecdn.com/doesntexist.html' was found in the following pages:
- /Users/hamelsmu/github/fastlinkcheck/_example/broken_links/test.html

- /Users/hamelsmu/github/fastlinkcheck/_example/broken_links/test.html

We now have one less broken link as there is indeed a file named test.html in the root of the path we are searching. However, if we add a path to the end of --host , such as fastlinkcheck.com/mysite the link would again be listed as broken because _example/broken_links/mysite/test.html does not exist:

link_check _example/broken_links --host fastlinkcheck.com/mysite


ERROR: The Following Broken Links or Paths were found:

- 'http://fastlinkcheck.com/test.html' was found in the following pages:
- /Users/hamelsmu/github/fastlinkcheck/_example/broken_links/test.html

- 'http://somecdn.com/doesntexist.html' was found in the following pages:
- /Users/hamelsmu/github/fastlinkcheck/_example/broken_links/test.html

- /Users/hamelsmu/github/fastlinkcheck/_example/broken_links/test.html

You can ignore links by creating a text file that contains a list of urls and paths to ignore. For example, the file _example/broken_links/linkcheck.rc contains:

cat _example/broken_links/linkcheck.rc

test.js


We can use this file to ignore urls and paths with the --config_file argument. This will filter out references to the broken link /test.js from our earlier results:

link_check _example/broken_links --host fastlinkcheck.com --config_file _example/broken_links/linkcheck.rc


ERROR: The Following Broken Links or Paths were found:

- 'http://somecdn.com/doesntexist.html' was found in the following pages:
- /Users/hamelsmu/github/fastlinkcheck/_example/broken_links/test.html

Finally, if there are no broken links, link_check will not return anything. The directory _example/no_broken_links/ does not contain any HTML files with broken links:

link_check _example/no_broken_links

No broken links found!


Python

Here is an example of how you can use this utility in GitHub Actions:

name: Check Links
on: [workflow_dispatch, push]

jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/[email protected]
- uses: actions/[email protected]
- name: check for broken links
run: |


We can a few more lines of code to open an issue instead when a broken link is found, using the gh cli:

...
- name: check for broken links
run: |
[[ -s err ]] &&  gh issue create -t "Broken links found" -b "$(< err)" -R "yourusername/yourrepo"  We can extend this even further to only open an issue when another issue with a specific label isn't already open: ... - name: check for broken links run: | pip install fastlinkcheck link_check "docs/_site" --host "docs.fast.ai" 2> err || true export GITHUB_TOKEN="YOUR_TOKEN" if [[ -z$(gh issue list -l "broken-link")) && (-s err) ]]; then