Sorting PR Checks by Their Status

If you’re only after a solution that’s going to sort the checks on your PRs, install the Sorted PR Checks action we developed in your repository, enjoy beautiful comments on all your PRs, and forget about it.

However, if you want to learn more about the issue itself, how the solution came to be, its’ customisation options, or alternatives, read on!

Who even needs sorted PR checks?

Have you ever worked on a codebase where CI produces tens, hundreds, or even thousands of checks? If so, you might have noticed one pretty annoying shortcoming of GitHub UI - it doesn’t sort PR checks by their status. In fact, it doesn’t sort them by any useful property!

Our investigation into this suggests that GitHub tries to order the checks by their creation date, but even that is not always exact. Nevertheless, even if that order were guaranteed, it wouldn't help anyone trying to find a single failure among a multitude of checks.

That’s the core of the issue; it is way too hard to find the interesting bits that you’re looking for under your PR - failures. Right now, it’s like looking for a needle in a haystack.

Developers have been asking for this feature for almost 3 years. The issue raised in the GitHub Discussions has accumulated 60 upvotes to date, which is a considerable amount for the forum. Unfortunately, there is no native fix in sight.

Some of our sorting attempts failed miserably

First, we tried to “game” the GitHub UI. We thought that maybe, just maybe, it would sort the checks by their last update time. Then, we could wait for all the checks to finish and update them in the order we deemed correct. Unfortunately, that didn’t work. As mentioned before, GitHub UI seems to be trying to maintain the check creation time order, which is not as helpful.

We haven’t given up on that general approach yet. Equipped with that knowledge, we thought that we might be able to recreate all the checks once they’re all finished, starting with the failed ones. We weren’t too thrilled about this, though. Deleting and then recreating checks sounds error-prone. Nonetheless, we did further research to see if it was even possible. Turns out, it’s not as easy as it sounds. In the end, we weren’t able to recreate the checks using the Create a check run in the same way GitHub Actions creates them. They would look different, link to different sections of GitHub UI than the original, etc. It was all too much, and we decided to move on.

Sticky comments to the rescue

Because we couldn’t modify the existing UI the way we wanted to, we decided to turn our eyes to other UI elements we could repurpose as checks summaries. Comments were the natural next best thing. Both creators and reviewers can see and interact with them; they are visible on the exact same page as the native check summaries are; they are perfect!

We decided it would be helpful if a comment saying that the checks are in progress appeared when the CI starts. This would let developers know early that they can expect results to appear in a comment once they are available. Creating and managing expectations well is a crucial characteristic of a good Developer Experience.

Once the CI finishes, we wanted the existing comment to be updated with the results. By updating the existing comment, we avoid creating unnecessary noise in the PR conversation flow, making the comment “sticky”.

Finally, we wanted the content to be a list of linked checks sorted by their status — exactly what we were missing from the GitHub UI in the first place.

Luckily, GitHub REST API allowed us to accomplish all of the above. We could:

• Create a comment
• Find the comment to update by listing all existing comments and looking for one with our secret identifier in the comment section
• Prepare content by listing all checks for the PR and sorting them by their status
• Update the comment

Events caused by GITHUB_TOKEN do not propagate

Now that we knew how to get the information to the developers, we had to answer one final crucial question — when to do it. There are plenty of GitHub Actions events we could react to.

check_run seemed very promising. Events of this type correspond 1 to 1 with all the PR status check changes, which means the comment could be updated as often as the native PR check summaries are. There might be plenty of them, but with sane deduplication in place (concurrency groups), it doesn’t need to be that big of an issue.

But…

Events caused by the default GITHUB_TOKEN issued to GitHub Actions workflow runs do not trigger subsequent GitHub Actions workflow runs 😭 In other words, check_run doesn’t trigger when you’re trying to react to checks created by GitHub Actions. It only triggers on checks created by a Personal Access Token (PAT) or GitHub Apps.

Fortunately, the same doesn’t apply to workflow_run events. These are issued whenever your workflow run starts or finishes. This provided us with a lower granularity signal since a single workflow run can consist of many jobs, all of which create individual checks, but it was good enough. It allowed us to accomplish our main goal - present developers with a sorted PR checks summary once all the workflow runs for the PR are finished.

I want it now!

All of the above comes together in the form of a GitHub Actions action called Sorted PR Checks. You can easily add it to your own repository and enjoy comments with sorted PR checks under your PRs.

But that’s not all! You can also customise the comment content! You can choose between one of the existing templates:

• Checks sorted by result
• Checks grouped by result
• Only unsuccessful checks sorted alphabetically

Or, you can create a completely new comment template yourself!