Open Source GRC, Part 1: Nobody Ships the Checks

Sebastian GalonskaNov 17, 2026
A cheerful compliance officer in a patched cardigan and taped glasses hugs a rusty filing cabinet labeled FREE, whose bottom drawer labeled CHECKS hangs open and empty, while a worried paper ghost floats nearby.

If you clicked on this, you either run compliance for a living and want to know if you can stop paying for your platform, or you build one of these tools and want to see what I got wrong. Welcome to both of you. Corrections are welcome, I mean that.

Short version for the impatient: I looked at Probo and Comp AI, and neither one gives me what makes a compliance platform worth using. The automated checks.

One scope note before anyone emails me. I am not talking about the full paid products from these vendors, the hosted versions that do run checks for you. This is the cheapskate route: what you get when you take the free open source code and run it yourself.

Renting Is Getting Old

Auditbahn runs compliance for SaaS and AI companies. I do the work, the customer gets the certificate. Today that work sits on top of a commercial compliance platform, and every customer brings a licence with it.

That bothers me for a simple reason. The platform is the part of my service I understand least, and it is the one part I don't own. When a test is wrong, I open a ticket. When I want a check that doesn't exist, I wait. So I spent some time with two open source projects that promise to replace those platforms: Probo and Comp AI.

My yardstick is simple. Automated checks are the sauce. They are what makes compliance digestible: the tool looks at your AWS account, your GitHub org and your laptops every day, and tells you what is broken. Everything else, the policies and the control library, is a filing cabinet. And a filing cabinet can also be Excel. If a tool only gives me that, it delivers no value I can't get from a spreadsheet and a shared drive.

A thrifty compliance officer ticks boxes by hand on a whiteboard while eyeing a glowing, roped-off AUTO CHECKS machine with a very large price tag

What I Actually Did

Let me be honest about the depth here, because a lot of "we evaluated X" posts are a screenshot of a README.

I installed Probo v0.334.0 on my own server and put no customer evidence in it. Everything else I say about Probo comes from reading the source.

I installed Comp AI Community Edition too, built from source on the same server, together with the self-hosted Trigger.dev stack it depends on. Neither instance holds customer data.

For the checks, I read Prowler's compliance mappings and compared both tools against the automated test catalog of the commercial platform I use, at the level of categories and counts.

Repo numbers are from the GitHub API on November 7, 2026.

Probo: A Very Good Filing Cabinet

Probo is MIT licensed, YC backed, about 1,500 stars and 45 contributors. It ships fast. Server v0.300 came out on October 2, v0.340 on November 4. That is forty minor versions in about a month, which is great for features and exciting for whoever runs the upgrades (me).

The data model is solid. ISO 27001:2022 comes with all 93 Annex A controls plus 30 clause items, and there are twelve more frameworks, from SOC 2 to DORA and ISO 42001. Policies have approval and e-signature, there is an employee portal, risk register, vendor register, audits, findings, a Statement of Applicability, a public compliance page (their word for a trust center) and an audit log. Access reviews have drivers for about 80 systems. AWS connects through OIDC federation, so there are no static keys lying around. There is a CLI, GraphQL and an MCP server. If you like to script your compliance, you will feel at home.

There is also a device agent with nine checks per laptop: disk encryption, screen lock, firewall, time sync, OS version, auto update, password policy, remote login and malware protection. Each one reports a result for the device.

Here is the catch. Those results are not linked to any control in the data model. You export the device evidence as a PDF and attach it yourself.

And that pattern repeats. Controls have a testingCadence and a nextTestDue, which sound promising until you notice they are a repeat interval and a due date. Nothing in the code runs a test. Webhooks exist for documents, tasks, vendors and users, but not for controls or evidence. Probo tells you when a control should be tested and leaves the testing to you.

Is that nothing? No. Approval flows, e-signatures, an audit log and a clean control library are more than Excel gives you, and they are nicely built. But they are the filing cabinet part. By my yardstick, that makes Probo an Excel upgrade.

Two more things a buyer should know. Every evidence file you upload is queued and sent to the configured LLM provider to write a description. Leaving out the LLM API key does not turn that off: without a configured provider (OpenAI by default, Anthropic if you switch it) the server refuses to start. I found no setting that switches the describer off. For screenshots of a customer's admin console, that is a data processing question you want answered before the first upload. Also, I found no built-in MFA in the code, so plan on SSO.

To be fair: Probo Inc sells its own managed compliance service, starting at $10K a year according to their own comparison page. So the open source project is the platform behind a service much like mine. That explains the shape. A human does the testing.

Comp AI: The One With Checks

Comp AI is AGPL-3.0, about 2,100 stars and 40 contributors. Its seed data is generous: ISO 27001 with all 93 Annex A controls, SOC 2, HIPAA, ISO 42001 and more, plus 204 control templates, 52 policy templates and 148 task templates. And unlike Probo, it actually runs checks against AWS, Azure, GCP, GitHub and Google Workspace.

Installing it was a project of its own.

First, the project changed shape while I was reading it. Since October 22 the README calls the repo "Comp AI Community Edition" and says Comp AI Cloud is "built and operated as its own product on a separate codebase". The last GitHub release is v3.111.1 from August, while commits keep coming. The integrations catalog in the repo lists 590 integrations and says the check definitions (the DSL with endpoints and parsing logic) are intentionally excluded. The engine that runs those definitions is in the repo. The definitions are not, and I found no seed data for them. You get the engine and an empty rule book.

Second, self-hosting. There are no published images, so you build from source. The upstream compose file has no service for the NestJS API that the app needs. The docs list a Trigger.dev cloud account as a prerequisite, and about 170 files reference the Trigger.dev SDK. Running Trigger.dev yourself added nine containers to my server, one of which gets a Docker API that can start arbitrary containers on the host. Then, between October 24 and November 4, Comp moved its SDK from 4.4.3 to 4.7.7, so the Trigger.dev version you run is a moving target too.

Third, the first start. The API crashes at boot without a MACED_API_KEY, the key for a built-in pentest module from a commercial vendor. It is in no doc I found. Upstream's own example env file ships a dummy value, which is what I used, and the pentest feature stays dead. After that the stack came up. The seeded database holds exactly the counts above, and the API log says "Loaded 0 dynamic integrations". That is the empty rule book in practice: the catalog-driven integrations have no definitions behind them. Real check code ships only for nine hand-written manifests (AWS, Azure, GCP, GitHub, Google Workspace and four more).

None of that is a crime. It is a team building a commercial cloud product and publishing a community edition. It just means "self-host it" is a project of its own, and the rule book stays with them. Of the two, Comp AI is closer to the sauce. It is also harder to run.

The Checks

This is the part I care about most, because it is where a compliance platform earns its money.

A real control test engine does a handful of boring things reliably. It has a catalog where each test says:

  • which data it reads (an AWS account, a GitHub org, the platform's own records like "was the quarterly access review completed")
  • the rule that makes it pass or fail
  • which assets failed and why
  • which controls it supports
  • how fresh the evidence is, and what happens when it expires

Then it runs on a schedule, flips to failed when things break, tells somebody, and treats a missing result as a failure instead of a pass. The commercial platform I use has three kinds of these: integration tests (around 48 for every connected AWS account alone), tests on its own data, and manual tests where a person uploads proof on a due date.

Probo has none of this. Its connectors exist to feed access reviews.

Comp AI has two separate check paths, and they behave differently.

The AWS cloud scan covers about 45 AWS services. When every finding for a service passes, it marks the linked tasks done. The code comment is refreshingly direct: "Only pass → done. Never mark tasks as failed from scan data." It also skips tasks that are already done, and when a task is linked to several services (Encryption at Rest is fed by KMS, S3, RDS and DynamoDB), it only looks at the services that were actually scanned. Put together: once a task is green, a later failing scan never makes it red again, and a service nobody scanned counts as fine.

The second path, the integration checks, is better. 44 checks across AWS, Azure, GCP, GitHub, Google Workspace and a few others can set a task to failed and send an email when that happens. Credit where it is due. But those 44 checks feed 14 of the 148 task templates. Findings name the failing resource, and the status then lands on a broad task like Encryption at Rest, which rolls up to controls.

And then the uncomfortable bit. A green check is not audit evidence. A scanner tells you a setting was correct at the moment it looked. An auditor wants to see that a control operated over a period, that someone reviewed exceptions, and that failures were noticed and fixed. A check that can only ever turn green, or that ignores what it didn't scan, produces exactly the kind of evidence that makes an auditor ask more questions. In a commercial tool, that is the vendor's problem. Self-hosted, it is yours.

Rolling Your Own Tests

If nobody ships the checks, you build the layer yourself. Here is the design I wrote down. I have not built it.

The scanner is Prowler, a separate Apache 2.0 project with about 15,000 stars and about 680 AWS checks. Neither Comp AI nor Probo includes it. It also ships an ISO 27001:2022 mapping for AWS, and that is where it gets rough. The file has 92 requirements, because A.8.28 (secure coding) is missing entirely. 37 of the 92 have at least one check. That is 40 percent. People controls (A.6) and physical controls (A.7) have zero, which is fair, a cloud scanner cannot see your clean desk. Less fair: A.8.8 (technical vulnerabilities) has no checks, even though Prowler has Inspector checks that would fit. A.5.23 (security for cloud services) has none either, in a cloud scanner. Meanwhile A.8.15 (logging) gets 72 checks.

So even with the best open source scanner wired in perfectly, 60 percent of ISO 27001 Annex A has no automated check for AWS. The mapping you do get, you will want to own and fix yourself.

Probo stays the system of record for controls, evidence, tasks and documents. Prowler scans. In between sits a small runner that reads Prowler's results and writes them into Probo: set the control's implementation status, attach the result as evidence, open a task for every failing test.

The part that makes it a test engine is a catalog file in the repo. One entry per test: which Prowler checks it uses, the pass rule, the ISO control it supports, how often it runs. Changes go through a pull request. That file is the thing you own, and it fixes Prowler's rough mapping.

Four rules go into the runner:

  • Missing scan results count as a failure, never a pass.
  • A control with several tests takes the worst result.
  • Prowler has no check for some things you need (RDS backup retention, Sentry 2FA, Slack), so those get small custom checks, only where a control demands one.
  • Tests on your own data, like "quarterly access review completed" or "risk register exists", read Probo's API.

What this still does not give you: that other 60 percent of Annex A. People, physical, most policy work. Those stay manual tests with an upload and a due date, and you still need to write the policies. It also leans on a Probo API that changes almost weekly, so you pin versions and test the runner.

Side By Side

ProboComp AI (Community Edition)The gap
LicenceMITAGPL-3.0, separate proprietary cloud
What I didInstalled, no real dataInstalled, built from source, no real data
ISO 27001:2022 content93 Annex A + 30 clause items93 Annex A + 26 clause items
Policy templatesNone bundled that I found52
Automated control testsNoneAWS scan (only turns tasks done) plus 44 checks that can fail tasksA catalog that maps checks to controls, fails loudly, and counts missing data as a failure
Device posture9 checks, not linked to controlsDevice agent exists, not evaluatedLink from posture to control
Tests on own data (access review done, risk register present)NoneNot evaluatedNeeds building in both
Self-hosting effortOne app image plus Postgres, S3 storage and Chrome; Compose or HelmBuild from source, plus Trigger.dev (nine more containers)
Scanner coverage (Prowler, AWS, ISO 27001)n/an/a37 of 92 requirements have any check

Part 2

Part 2 covers the rest of the open source field. I am looking for one project that ships the checks.

About The Author

Sebastian is a seasoned CTO and startup founder, from building Germany's largest social network to developing fintech startups. He coaches young leaders here at cto.berlin, and runs compliance (SOC 2, ISO 27001) for SaaS and AI companies at Auditbahn.

For individual coaching sessions, just book your call down below.

Let’s talk. One call. Zero fluff.

Contact Information
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.