We built a high-performance license classifier in pure Python

Our license scanner couldn't identify BUSL-1.1. A missing pattern doesn't look like a failure, it looks like a clean result.

At Fencer we scan dependencies, and part of that is license identification: for every package in a customer's tree, read the license text and work out which SPDX license it is. The license field in package metadata isn't good enough on its own, it's optional, it's often wrong, and it frequently says "BSD" when the file in the tarball is BSD-3-Clause.

For eight months we did that by calling a Go engine through cgo. Then we noticed it couldn't identify BUSL-1.1, the Business Source License, which covers Terraform, Vault, Sentry and CockroachDB, and which isn't an open-source license at all. The engine was fine. Its bundled list of licenses had stopped being updated.

So we ported the algorithm to Python, took over the license data, and open-sourced the result. licenseclassifier is pure Python, no dependencies, no native code, fully offline, about 2 ms per license file, and BSD-3-Clause, the same license as the project it descends from.

from licenseclassifier import identify_license

identify_license(open("LICENSE").read())
# [LicenseIdentificationResult(id='Apache-2.0', start=0, end=11324)]

Why we started in Go

The best license identifier we know of is google/licensecheck, the engine behind go-licenses and a good chunk of the Go ecosystem's compliance tooling.

Its approach is unusual, and it's why we picked it. Rather than scoring text similarity against a folder of license files, it compiles a corpus of license patterns into a single word-level matcher. The patterns are written in a small regexp-like DSL over words instead of characters, so a pattern can say "this paragraph, then any 20 words, then this clause". It reports which regions of your input matched, with character offsets, and it tolerates typos. A file with nine licenses stacked in it comes back as nine results.

It's Go-only. In November 2025 we wrapped it in cgo, built it as a shared library, and shipped it to our Python application as a package that loaded the .so through ctypes and passed results back as JSON.

The license list stopped being updated

The problem showed up as missing coverage. Licenses we expected to be identified came back unidentified.

The version we pinned, licensecheck v0.3.1, is still its latest release. The corpus it ships covers 423 SPDX identifiers, and the SPDX license list has grown since that corpus was assembled.

Some of the gaps are obscure enough that nobody would notice: Bitstream-Charter, Linux-man-pages-copyleft, Cornell-Lossless-JPEG. Others aren't.

BUSL-1.1 is what made this urgent. The Business Source License carries a usage restriction that converts to an open license on a set date, which makes it one of the licenses a compliance product most needs to flag, and we were returning "unidentified" for it. That's the same answer we return for a package with no license file at all. The failure arrived looking like ordinary output.

Why that meant a rewrite

Forking the Go module and shipping our own corpus was the obvious fix, and it's the one we rejected. It leaves us maintaining a Go fork to feed a Python application through a cgo bridge, with the license data, now a file we need to update regularly, sitting behind a build step involving the Go toolchain, a per-platform shared object, and a rebuild on every Python upgrade. Updating a JSON file shouldn't need a compiler.

Switching engines didn't help either. ScanCode is more thorough but much broader in scope, and the Rust and Ruby detectors put us back in the same cross-language position.

So we ported the algorithm into the language the rest of the product is written in, and kept the corpus as a data file we regenerate on our own schedule.

That's also why licenseclassifier uses CalVer (YYYY.MM.MICRO). Most of what changes between releases is the vendored SPDX data. The rest is the yearly pass over the Python release cycle: add the new interpreter each October, drop versions as they go end of life. Both are questions of how current the package is, which is what a date tells you and 1.4.2 doesn't.

The port, and how we knew it was right

licenseclassifier reimplements licensecheck v0.3.1's identification algorithm in four stages:

flowchart LR
    T[Tokenize<br/>case fold, strip accents,<br/>normalize ©, skip markup<br/>→ interned int IDs] --> P[Parse patterns<br/>LRE regexp-over-words DSL<br/>→ syntax tree]
    P --> M[Match<br/>compile to word-level bytecode,<br/>Thompson NFA + lazy DFA,<br/>leftmost-longest, spell-tolerant]
    M --> C[Cover<br/>word matches → char offsets,<br/>back-fill copyright lines,<br/>detect license URLs,<br/>compute coverage %]

Everything after tokenization operates on integers. Words are interned to IDs at the boundary, so the matcher never handles a str.

Two details from upstream matter for accuracy. The matcher does context-sensitive spell checking, single-character edits, joined words, split words, because real LICENSE files contain typos, and a matcher that demands verbatim text under-reports. And the cover layer applies a coverage threshold: unless the matched regions account for 75% of the input, it reports nothing. That's what keeps a README that mentions MIT from being classified as MIT.

>>> identify_license("This project is released under the MIT license. See LICENSE.")
[]

The port was written by an LLM, and we validated it against the engine it replaces. All 672 fixtures in licensecheck's testdata tree, asserting that the matched license IDs and the coverage percentages agree exactly. It passes on every one. Nobody was going to catch a subtle bug in a hand-reviewed Thompson NFA; a differential test against the reference implementation catches it or nothing does.

Four decisions that made it fast

Build the matcher once. The compiled program is a module-level singleton, constructed on first use and kept for the process lifetime.

Move the compile to build time. Compiling 551 patterns into a matcher takes about 1.1 s in Python, too slow to pay at import in every worker and web process. We do it at packaging time instead and ship the result as a marshal-serialized, gzipped artifact inside the wheel, which deserializes in about 9 ms. A version guard falls back to compiling from the vendored patterns if the artifact is missing or stale, and a test asserts the committed artifact matches a fresh compile.

Store the program as flat integer arrays. The bytecode lives in array('i') int32 arrays rather than lists of Python objects, roughly 3.5× smaller in memory and faster to load, since a million small int objects is expensive in CPython.

Keep the DFA lazy. States are built on demand and memoized, so a scan only materializes the states it visits.

Measured on an M-series laptop, Python 3.14, classifying a stock 11 KB Apache 2.0 file:

A 23 KB THIRD_PARTY file with nine stacked licenses takes 15.8 ms and returns all nine regions with offsets:

MIT               678 to 1764
NCSA             1845 to 3383
MIT              3628 to 4852
Apache-2.0       4941 to 16298
Zlib            16404 to 17310
Unlicense       17417 to 18627
BSD-2-Clause    18828 to 20214
BSD-3-Clause    20356 to 21868
BSD-2-Clause    21949 to 23251

The duplicate IDs are correct, that file contains two MIT texts and two BSD-2-Clause texts from different vendored components. Results are per-region, so deduplicate if you want a set.

At 2 ms per file, license identification costs far less than the network fetch that precedes it.

The 200× speedup was our bug, not Go's

Switching engines made our enrichment pipeline dramatically faster, and we want to be clear about why, because the obvious reading is wrong.

While benchmarking the port we found a bug in our own cgo wrapper:

//export IdentifyLicense
func IdentifyLicense(timeout int, licenseText *C.char) *C.char {
    ...
    identifier, err := identifier.NewDefaultIdentifier()   // <-- every call

NewDefaultIdentifier() compiles the entire pattern corpus into a word-level DFA, hundreds of thousands of states. licensecheck's API expects that to happen once per process. We were doing it once per package. The scan itself was sub-millisecond; the remaining ~435 ms was rebuilding the matcher, with no warm state to reach on the next call.

Same laptop, same fixture, both paths under Python 3.14:

On the real enrichment path, 80 packages went from 34.8 s to 0.17 s, 2.3 packages/second to 469.

Those numbers measure our bug. Caching the identifier in a package-level variable would have fixed the latency in about four lines, and Go would then be faster than what we shipped, because Go runs a DFA faster than Python does. licensecheck isn't slow. We were calling it wrong, and it was easy to call wrong: a ctypes boundary gives you nowhere obvious to cache anything, and no sense of whether the thing on the other side is warm.

Extracting it as a library

The port ran inside our monolith for a month as pylicense. Nothing in it was Fencer-specific, no models, no settings, nothing imported from outside the standard library.

The rename came first, because pylicense is taken on PyPI. It shipped as licenseclassifier.

Extraction was also when we fixed the things that were acceptable internally and wouldn't be in public.

The public API got smaller. Three names: identify_license, LicenseIdentificationResult, COVERAGE_THRESHOLD. We dropped the timeout argument, a leftover from the cgo era that had been a no-op since the port, and LicenseIdentificationError, which was never raised. We added a per-call threshold override:

# Accept files that embed a license alongside a lot of other prose.
identify_license(text, coverage_threshold=40.0)

# Demand a near-verbatim license file and nothing else.
identify_license(text, coverage_threshold=98.0)

The Python floor came down from 3.14 to 3.10. Internally we only ever ran 3.14, so the code had picked up a PEP 758 unparenthesized except tuple, a syntax error on anything older:

    except OSError, ValueError, EOFError, KeyError, TypeError:      # 3.14+ only
    except (OSError, ValueError, EOFError, KeyError, TypeError):    # everywhere

That plus from __future__ import annotations in three engine modules was the whole cost of supporting five more Python versions. It's now tested on 3.10 through the 3.15 prerelease, one CI job per version, using nox with uv-provided interpreters.

The test suite grew to cover a public contract. 346 tests at 100% line and branch coverage, enforced in CI and merged across the version matrix, a branch reachable on only one interpreter would otherwise look covered. Beyond the API surface it covers the canonicalization rules, typo and word-boundary tolerance, license-URL resolution, the pattern DSL and its handling of malformed input, region boundaries and copyright back-fill, and character-offset correctness on non-ASCII input.

The one we'd point at is the prebuilt-artifact check, which runs on every supported interpreter. The artifact is marshal-serialized, marshal isn't guaranteed portable across Python versions, and the runtime silently falls back to recompiling if it can't read one. A bad artifact wouldn't raise on the affected version. It would just cost 1.1 s at import, quietly, for however long it took someone to notice.

Releases go out through a PyPI Trusted Publisher on a v* tag, gated on the built wheel being installed and run against the full suite on every supported interpreter. No API tokens in CI secrets.

The license we shipped it under

licenseclassifier is BSD-3-Clause, matching google/licensecheck, whose algorithm it ports and whose pattern corpus it vendors verbatim.

Upstream's license text is preserved at third_party/licensecheck/LICENSE, the derivation is documented in NOTICE, and both ship inside the wheel under .dist-info/licenses/, which is what satisfies BSD clause 2 for a binary distribution. A file sitting in the GitHub repo doesn't reach anyone installing from PyPI. The distribution is uniformly BSD-3-Clause; only the copyright holders differ between parts. We're not affiliated with or endorsed by Google or the Go Authors.

One gap worth knowing about before you depend on it: the parity harness isn't vendored yet, because it needs licensecheck's Go testdata tree, so the 672-fixture result isn't reproducible from a clean checkout. It's the top item on the roadmap.

What we learned

In a vendored dependency, the data goes stale before the code does. The algorithm we adopted in 2025 is still the best one we know of. The license list shipped alongside it was the part with a shelf life, and it's the part nobody puts on a maintenance schedule. If a dependency bundles a corpus, a ruleset, or a signature database, find out when the data was last regenerated, not when the library was last released.

A missing pattern doesn't look like a failure. BUSL-1.1 coming back unidentified is indistinguishable from a package that ships no license, and those two cases deserve very different responses from a compliance workflow. Gaps in a detector's data arrive as clean results, which is what makes them survive.

Own the part that changes often. We didn't port the engine because we could write a better one, we ported it so the file we need to update regularly lives in a repo we control, in the language the rest of the product uses, updatable without a compiler.

Representation is where pure-Python performance comes from. Interned int IDs instead of strings, array('i') instead of object lists, lazy memoized states, and the expensive compile moved to build time. None of it is clever. All of it is the difference between 2 ms and unusable.

Check what your benchmark is measuring. We could have published a 200× speedup and left it there. The 200× was our own bug in a cgo wrapper, and the engine we replaced is still faster per scan than the one we wrote.

We set out to fix a coverage gap and came away with a library. That's the part worth repeating: the reason to own this code isn't that we can classify licenses better than Go can, it's that the list of licenses is now a file we can update on a Tuesday.

pip install licenseclassifier

licenseclassifier is on PyPI, BSD-3-Clause. If you're writing Go, use licensecheck, this exists so Python callers don't have to bind to it through cgo. If you have a license file either project gets wrong, open an issue.