Getting Data Into DefectDojo: A Practical Introduction to Parsers and Connectors

Every security team inevitably will run into the same problem when utilizing multiple different types of scanners. One tool exports JSON, another CSV, and another XML. One vendor's "critical" is another vendor's "medium” and you have the unfortunate task of making sense of your scan data.

In last month's Intro to Parsers, our Co-Founder Matt covered the basics on what parsers are, the five ways to get data into Dojo, what to do when a tool isn't supported, and how deduplication tuning works. Here are the essentials.

What a parser actually does

In DefectDojo, a parser turns a scan report into findings. It takes the file your scanner produces, breaks it apart, maps it into DefectDojo's finding data model, normalizes vulnerability identifiers like CVEs, and creates findings. Once findings exist, DefectDojo can deduplicate them and give you one consistent way to work with output from any tool.

That normalization is the fundamental problem DefectDojo was built to solve. Every scanner identifies security issues its own way. Parsers turn all of those formats into a single, consistent finding, so you deal with one data model instead of dozens of report formats.

Two vocabulary notes worth knowing:

Scan type is what DefectDojo calls a parser in practice. If you're importing Semgrep results, the scan type is Semgrep, and under the covers Dojo uses the Semgrep parser to read your JSON and create findings.

Re-import is a special way to bring data in when you're scanning the same scope repeatedly. Dojo compares what it found last time with what it found this time, then opens and closes findings as appropriate, leaving you with one actionable list. The key caveat: re-import only works when the scope is the same. Re-importing a scan of a different repo into the same test will not give you the results you want.

It's also worth noting that DefectDojo keeps its definition of a finding deliberately loose. If it's something you want tracked and addressed, it can live in Dojo. Matt described a financial services customer that stored the results of manual security control reviews, a desk exercise with no scanner involved at all, as findings.

Where findings live: the hierarchy

Findings don't float in space. DefectDojo organizes data in a hierarchy:

• Organization (formerly called Product Type): a grouping of any number of assets
• Asset (formerly called Product): a system, a repo, a scope of something you test
• Engagement: a testing activity, which can combine multiple scans or mix scanner output with a manual pen test
• Test: an individual import, typically one scanner's output
• Finding: the individual issues themselves

One feature that makes this hierarchy painless in automation: auto-create context. Tell DefectDojo which organization and asset a scan belongs to, and if they don't exist yet, Dojo creates the whole hierarchy down to the engagement and test automatically. Your CI/CD jobs don't need lookup-then-create logic. As Matt put it, it's a simple feature that turns out to be a godsend for productivity.

Five ways to get data into DefectDojo

1. UI import. Run a scan, log into the web UI, upload the file. Not glamorous, but it works, and it has been in DefectDojo forever. Available in both Community Edition and DefectDojo Pro.
2. REST API. Push results in with a REST call instead of logging in. The import and re-import endpoints both support auto-create context. Also available in both editions.
3. Universal Importer CLI (DefectDojo Pro). A single binary, configurable through config files, environment variables, or command line options. Instead of maintaining a GitHub Action, a GitLab runner, and a Jenkins plugin separately, you drop one binary into whatever CI/CD you run, and it works.
4. Connectors (DefectDojo Pro). For vendors that expose findings through their own APIs, connectors reach out on a schedule and pull results in automatically. Set it up once and DefectDojo keeps syncing. There are more than 80 connectors today. One recent change for Pro users: the old "connectors and integrators" split has been streamlined into upstream connectors (pulling data in) and downstream connectors (pushing data out to Jira, ServiceNow, GitHub issues, GitLab issues, and similar destinations), consolidated under one menu.
5. Smart Upload (DefectDojo Pro). Built for broad infrastructure scans. Run a Nessus scan of everything, and DefectDojo can split the results up and match them to the right endpoints or locations automatically.
   
   

When your tool isn't supported

DefectDojo ships parsers for more than 330 security tools, but every team eventually hits an edge case: an internal tool, a homegrown process, or that one scanner without a parser yet. You have options.

Generic importers (available in both editions) accept CSV or JSON. If your data is simple, CSV is fine. If it's more complex, JSON is the better choice: multi-value fields are much easier to represent, and you get access to the full finding model. Convert your data, upload it, and you can even rename the test type afterward so it reflects your actual source instead of saying "generic importer."

Universal Parser (DefectDojo Pro) lets you build a parser from a sample file with no code. Upload a JSON, CSV, or XML sample, give it a name, and DefectDojo lists every finding field alongside every data field it found in your file. You map them, preview what the findings will look like, and submit. The result behaves like any other parser.

Like we talked about in the training, Universal Parser is a parachute. It’s a fast way to get data in when no parser exists, and often a temporary bridge until a full parser covers your tool.

A few Universal Parser caveats worth knowing up front:

• Scan types can't be deleted, and this applies to all scan types, not just Universal Parsers. Deleting one would orphan any findings attached to it and break data integrity. If you misconfigure a Universal Parser, create a corrected version and deactivate the old one, which hides it everywhere while preserving the data underneath.
• Renaming is supported. Customers with internal tools that change over time often version their parsers (Internal Tool V1, V2, V3) and deactivate old versions as they retire them.

Contributing a parser is always an option too. DefectDojo is an open-source project, and parser contributions via pull request are one of the most welcome ways to give back to the community.

Tuning deduplication

Deduplication is where parsers pay off. DefectDojo compares a selection of fields from each scan type to decide whether two findings are the same issue.

Sometimes it's easy. Some scanners, like Qualys, assign a unique vulnerability ID in their output, which makes matching straightforward. Most tools don't, so DefectDojo ships sensible default field selections for each scan type. But defaults can't anticipate everything. A framework that trips up scanners or a monorepo that confuses SCA tooling can break the general case, so DefectDojo lets you tune which fields determine a duplicate.

In Community Edition, you edit the dedupe field lists in your settings.py file (your local copy, so updates don't overwrite it), then restart Dojo. You can also run a manual job to recalculate deduplication after changing fields, since older findings were hashed under the previous formula. Community Edition supports dedupe tuning within a single tool per asset.

In DefectDojo Pro, there's a UI for the same tuning, no restart required, and you can also configure deduplication across tools. If you run both Veracode and Checkmarx and want duplicates matched between them, Pro can do that. Re-import matching uses the same hash code mechanism, so it can be tuned the same way.

Community Edition vs. DefectDojo Pro at a glance

Final thoughts

DefectDojo gives you many ways to get data in because no two AppSec programs work the same way. As Matt put it, the founding principle was never to make you adopt how Dojo works, but to have Dojo adopt how you work.

If your tool is supported, pick the import method that fits your workflow. If it isn't, the generic importers and Universal Parser have you covered. And once your data is flowing, think carefully about deduplication scope, and tune it when your environment doesn't match the general case.

Ready to try it? Explore the supported tools list or dive into the import method comparison in the DefectDojo docs.