httpx Integration with DefectDojo

httpx Integration with DefectDojo

httpx is an open source HTTP toolkit from ProjectDiscovery, written in Go. Given a list of hosts or URLs, it probes each one and reports what answered: the status code, page title, web server header, content type, resolved addresses, redirects, and, with technology detection turned on, the frameworks and software it can identify. It is commonly used in reconnaissance and attack surface work, and it writes JSON Lines output that DefectDojo imports with the httpx Scan type.

httpx Integration with DefectDojo

We run httpx against our domain and host lists to answer a simple question: what is actually serving HTTP right now, and what is it running? Importing that output into DefectDojo turns each probed URL into an informational Finding on the right Asset, with its endpoint and fingerprint attached. That gives us a running inventory we can compare from week to week, and when something unexpected shows up, like an admin page answering 200 or a server header disclosing an old version, we can assign it and follow it like any other finding.

Why httpx Matters

Most exposure problems start with a service nobody knew was reachable.

• It is fast enough to probe large host lists on a schedule, so the inventory stays current.
• Technology detection (-tech-detect) records software names and versions where httpx can identify them, which helps match exposed services against known vulnerabilities.
• Status codes, titles, and redirects show what a URL actually does, separating real applications from parked pages.
• It complements vulnerability scanners: httpx tells you what exists, and other tools test it.
• On its own, httpx output is a flat file per run. Nothing records which services are new since last week.

Advantages of This Integration

What we gained by importing httpx output into DefectDojo:

• A trackable inventory. Every probed URL becomes a Finding with an endpoint, so live web services are listed per Asset instead of in a pile of JSON files.
• Stable deduplication. Findings are hashed on title and endpoints only. The description, which changes with content length or detected versions, is left out on purpose, so rescanning an unchanged target does not create duplicates.
• Change detection through reimport. Reimporting each run into the same Test adds newly reachable URLs and mitigates the ones that stopped answering.
• Honest severity. Everything imports at Info, because httpx reports what is running, not that something is wrong. Analysts raise severity on the findings that warrant it.
• Workflow when it matters. An exposed admin interface or a disclosed version can be assigned, noted, and pushed to Jira like any other finding.
• Context for other findings. Because httpx findings carry endpoints, they sit on the same Asset and endpoint as DAST and vulnerability scanner results for that URL, which makes it easier to see which exposed services have actually been tested.

How This Integration Works

DefectDojo imports httpx output with the httpx Scan scan type.

1. Produce JSON Lines output. httpx writes one JSON object per line with -json. Include technology detection, titles, and server headers for useful descriptions:

httpx -l urls.txt -json -tech-detect -title -server -o httpx.json

The parser expects JSON Lines, not a JSON array. Each line is one probed URL.

2. Import it. In the UI, open the Engagement, choose Import Scan Results, select httpx Scan, and upload the file. For automation, use the API in Community Edition or DefectDojo Pro:

curl "https://YOUR_INSTANCE/api/v2/import-scan/" 
  -H "Authorization: Token $DD_API_TOKEN" 
  -F "scan_type=httpx Scan" 
  -F "file=@httpx.json" 
  -F "product_name=external-web" 
  -F "engagement_name=Weekly Recon" 
  -F "auto_create_context=true"

DefectDojo Pro users can run the same import with Universal Importer:

universal-importer import 
  --defectdojo-url "https://YOUR_INSTANCE.cloud.defectdojo.com/" 
  --scan-type "httpx Scan" 
  --report-path "./httpx.json" 
  --product-name "external-web" 
  --engagement-name "Weekly Recon" 
  --auto-create-context

3. Reimport on a schedule. Send each run to /api/v2/reimport-scan/ against the same Test so the Test reflects what is reachable today and keeps the history of what came and went.

Two behaviors to know: a target that does not answer produces no output, so an unreachable host simply contributes no findings. With -probe, httpx also writes records with "failed": true for targets it could not reach, and the parser skips those.

Data Granularity: What Gets Imported

Timestamps and response times are deliberately left out of the finding, since they change on every run and say nothing about what was found. If the same URL appears twice in one file, only the first record is imported.

Use Cases

Continuous external inventory: A security team runs httpx weekly against every domain the company owns and reimports into one Test per perimeter. New URLs show up as new findings, so a newly exposed service is visible the week it appears.

Version disclosure review: With -tech-detect and -server, descriptions list identified software and versions. Analysts search those descriptions for components with known issues and raise the severity on the findings that need attention.

Before and after migrations: During a move to a new hosting provider, the team compares httpx imports before and after cutover. URLs that stopped answering are mitigated on reimport, and any that unexpectedly remain are easy to spot.

Feeding deeper testing: The list of live endpoints in DefectDojo becomes the target list for DAST tools, whose findings then land on the same Asset.

Operational Tips

• The status code is part of the title, and the title is part of the dedupe hash. A URL that changes from 200 to 403 imports as a new finding, which is usually useful, but expect it.
• Always pass -json. Plain text output will not parse, and a JSON array instead of JSON Lines is rejected with a clear error.
• Don't set minimum_severity above Info for httpx imports, or every finding will be filtered out.
• Use separate Tests for separate scopes (for example production and staging host lists) so reimports of one don't mitigate the other.
• Tag findings you have reviewed and expect, or close them as accepted, so new arrivals stand out on the next import.
• Keep -tech-detect on. Without it, descriptions lose the technology list, which is the most useful part for triage.