00:00 Why Reporting Matters
01:26 Asset Hierarchy for Rollups
05:14 Metadata Tags and RBAC
07:27 Dashboards and Insights Tour
11:08 Community vs Pro Reporting
12:28 Pro Report Builder Basics
14:13 Themes Blocks and Templates
18:28 Frozen Reports and Scale
19:28 Live Demo Walkthrough
27:08 Scheduling Automation and MCP
29:35 API Gotchas and Filters
32:13 Report Recipes and Wrap Up

Awesome. Thank you, Chris. Yeah, things are changing rapidly, go figure, at DefectDojo. No surprise there. Um, let's see. Yeah, so we're gonna cover reporting, um, both, like, traditional reporting as well as reporting in the, the platform itself So, you know, why, why reporting?

Why is it important? Well, uh, like, you, you likely have a job, and your job likely wants you to show output, and one of the ways you can show output in a security program is to do reporting, right? Like, and the, the audiences or the stakeholders that you need to talk to are generally three people. Engineers, right?

These are the people that are gonna go fix whatever you found, like, hey, like this, you know, I don't know, there's cross-site scripting in this thing, there's SQL injection, there's whatever, an outdated library. That's gonna go to your dev teams, your engineers, whomever. Auditors are gonna come by if you're a business of any size and ask you to prove that you're doing things according to policy.

Are you matching whatever your SLAs are? Are you doing risk acceptances? You're closing findings within whatever time period. All those kind of things are kind of the auditor audience. And then leadership is really gonna be worried about trending. Like, are we heading in the right direction? Are we heading in the wrong direction?

Who is our sort of strong teams? Who are of our weaker teams? Where do I need to focus my limited time and attention on so that we can get these things hopefully driven down as close to zero as possible So what we will cover today is I'll talk a little bit about the asset hierarchy, which doesn't sound like it's about reporting, but it actually is, I promise.

Um, it actually is. It'll, it can make your life a lot easier if you do this, uh, right up front. Although Dojo's flexible, if you don't do it right up front, you can change it later. But who wants to change stuff later if you can do it right the first time? Uh, metrics and dashboards, I'll talk about what's in Dojo itself.

And then I'll get into the more, what I would call traditional reporting, where you're producing a report like a snapshot in time. Um, and talk about Pro's report, um, engine versus, uh, community one as well Um, I'll, I'll also speak a bit about what I just mentioned, that reports are frozen in time, so you, you know, you have to keep regenerating them basically.

If you need to do a quarterly report on something, you know, you obviously have to create one per quarter. I'm gonna tempt fate and do a live demo, and then I'm gonna talk a little bit about automating reporting and ways you can make that, uh, less clicky and more quick So why does the, the model hierarchy, um, have an impact on reporting?

It really, to me, when I, when I've done trainings on Dojo, I've tried to drive home the point of when you're deciding how to organize your stuff, your data in Dojo, you really wanna have reporting in the back of your head. Let's say you work at a financial institution, and you have a VP who's in charge of the insurance arm of that financial institution.

You probably wanna have an organization that represents that insurance arm, right? Because if that VP asks you, "How's my part of our business doing?" You can report at the organization level, that's a filter, and bam, you have a result. So you can do the same thing at an asset, right? Let's say that insurance, uh, VP has a, I don't know, a policy checker, right?

And that policy checker has a front end, a back end, a, um, iOS app and an Android app. Well, there's four assets really that make up that logical thing of the policy checker. Well, all of those assets, the way you set them up, you can use that to report either how is the policy checker, or are there problems with the iOS app for the policy checker, right?

That's another way that you can do that reporting. Um, engagements. I like to think of engagements as a way to just glue together or bucketize multiple tests. So if you're doing, say, a CICD on your Android app for this policy checker, you can use an engagement to glue together the results of, let's say, the SAS, the, uh, i- infrastructure as code, and the SCA tooling that you're running and maybe secret scanning.

So there's four tests that are gonna happen. But you wanna report on that CICD run, that's the engagement. Test, like I just mentioned, is that specific tool. So if I wanna know about secrets management in the Android app, I can look at just that test, and then a finding is a finding. It's an issue to go fix.

So just think about how you're gonna have to report the output of these tools and who wants to read it, uh, or look at those results. Maybe not necessarily read it like a PDF, but look at those results or is gonna ask you questions about them, and set yourself up for success

So this is actually related to my last slide. Like, you wanna organize stuff for roll-ups. I already talked about this with business units and the different, um, asset hierarchy, or the model hierarchy that you can do. The other thing to do is there's a bunch of metadata in DefectDojo, business criticality, like where it is in the life cycle, the platform, internet accessible, whole bunch of, uh, custom...

Actually even custom, uh, key value pairs you can add to assets. Those are all things you can filter on. And then tags, I kind of like to think of that as the last mile of filtering. If there isn't something in Dojo that matches how you do things, a tag can sort of catch you or get you over that last mile gap.

And so if there's a specific, a specific way, like of your assets, a chunk of them are PCI related, and you have to do PCI reporting, tag them. Even if those, say, assets span orgs, if you tag them all with PCI, then you can do a report of all the PCI things and hand it to the PCI auditor or what have you, right?

So it's just another way to do filtering. Um, and then one thing to note about reporting, this is either the in-platform sort of visual in-browser stuff or a traditional, like, printed up email it Excel spreadsheet report. Those have RBAC applied to them. So if you are a global reader or better, in other words, you can see everything in Dojo, those reports will include everything in Dojo filtered down to whatever the report wants.

However, if you are, say, someone who has RBAC and can only see one asset, uh, all those reports are only gonna have what that one asset has. So it's just something to keep in mind. The-- It's a little bit confusing to-- in some ways, that the person running the p-report determines what's in the report, but that kinda makes sense because otherwise you'd have info leakage, right?

We could have somebody who shouldn't see team two's thing seeing team two's stuff just by doing a report. So that, that's a little caveat there about RBAC. So main dashboard. So when you log into community DefectDojo, you will see this. You know, the main dashboard. There's active engagements, findings in the last seven days, accepted in the last seven days, and closed in the last seven days.

So what's happened last week, basically, or in the last week. There's a historical pie chart and reporting severity by month. And when I took this screenshot, uh, quite honestly yesterday, I realized that I need to update the demo because the dates are so old they're out of the seven-day window. Um, so the refresh script for the, the, uh, public demo needs a bit of a, bit of work, so these charts look a little bit better.

But anyway, this is what you get with the community. And then in Pro we have a command center that we quite honestly just launched Monday. And then insight dashboards, um, have been around for quite some time since the spring. I don't remember when, February, March. I can't remember. It's been a while. Um, these allow you to see your stuff live and, and I'll run through this in the, in the demo.

The command center is like an overall view of all of Dojo. It's actually really pretty cool. And then their insight dashboards with a bunch are broken into two executive priority program remediation and tool. And then you can set those by time frames, and you can do filtering on organization, on assets, on tags.

So if you wanna have for an executive what PCI looks like in my prior example, I could go to the executive dashboard, filter by the PCI tag, and bam, I have results. You wanna know how the tools are doing in finding problems in your PCI environment? Same thing. Go to the tool insights page, filter by that PCI tag.

Done. Um, a couple other things to mention about the insights. They are bookmarkable, so if you do a bunch of filtering and you wanna share that with a coworker, with a VP, with whomever, you can... Those filtered elements live in the URL, so you can copy and paste that URL and bam, right? They have a way to see the same thing you're seeing.

Uh, the charts and tables that are in those, uh, insight dashboards you can download as a SVG or, um, as a, uh, as a CSV. And then you can export the whole dashboard that you're seeing as a PDF if you wanna have a more traditional hand it to somebody, email it to somebody more likely kinda view of the world Uh, this is the command center that I was talking about that kind of shows your overall program, what's coming in, what's going out, um, and how your program is doing overall.

It's like a, like a high level view of all of Dojo, which is actually pretty sweet. I'll show you that in the demo. Um, and then for those different charts that exist both in reporting and in the insight dashboards, right? You can pick those so that they match the, the stakeholder that you're getting this data for, right?

Like an auditor is gonna care about the SLAs, which ones are activate, uh, active, mitigated, and risk acceptance. Which ones are past SLA, which means they're problematic, right? Engineers are gonna care about like what's my priority, which ones are approaching SLA, which ones are really old, the ones that are gonna potentially, you know, get me yelled at because I haven't fixed them quick enough.

And leadership is gonna wanna look at time trending data, like risks over time, open findings over time, how long does it take to fix those ti- things over time, and is that trending up or down? Those kind of things

So reporting Community versus Pro. So in Community Edition, you can grab these, these basically little widgets, drag them into a canvas, set the order, do filtering within each of those widgets. It will generate an HTML report that you can then print to PDF. Um, the layouts and output aren't saved, so you kind of have to do these one by one, uh, generate them as you go.

And then for Pro, we have reusable blocks. We basically give you a Lego kit. In essence, you can create themes, your own color scheme, header, footer, et cetera, et cetera. Blocks, which are sort of content pieces, and then templates where you stick together n of those blocks to make a report. I'll show you all this when I demo.

And then once you have that template done, you can run it and rerun it as many times as you want. Uh, we support HTML, PDF, CSV, Excel, and JSON output. Uh, there's a full REST API for this report, uh, builder as well as, uh, you can use an LLM to assist you in authoring them. Um, and then if you are currently a Community user and you're moving to Pro, we do have a way to, uh, bring in those prior created, uh, classic reports

So how do I, how do I do this reporting in Pro? So if you want to, which I would suggest you should do so it looks nice. Well, not nice. Our default template is nice, but if you want it to look like the co- the colors you're used to seeing at your work, you can create a theme, right? Once you have a theme, you can apply it to anything.

So the next kind of step to do is look at blocks. What kind of data do I need out of Dojo in these reports? These can be tables, these can be charts. These are... We have standard things like cover pages, page breaks, et cetera. Those are all blocks. The building blocks, the Lego blocks are gonna make your report.

And then the template is, okay, I've got this bag of blocks. Let me pull out the ones I need. Let me stick them in the order I want them to peer in, appear in, and then I can then create, generate that report. And once again, obviously, reports are frozen in time, um, and you have to regen them whenever you want a fresh copy.

But if you have a mandate to provide a report every quarter on something, you go through this process once and then you just pop out a new report every quarter and done And then just as a note, you filter at the block level. So if I wanna only see active findings that are critical or high for the Android app in my example, I would have to create a block that has that, and then take that block and add it to a template so I could report on that.

You don't do filtering at the templates. Really, templates are just a way to gather blocks together in a, in a report

I skipped. Oh, I did. I did two. Oh, I double-clicked. Excuse me on that one. So themes. Themes are branding, right? You pick hex colors, primary, secondary accent. You do header, footer. You can do header and footer images. You can put footer text if you wanna do, you know, internal use only or what have you, and then you can turn page numbers off or on.

Just standard theme stuff. Um, a lot of our customers like to have sort of two themes, where you have one for internal and one for external. Uh, your mileage may vary, but you can create one or more of these, however many you want, and we ship with a default one, so if you don't wanna create one, you can just use the one that ships with Dojo Uh, blocks.

So there's five types of blocks. Stock is just kind of your standard cover page, table of contents, an image, just a block of text, of arbitrary text that you put in there, whatever you want. These are just standard blocks. So if there's like a standard disclaimer you wanna put on every report, you could do a text block and then just add that in to the template.

Uh, tabular, these are table-based data. So if you're doing summaries, listings of things, you can decide what makes up the, the, basically the columns of the table, and Dojo fills in the rows for you. Detail is like your standard pen test or any kind of assessment report where you have each issue or finding has a detail of this is the description, here is the impact, here is the, uh, mitigation, here are references.

Standard kind of reporting stuff. You can do charts. And then you can do widgets, which are... I'll show you when the, when we demo, but those are y- If there's a particular dashboard thing that you like, you can also replicate that in your, uh, reporting

Uh, ways to save rework. Particularly for blocks, this can be confusing. You can think, oh, I've got a block, let me take that, but now I want it to be high or critical high, or critical... Gosh, I can't speak today. Ugh. Critical, high, and mediums. Uh, you need to duplicate the block because if you edit it, then any template that uses that block will suddenly have all three instead of two.

So this is why I keep... You'll see here, I'll probably say it a lot, a lot of times, but you filter at the block level. So you want- you'll end up creating a lot of blocks, basically, that represent what you need to report on, and then you just drop those into a template. You can preview a template. You can also preview blocks for that matter, to make sure they look like you want them to look like.

So do, uh, do that to make sure they look good, and it's what you're gonna want in the chart. And then you can share charts between the dashboard and reporting, which is really sweet. So if you do something once that's cool, you can use it in all the places

And I've said this many times, it's just the ordering, right? Ordering for the template. So if you wanna have, like in this example, a cover, cover page, an executive intro, open findings, maybe a listing of the KEV, uh, findings that have a KEV, a page break, and then an asset inventory, fine. However you wanna arrange those blocks in order.

Now, if you do, like, documents, which is HTML or PDF, it's gonna be output basically in the exact order of those blocks. So cover, executive intro, et cetera, et cetera. If you do a more data-driven output, this is CSV, Excel, JSON, obviously, like, that order becomes a little bit different, um, and not every block works for a data export.

So tabular and detailed rows work. Um, you get one sheet per block if you're doing an Excel export. And these are things you can use for if you have a data analyst who just loves Excel, if you wanna hand this off to a script, or you wanna hand it off to an LLM, right? You can do all those things with these more data-driven, 'cause LLMs won't care about the fancy header, header footer stuff, uh, but humans will

And I've said this four times already, but yes, they are frozen in time. Um, so like I said earlier, if you are going to do a report because policy says I need to report on X every so often, or there's a, I don't know, a engineering meeting, uh, every six months, and you wanna have a report of where engineering's at, right?

Create your report one time. Oh, and I should mention these are asynchronous, so if you're generating a very large report... And I think, what was the one-- We had one of the internal people do something crazy. It was like a 15,000-page report or something silly, just kind of to stress test, uh, Dojo. It did fine.

Took a while, but it's, it's-- they're asynchronous. So you may see it pending, you may see it processing, and then it'll complete. Um, and then you can just repeat this process every, every however often you need, um, to get those reports generated and handed off

All right, here's where things get fun. I'm gonna go do a demo now 'cause I like living risky. So this is the main dashboard. Um, oh, and I should show you the Here is that, um, here is that control center that kind of gives you the overall complete picture. These are my different sources. This is what the outcomes are.

I have three thousand that need attention. This is obviously a demo environment. A hundred percent of it's automated, which is pretty cool. I have forty-two of my seventy-two assets have vulnerabilities. So this is a really nice little, uh, feature that we just added. Let me go into the insights, and I was going to show you the executive insight.

I'll clear any filters I have cached. And so I was talking about the, uh, the hierarchy or setting up your asset, as- your org's assets, et cetera, in a way that helps you with reporting. So if I need to report for billing, I can literally just go here, apply filters, and these are the numbers for that particular org.

Maybe I, in fact, I need to do a report on commerce instead of billing, change those up, and bam, now I have those. Maybe your, uh, company has done a reorg, and now commerce and billing are together. I can just do that and get numbers for it. So super flexible. Um, you can do tags as well as assets. If I only want a single asset, I can pick a single asset.

If I wanna do tags, I can pick any of the tags. You get the idea. Uh, same thing for priority, right? I can do any of these things based on any of those filter criteria. This talks about the priority and risk of things inside of DefectDojo. A program talks about how effective your program is, how many assets were tested, uh, how much testing you're doing, the efficiency gains or things like, uh, actionable findings versus ones that were duplicate versus false positives versus re-import will auto-close things for you.

Uh, remediation, uh, once again, same kind of thing. Open findings, criticals and highs, open findings over time. You can tell this is a demo. Um, average time to remediation, you get the idea. And then tools, same thing. Tools are kinda fun because you can also kind of get an idea, quite honestly, of which tools are producing and not producing for you.

Like, where are all your false positives coming from? Maybe I need to do some work on this MobSF scanner to figure out why it's producing false positives. Also, if you're worried about exploitability, this is a metric of what is providing you data that either comes with EPSS or DefectDojo Pro will augment and add EPSS if it matches a CVE on any reported finding.

So which of your tools are actually giving you exploitability as well and which aren't, um, if that's important to you. That's another nice little bit of Um, those insights, particularly for tools. But let's, let's go into reporting now. So we're gonna start with themes. And so, um, this is a theme. It's nothing exciting.

I'm picking colors. I'm picking some header images and footer images and a footer text, right? Not too exciting, but this is nice so that it matches whatever your company uses. This, by the way, I'm using this Kestrel Labs as a sort of a fake company for this demo. Let's go to blocks. So, um, blocks, you can create a block, you give it a name.

It can be any of those five types we talked about. You can give it a, a header if you want or a description. This is where you can pick any of the parts of Dojo, like say if you wanna do something on risk acceptance for your auditors, right? You can create that. You can choose what fields come out of that.

You can choose the order of the fields. You can apply filters. Um, all the things that you need to do to sort of get exactly the data you want However, I've already created a bunch of these. So here's all the Kestrel ones. Um, I have a cover page, active critical, open findings over 90 days, right? All this kind of stuff I can use then to build reports.

And I can build reports with a template, and here is my, uh, template for Kestrel Labs. If I edit this template, give it a name, give it a description, right? I have-- I can apply a theme. This is our internal theme, and then you can just do whatever you need to do. I can move this... I can take this block up and move it up here and decide, actually, no, I want this in the bottom.

I can move it back. Right, I can do whatever I need to. I can add these. I can edit them. Remind- reminder, though, that editing a block here will edit it for everybody who uses the block. This is the duplicate versus edit, uh, thing. So you really wanna do your editing in blocks. But this is all I need to do to, to create a template, and then once I have that created, I can duplicate it if I wanna just tweak a couple things in it.

But I can also preview it or I can just go ahead and generate it. I'm gonna generate this in HTML It'll chew for a minute. Um, I can move past this if I want to, but, um, if you look here, I just generated this report. I can open in a browser, and here is that report with the header and the header image, the active findings, right?

It's all here. Pretty quick and simple. Um, you could also export, like I said, to CSV. This is one I did earlier. If you look at this guy, I, I don't have Excel. I've got, um, uh, word- workshop, a Google Workshop or Workspace place. Um, but here per block, I have a tab. So here's my active critical findings, here's my critical findings details, and here's my active high findings, right?

So now if I need to hand this off to somebody or an LLM or whatever I wanna do, I can do that, right? It's got very programmatically friendly, readable data

Oh, and the other thing I wanted to show you was dashboards. So if I really have spent some time making these dashboard blocks exactly what I want, and for us, let's say failing products is quite the thing, I can add this to a report and in fact even add it to my existing, uh, template that I just showed you.

And now if I go back over here to reporting and look at that template, I look at the template, you will see failing products widget is there now. And I can, I can move this up if I want it way up at the, after the cover page, I can do that, right? I can save this and now the next time I report it, it'll have that data from the, uh, from the, the dashboard widget inside of itself, which is pretty sweet.

So that's a very quick flyover of reporting. Nothing broke for my demo, which is beautiful. Um, you never know, but hey, it all worked, so yay So I've got reports, I've got blocks, I've got a theme, I've got all the things. Now I wanna schedule them, right? You can schedule them, uh, you can just run them ad hoc like I showed you.

You can also schedule them with the rest API if you wanna do automation. I should also mention, actually, now that I'm thinking about it, you can also go to the triage engine, and if you look at the bottom here, generate report. So I can use a generator report, uh, uh, out of the triage engine as well based on an event or whatever I want, as well as do some filtering of that, as well as then maybe send it over t- send it as email, right?

I can do all those things and connect them. You can, you can line this stuff up and do a triage engine. I don't wanna go into triage engine today. That's a d- that's a different office hours. But you can do this, uh, also via the triage engine, or you can do it via rest, uh, endpoint or, um, actually, you can do it with the MCP now.

We also just recently added MCP functionality, uh, that's... Well, the MCP you can turn off and on globally. If you don't like LLMs for whatever reason, that's fine. You can just leave the MCP off. If you do like LLMs for whatever reason, you can turn it on. There's a default set of, uh, tools that come with the default MCP, and then these are additional tool sets you can turn off and on.

One for the asset hierarchy. This is to understand how to lay out your hierarchy, particularly if you have a very complicated or you have, you know, a thousand repos in GitHub. You can let the LLM do that grunt work for you. Um, reporting and dashboards both now have LLM ability, so if you wanna point your favorite AI at, um, DefectDojo using the MCP, you can And by the way, those are-- You can just turn them on, and then they're on, and you can turn them off, and they're off.

So, um, and the one thing I should mention, the reason we made these not just all on is if you're never gonna do, say, asset hierarchy or dashboards with your LLM, you get a bunch of tools added to your context that you're never gonna call. This way, you can keep that context window as small as possible.

That's, that's why they're, uh, enableable individually

Um, okay. If you are doing automation and dealing with the API, here are some things you can stub your toe on, and hopefully we can keep you from stubbing your toe. If you put a filter that DefectDojo doesn't understand, it doesn't shout at you. It just silently drops filters that don't match. So, uh, if nothing else, go to the Swagger docs and check your filters, um, and make sure they work correctly before you actually automate them.

If you're worried about valid names, either for fields, uh, which are the columns basically in those tabular things, um, or filters, you can look at the block field options, and then the, uh, schema for the o- the open API schema for the REST API tells you the filter names. Um, you know, in cheat code, you can point your LLM at our schema and let it tell you what the filter names are if you don't wanna dig through that wad of JSON.

Um, multiple severities, you can use filters for this. So if you wanna do a block that has criticals and highs, it's very easy to do like I showed you. You go down to that filter section and there's a, a tabular example of the data. It's usually, I think, the first five or 10 rows. And then you can use the filters that you use everywhere else, which is by filtering with the column heads in all of Dojo to filter down to just severity critical high.

Then obviously do yourself a favor of naming that block, you know, I don't know, critical and high for insurance portfolio, whatever. Uh, patch, you have to be careful on patch and send the full filter list every time you do an update. It does not do sort of partial patching of filter entries, so you need to send the full filter list.

So if you add a single new filter, you will actually nuke all your old filters and only have that single filter. That's a great place to stub your toe. And then you write blocks and you organize templates. It's, it's really easy to think that you can edit blocks, but particularly from the API perspective, it's all about ordering.

It's not about editing. So when you're making REST calls, you make REST calls to the blocks portion of the API to edit them, and the-- really the template section is just about put these blocks in this order. And then there's a, a full example script in the docs in Python, which I have right here, right? We give you this.

You can just copy and paste this thing and run a full example of doing reporting automation via Python and DefectDojo should you get a wild hair and want to try that.

Um, and then these are just some examples, uh, mostly for posterity of things you can do or reports to think about for engineers. So for engineers, maybe you do some detail blocks filtered by the asset that they're a team member of, active and priority, right? Because you obviously probably want them working on the most highest priority thing first.

So it's almost like a burndown list. You could do that in Excel or CSV, um, if you're not already integrated with one of the many ticket systems we have in DefectDojo, or if they wanna hand that off to an LLM to ask them like, which one should I go fix? Or PDF if they wanna do like a traditional read and review thing.

Auditors, same kind of thing. You can do SLAs. Risk acceptance tables are very loved by auditors, at least according to a lot of our customers. Um, and then you can do that however your reporting period is. Like if policy says you need to do a quarterly report on whatever, you just do that. Piece of cake.

And then leadership, these are all the trending things. Average risk over time, open findings over time, average time to remediation. I would make things very short, um, throw some graphs in there. That's also usually appreciated. Um, but that's the idea. So we give you some recipes here if you need to sort of, uh, be inspired in which direction to go down Whoo.

Okay. I, that, I've made it to the end, and I am ready for questions if you have it. Um, but hopefully you have a pretty good idea of the reporting options you have inside of DefectDojo, either Community or Pro, um, as well as the sort of l- live reporting, the in, in-app dashboards and insights and what have you.

Um, other than that, you should be good to go, and I'm, I'm happy to answer questions or whatever anybody may have