00:07
Everyone should be seeing my screen now. So yes, we're talking about the universal parser today. It's a new feature we just added to DefectDojo Pro. And like Dawn said, I'm Matt Tesauro, I'm the CTO and co-founder of DefectDojo Inc. And I've been around the project for ever in a day, quite honestly. So just a quick overview, I'm gonna sort of define the problem we were solving or trying to address with the universal parser.

00:37
I'll move on to what we actually created. I'll do a demo. I'm invoking the demo gods. Hopefully they will be kind to me today. And then I'm gonna give you a couple of brief updates on the metrics and reporting aspects of DefectDojo Pro that were mentioned in your invite.

00:58
So let's get to the intro and I'll define the problem. So there are many times in life where you're just trying to make some progress, right? You're moving along, you're hopefully going downhill and things are going your way, but suddenly you hit a brick wall and you're forced to stop. And so we noticed there's a use case in DefectDojo where this can happen and we wanted to address that use case with DefectDojo Pro. So how do you break through that wall that blocks you?

01:27
I will get into that in half a second. So the big picture, this is where we'd like to see everybody using DefectDojo to get to, to where you have this nice flow from left to right. Of your tools producing results, those flowing into DefectDojo, those then going down to downstream systems. We're up to 194 plus supported tools. I counted last night, that number grows every time I counted. It's kind of crazy. But for this specific talk, we're really focusing on

01:57
data ingestion, so getting data into DefectDojo, right? And some of the potential pain points that can exist around that. So when we talk about ingesting data into DefectDojo, it generally works fine. I mean, we have 194 tools we support, so we pretty much cover a broad swatch of the market of security tooling, but there's a couple of cases where it doesn't work. This is when vendors just randomly decide to change their tool output.

02:27
They add a whole new stanza to a XML, or they add a chunk of JSON to their JSON and change the format enough that it impacts our ability to parse that file correctly. There's also several cases where vendors will drop these places to pull output out of their tooling, but in a bunch of different places, and each of one of those is slightly different.

02:54
And so if you don't have just the right sometimes configuration of output settings, you'll get a file that looks a little bit different enough that it can sometimes trip up the parsing or you have an unsupported tool, which is getting to be less and less the case anymore with almost 200 tools. Now you have an internally developed tool. This has definitely happened to us for some of our customers. Or you just don't happen to like how the current parser parses things. These are all sort of places where this could address.

03:23
um, address your need and not keep you from having to slow down. So in that regards, what have we created? Well, we created this thing called the universal importer. And the idea here is that you upload an example of the file you want to parse. You give it a name, you map the fields to where you want them to go into DefectDojo, you preview those mapped results and you save it and now you have a new parser.

03:53
It's like an easy three-step thing. So.

04:00
The first step in the universal parser is, like I said, you're going to select a file, like I have shown here. You select a file, you give it a name.

04:13
The next place I'm going to map like title to title severity, severity, CW to CWV. You map fields from that parsed file into where they're going to end up in DefectDojo. You map whatever you want. You click next.

04:32
And then I have a preview of the parser results. So I can get an idea of what this is going to look like. If for some reason, this doesn't look like I want it to, I can back up. I can change the mappings. I can make adjustments. I can go back here and see a preview. But once you're happy with it, you simply submit it. It's you're asked you, Hey, do you want to create this universal parser? Presumably you will say yes. And then bam, you've got a universal parser. I can just go to add findings like.

05:01
anything else in DefectDojo that new I update my file and then I third step I select the new parser type I just created in this case the fictitious my favorite scanner right so this is the process by which we're allowing our customers to have anything put into DefectDojo just by simply mapping those fields so if we happen to not have a parser for it you have an internally developed tool

05:27
Whatever the case is, you can now not be blocked from getting that data into DefectDojo, which is pretty sweet. And then by the way, once this universal parser is there, it works like any other parser. So in this case, if I'm gonna do an API import or an API reimport, that's a valid parser to select when doing those API calls. So I'm not limited to just the UI, I can do this with the API.

05:54
Pro also has the universal importer, which helps you with CI-CD and automation. This also works perfectly fine with this new universal importer that you just created. So the goal here was to make sure that you weren't blocked and getting data into DefectDojo because of either a vendor tool choice change or what have you.

06:16
Oops, there we go. Oh, and yes, just in case anybody was afraid, we do have dark mode. Please don't be scared. We do support the dark side. For those who want to do dark or light, we're open to either.

06:31
Okay, demo time. Here's where I get to invoke the demo gods and hopefully you'll have a face like that while I'm doing this demo.

06:41
Okay, so I'm logged into DefectDojo with this particular instance. This is a demo instance that's running locally for me. I've got some data in here. I've got a number of products. I've got some grades for those products. In this particular instance, I'm going to look at, you might notice a gaming theme here. Super Mario Brothers 2, which is our product that I'm using for this particular demo.

07:10
I've already got a couple engagements and tests and about 500 some odd findings. If I scroll down here, you can see I've done a SemGrep scan, a sneak scan and a Trivy scan. I've got these finding results. So, so far so good. However, I have another scanner I wanna add to this particular product, or I wanna have results I wanna add to this product. So I'm gonna go to add findings. I'm gonna choose my file.

07:42
I'm going to search for that scan type.

07:46
when there isn't a Mario scanner. So I might be afraid that I'm kind of stuck here, but I'm not, because that's okay. I can go to the universal parser, which is temporarily linked up here. We'll move it to a better location when it's not in my local demo. But let me grab that file again.

08:08
Let me call this the Mario scanner.

08:12
And now I need to select the root node. So I wanna pull the results out of this file. And in this case, I have a title field, so those match up nicely. And I also have, I believe, a severity field, which I do, which is great. And I need a description field, because those are the three required ones. Well, I have two, so I'm just gonna include both of them, so I get all of the description pulled in here. Let's see, I've got a CWE, I can map that.

08:42
I had a unique ID from tool. Let me map that. I have some tags. Grab those and I have some vulnerability IDs. There you go. So now I've mapped everything and I know where it's going and all this looks good to me. So I'm gonna click next.

09:04
Here is that preview that I told you about. I can look and see how these things are gonna come out. I can read the descriptions. Here's my description in detail description. So it looks like everything is coming in like I want it to, which is cool. So I'm gonna go down here and submit this guy. And yes, I do wanna create a universal parser. So bam, I just created a universal parser. I can go back to add findings now, go grab that file.

09:33
that I wanted to add to my product. This is a Mario scanner. This was in the Wii product type.

09:47
and Super Mario Brothers 2. I call this Mario Scan.

09:56
and I'm gonna submit.

10:00
And so now I have the import starting. I can go back to that product and we can check things out. And so I've got my three prior ones, as well as now this new Mario scan that we didn't have support for a minute ago, but now we do. If I drill into here, I can see that my findings, if I go into the test, my findings are all here.

10:22
Everything came in, I got the CWEs, I got the tags, everything worked, so this is awesome. So a couple of minutes, bam, I've got data going into DefectDojo. And now from here on out, if I wanna use an API call or however I'm doing CI, CD, whatever, this is now a new target for ingestion for DefectDojo, which is pretty sweet. And so this is now looking good. I've got four.

10:46
engagements in here with four different scanner data coming in about this product, but there's an internal tool that we created that I also want to add. So let me go ahead and do that and I can't imagine it's supported right because it's an internal tool. So if I look for the spooky scanner, not surprisingly it's not found. So let me go back here and I'll repeat that process of grabbing an output from the spooky scanner.

11:16
And once again, I need to select where I want to pull those results out of that scan file. And I've got a title. And for severity, we don't have a severity, but I do have spooky factor, which was our silly way of using severity in this internally developed tool for description. I once again have a two-part description and I have, I know some tags and I have some vulnerabilities.

11:46
So I want to map those.

11:50
So here's that preview again. I have the Phantom Mask Psychological Exploit. These are pretty awesome, right? Salient AI, enemy AI manipulation. Awesome. What a great finding. So let me submit this one. And do I do wanna create the parser. And now once again, I can go to add findings and it's just another day at the office. I grabbed the spooky scan results. I select spooky here.

12:19
And this is going into Wii.

12:24
and Super Mario Brothers 2. Let's call this the spooky results.

12:35
And let me submit it. And bam, I've got those results coming into DefectDojo. So now in a couple of minutes, I've done two different imports into DefectDojo, which is pretty sweet. I can go back to the product.

12:50
I have five engagements now, which you can see down here, including my spooky results, which include the output from that spooky scanner. And here's all my findings. So it all just works. I mean, it's a super cool way to get around that roadblock of I have an internal tool that doesn't happen to have a parser and, or I have a vendor who's changed their thing, whatever the case may be.

13:16
I'm now not blocked artificially from getting data into DefectDojo. And like I said, this could be API, this could be the universal importer. So by the way, these external tools are just here in DefectDojo now for all of our pro customers. You can go grab that universal importer if you are doing CI, CD work. So that's my demo. I now have more juicy data.

13:45
in my metrics, which is pretty sweet. You'll see December was a rather busy month for some odd reason since this is my local install. But that's the demo and this stuff just works, which is pretty, pretty awesome.

14:00
So going back here, the other update I had was on metrics and reporting. And this is just a quick kind of overview.

14:12
We added program insights, remediation insights, and tool insights prior. We have recently added an additional executive insights, which gives you this dashboard that you're seeing here. The nice thing with this dashboard is that it allows me to filter the results of this dashboard, either by time, by product type, by product. You can even have tags on products and filter by those. You can obviously clear and then export.

14:41
the results as a PDF as well, which is pretty sweet. So here's me doing some filtering. So in this case, I'm doing the last 30 days of the billing product type, and I'm selecting two specific products I wanna get a report on.

14:58
If I click export, it'll give me this pop-up window and say, hey, I'm exporting your report. It'll show up as a download and then bam, I've got a PDF that mirrors the results that were just shown to you in DefectDojo, which is pretty awesome. And then after we did the executive insights, we also added

15:19
the same features to the existing three insight dashboards. So I can do the same thing for the program insights, remediation insights and tool insights as well. So all of those have the ability to do that filtering. And I actually have, if you give me half a second, a different instance with way more data. Let me stop sharing my screen for a second.

15:45
Let me share a different screen that has more data.

15:50
And we do have several questions on the universal car. So when you get done with this. OK, cool. Yeah, let me run through this quick demo, and then I'll answer everybody's questions if that works. Thanks.

16:07
So just to show you, this is a different instance that has far more data than my local instance. Here are those metrics pages, the different insights, the executive insight I mentioned, right, has these filters, I can do a date range, all the product types. I can select products once I've selected a product type like that billing and, you know, bodget and apple, like I showed in that example.

16:34
I can apply the filters, I can clear the filters.

16:39
And then I have these, this is just summary information, kind of at the executive level to get an idea of a sort of a broad overview of how your app sec or dev sec ops or whatever you want a product security program is doing. We have active mitigated and accepted risk findings, active findings by severity, fixed findings by quarter, products tested yearly and test performed by quarter, average time to remediation, this average active finding by severity.

17:09
EPSS, just as an aside, DefectDojo will enhance any finding, DefectDojo Pro will enhance any finding that has a CVE that has a matching EPSS by adding that EPSS automatically for you if you're a pro customer. And so these are those EPS, average EPS ratings by tool, get an idea of which tools are giving you a good window into exploitability, and then mitigating findings by SLA to get an idea of how well your teams are producing.

17:37
or fixing things within SLA. And then like I said, those same insights and filterability exists in all the other insight graphs. We had a different office hours on this, but this is the same kind of thing where you have a whole bunch of different ways to present data to you across products and then across tools as well, just to get a better feel for what's going on with your product security program.

18:07
All right, that's what I got for that demo. I'm gonna stop sharing this and go back to the other one because I'm assuming. The questions are all about the parser. Yeah, so I'm gonna go back and share the screen that has that. Let me go back here. Okay. All right. Okay, so yeah, question time. Okay, first question, which formats are supported? Only JSON, what about XML or CSV?

18:36
Ah, okay, perfect. In its current iteration that you saw today, it's JSON. We are currently adding CSV and XML. So all three of those will be supported. Do we have a timeframe for that? JSON's done and they're currently working on CSV. CSV is silly easy because it's just basically an X by Y matrix. It's not hard. I expect that one to be done relatively quickly, I would say. And then XML support is more interesting.

19:05
in that we're trying to decide how we want to handle parsing that in a generic sense. And we're looking at a couple methodologies that we haven't honestly filtered down to a final one, but I would say very early Q1 of next year, probably Jan, I would guess, but I don't really know. It depends on how well the libraries we're trying work, to be totally honest with you, but they'll be fast follows because we've sorted out the bulk of the problems using JSON.

19:35
And the real problem is taking the input, coercing it into our internal format that makes that mapping page that I showed you this. Let me grab one really quick.

19:49
Do I have another one? Yeah, I'll just reuse the Mario one.

19:57
We take your file and basically turn it into an internal representation that drives this page and how these fields are selectable. Oh, I need to select a root element. How these fields are selectable, because if you notice, certain fields have the ability to select more or less things based on the data type. So I have fewer things in tags than I have in others. And so we're just working on coercing XML into this internal format.

20:27
And that's the one thing we don't have currently solved. Although I haven't talked to the engineer working on this in a couple of days, he might be there. Okay, thank you. Okay, next, can you configure tune deduplication for universal parsers like for the built-in parsers? You can, yes. The same ability to do the tuning that we have in DefectDojo works, these are just other parsers. They just get added as if they were source code.

20:56
So yes, all of the, for pro, you can do the same tool, cross tool, dedupeing, adjust them with the UI, just like you do any other parser in DefectDojo. Yep, there's no difference. Well, there's a difference in that you created it, but there's no difference in how they're treated by DefectDojo. Okay, great, thank you. Next, is there only one-to-one mapping or can you combine or split fields?

21:23
Ah, so I showed you how to combine fields in this particular one because in this one there was two descriptions. So you can combine fields. I guess if I wanted to put CWE in the description, I could, right? If I wanted to do that. And then this parser would have a description that included those three fields. The real restriction is on field pipe. If there is a, well, that's not it. That's not 100% necessarily true to because CWE in this case is an input, we can coerce it to a string.

21:51
But there are certain field types that don't map nicely to other fields. This is why we did this distinction for tags and unique ID for, for tool only lets a single element. So there's some constraints, but generally speaking, if I wanted to map a whole bunch of stuff, I don't know if I want to put title in the description too, I could do this right. And make a parser that did that. That'd be fine. So there's not, there's

22:16
Only like structural constraints of how the data is stored in DefectDojo would block you from combining fields. Okay. Next, what about more deep, not only flat data structure of finding? But we support that as well. The examples I have are fairly contrived. But yes, we have support the ability to walk down. That's why we have this root node selection with more complicated ones you can select different

22:44
Data depths, I guess you could say, I don't know how to describe that well, but yes, you could do the different data depths. Next, I noticed the message import started. Does it mean that Pro is not blocking as an open source? Do you implement async import? Yes, for Pro, we do have an async import that you can choose to optionally turn off or on. By default, we have it on, so it's non-blocking.

23:13
Mostly because this well, there's several reasons for really large files. Obviously that helps. But also for this UI, we don't have to block the UI waiting for that file to be parsed. Okay, thank you.

23:30
Next question is, I see API connector. How is API connector different to API pull available in open source? OK, so the API connectors, I'm going to abandon this parser, by the way. It was a bit silly. So the way API connectors work is you configure, I don't have one configured in this instance. You give us a secret or a token or whatever it requires in this case. I think it's a bear token for sneak.

23:58
the URL and then you give it a label. And then we automatically pull these into DefectDojo on a daily basis. The primary difference is that once, I don't have it configured dang it in this instance, but once you have it configured, you can map the equivalent to DefectDojo products in that vendor's tool into the products in DefectDojo. And then it just automatically puts them in the right place going forward. We do diffing to say,

24:28
You added a new thing in sneak that isn't mapped to a product. Where would you like us to put it? We can also auto map them. There's a whole bunch of features around connectors that make them much more robust.

24:41
The idea with connectors is basically you have a vendor, you're paying for a vendor's tool that has an API and you just want to get the results into dojo after doing a one time setup. That's what connectors gives you.

24:56
Okay, let's see, next one. Do you also support the processing of endpoints?

25:04
With the universal parser. Yes. The endpoints. Yes. Those are, those are something we handle as well. If you happen to have a tool that has endpoints. Okay. And then next.

25:20
How can I edit the existing Universal Parser? So the way that the editing of Universal Parsers works is you can in essence deactivate the current one and make a new one. The reason we did that was specifically because if you have done any dedupeing with the old one, we didn't want to negate that dedupeing that exists. That way we can leave you with the opportunity to create a new one and then decide to reconfigure

25:50
those dedupashes for the new one or stay with the old one or do whatever you wanna do. So it's mostly about flexibility. And there's some interesting edge cases you can get if you edit an existing parser. Does that mean you wanna change the prior ones or not? And so making it maximum flexible was better to have a user choose to create a new version of that parser, universal parser versus going back and

26:20
modifying the other one and then causing some ripple effects that you may not realize.

26:26
Thank you. And then the last question we have. Not related to the cursor. How does the new UI for open source. Compared to what we see in. The new UI for open source is still in progress. So there's I can't show you anything. We're still trying to find a decent toolkit. This is the New UI for pro if I

26:54
Open this in a new tab. You can see the, this is the pro version of the existing UI. If you're familiar with open source, this should look more or less familiar. There's a few additional things that are in pro with the current UI versus the new UI. But we still haven't found a good toolkit. We're doing that research. There's also a community member that had asked if they could do that work, and I'm kind of waiting on them to come back and give me their results from doing some looking at, or looking at

27:23
different libraries to use to replace our existing libraries. I think if I remember correctly, this is a bootstrap for the CSS and sort of look and feel. And then we're using the Django, Jinja templates in the current UI. And also for open source, same one.

27:44
Okay, thank you. Are there any other questions?

27:54
Well, we got a lot. So I think that probably answered a lot of people's questions. Yeah. Well, and unfortunately like the universal parsers just easy. That's a good thing. Yeah. I know it's like, I know I will. I was practicing for this demo. It's like, wow, I'm doing this in like five minutes. So like, uh, I'm almost glad I had the extra stuff about these metrics because. Uh, ah, all right. Well.

28:24
Everybody gets 30 minutes back in their day. And always, if you have further questions, you can always email us and keep subscribing to our newsletter, which you can find on our community page on our website, just to get all the information about any upcoming sessions. And this is our last session of the year. So we will wish everyone a happy.

28:53
December break and we will see you in the new year.