Transcript

00:07
I'm Matt Tesauro, I'm the CTO and co-founder of Defectojo Inc. This is the June office hours. We're gonna be covering metrics 2.0. So let's get.

00:22
So what is metrics 2.0? Fundamental question. And I think the reason I put this graphic up on the screen is I think most people, when they think about DefectDojo, they think about the bigger picture, right? DefectDojo is your single source of truth. It's where all of your vulnerability management, all your tool output goes, gets normalized, gets deduped, gets made sane, right? And then from there, you can push it onto downstream stakeholders, right? But, and in...

00:50
This is what I think most people think of with DefectDojo, but the idea when we started talking about Metrics 2.0 was to make one change in sort of this diagram and the idea being let's reverse that arrow, right? What if we could reverse that arrow and let's push more reporting and metrics into DefectDojo itself, right? We have a decent amount of charts and graphs kind of sprinkled through the existing UI, but can't we do more? And that's kind of the driver behind what Metrics 2.0 is.

01:20
Our desire to take sort of our fundamental work, I think, was solid for the existing graphs and charts that are in DefectDojo and sort of move it up a notch and really sort of push that needle a bit. So we want to take that arrow and bring that into DefectDojo was the idea.

01:40
So what's the virtual visualization vision, right? What were we thinking when we were thinking about metrics 2.0 and that's really data visualization is really what was driving that. Couple of things. So one is this idea of having interactive, responsive charts and graphs and numbers calculated about the most important things in your security program. Like, what do you care the most about? Let's provide those to you.

02:09
And even better, multiple views of that data, right? Helping you understand different aspects or slices or views or whatever you want to say about your security program. So you can understand, you know, where you're doing well and where you're not doing so well, you have the, I think that the, the cute term is you have opportunities, right? So where you're, you're doing great and where those inevitable opportunities. And then ideally to be able to dynamically view that, right? So I want to be able to remove things.

02:38
that aren't maybe important to me and keep the things that are, because I've worked at a bunch of different security programs. I've run a bunch of different apps, like programs from big to small companies. And everyone's different because quite honestly, if you are creating software, you are doing that because you can't go buy it generally speaking. And because you're doing it, it's a one-off for your business, which means generally app sec programs are one-off for that business.

03:05
Now there's several things that are shared amongst all the AppSec programs, but generally speaking, security programs are very much focused on things that are specific to that business rather than, well, there's some generalities, but the, nuts and bolts go down to the context of that business.

03:24
So the first thing we thought about doing was doing some dashboards. So take the data that already exists inside of DefectDojo and create those interactive charts and graphs and do that data visualization for you. And we kind of broke these into three main areas, at least for this initial sort of MVP launch, Metrics 2.0. Remediation insights. So this is understanding how effective and well-run or where maybe there are gaps.

03:52
and there are laggards and there are people that are strong producers in your remediation efforts, right? Because you're collecting findings, vulnerabilities, and DefectDojo normalizing them, deduping them. How are we actually doing at making those things go away? What do the trends look like? What is my mean time to repair? Those kind of things are what we're looking at for the remediation insights. And then tool insights, right? DefectDojo supports, shoot, I can't remember the number anymore, over 177-ish.

04:20
Uh, cause there's probably more now since the last time I count tools that we can take the output from. So what are those tools doing for your program? Are they being effective? Can I, am I getting more out of one tool than the other? Is it time to maybe think about switching my SCA tool because it's producing a lot of false positives. And I'd much rather start maybe comparing to the competition or maybe I just inherited a business unit and they're using a different tool that I can compare.

04:47
Like how their tooling and say DAST is doing against our existing tooling and make some, you know, enlightened and informed decisions about what to do with tools. And then program insights, right? Is your program healthy? Right? How, how is my security program doing? Right? This could be a traditional AppSecs program. It could be a DevSecOps shop. It could be a security program that covers all the things. When I worked at Rackspace and the product security team, I owned everything that made the cloud run from the iron up.

05:16
So we kind of had to keep track of all the things. So how is that program going? And then where are you spending and or saving time? Which is a really important criteria because inevitably security programs are bandwidth limited on team. Like I have never found a security team that said we have too many people, we don't know what to do. You're always buried. So for those precious and limited resources of the security team,

05:46
Am I making them able to get the most, do the most with their time, their limited time and effort? Or do I need to make some changes to help us be more effective? Those are the ideas of those three major areas that will be in our MVP for this Metrix 2.0.

06:06
So before I kind of teased you a bit with the vision, before I get deeper into demonstrating the vision, I wanna tell you what metrics 2.0 isn't. Because in calling, talking with some customers and people just in industry, the metrics idea or the metrics term can mean several things to different people. And so I just wanna be clear about what metrics 2.0 isn't. It's not generating reports based on some kind of filtering of the data and DefectDojo. We already have that with the report generator.

06:36
We just pushed a pretty good update to the report generator for our pro customers, but that is not part of metrics 2.0. That's a separate effort that we are trying to make that better as we go, but it's not part of this particular effort.

06:51
And it isn't those particular reports that you can click on kind of sprinkled all over the place in DefectDojo. So this is not those reports. That is a separate effort and not to be confused with sort of Metrix 2.0. And it isn't exporting filtered data into CSV or JSON, which is another thing you can do to get data out of Dojo. You can also call the API for that matter. But Metrix 2.0 is not covering that in terms of scope. Just to...

07:21
Just to be clear, so that's what metrics 2.0 isn't. Why don't we have a sneak preview of what these dashboards will look like in the new UI?

07:34
So remediation insights. So I've opened findings of severity high or critical, as well as EPSS score, average time to remediation, right? You can track that over time to see how you're doing. Findings with exceptions are findings that have a risk acceptance in them. And then findings past SLA, grouped by both products and the manager if you have assigned a product manager to that particular product.

08:02
And then tool insights, how many tools are you using? Just a raw count. So in this case, there's 12 tools being used. Severity by tool, what am I getting out of those tools? What are they finding? And then tool EPSS average. So what is the average? EPSS is a measure of exploitability created by FIRST, which is an organization that looks at incident response and has a...

08:30
EPSS is a score that is attached to a CVE that talks about how quickly or how exploitable that is, particularly in terms of the wild or being seen in the wild, people actively exploiting it. And so we'll give you an average of what those tools are finding to give you an idea of how much of a bead your tools are giving you on actually actionable, exploitable vulnerabilities. And obviously with limited resource, you want to focus those on the things that are exploitable.

09:02
And then program insights, things like noise reduction. How much is DefectDojo saving your limited time of your staff from having to deal with deduplication, diffing scan to scans, dealing with false positives? Right? Here's, this will give you a report of what you're, what you're seeing in terms of false positive rates, as well as deduping and that automatic diffing, the defect, defect, yeah, the DefectDojo does for re imports.

09:31
as well as the cost savings. So since your employees are not having to deal with these issues and burn cycles, to be quite honest with you, doing what DefectDojo will do for them, getting rid of that drudgery, what are you savings? And this is sort of a cost estimate of what you're saving in terms of people's time. We looked at the sort of average salary of a middling security analyst and did some math. So that's a sneak preview, but...

09:59
That's not as good as seeing it for Realzy. I don't actually have a running DefectDojo instance with it, but I do have the mockups that I'm going to show you. So let me go to my actual mockups. So here is that remediation insights dashboard. And this is the mockup we're using just to kind of the POC these before they get embedded into DefectDojo's new UI. So here's that average open findings higher or greater severity, BPSS score.

10:26
Open findings over time, you can tell when I started to insert things into this demo. So, uh, so I had good charts, your average time to remediation, also findings past SLA by product types and products, or by, um, like I mentioned earlier, products and product managers, and then findings with exceptions. Now, besides being able to sort of hover over here and get sort of point in time data out of these charts, you can also interlink them. So for this.

10:55
Sam's awesome product. If I click on this, we'll actually update these dashboard values for that specific product. I can click on it again and it'll refresh back up to all of the products. So this is what I mean by interactive dashboard elements. And then let me pop over here to tool insights. So total number of tools in this case is 13, false positive by tool. You can tell I sort of programmatically added a bunch of false positives to sneak. So I had something to show.

11:25
Severity by tool. This is where I can look at what type of findings I'm getting from which tool in terms of severity. And also these are, by the way, interactive. I can get rid of sneak from this graph if I want to see what sneak and sem grip are producing. I can hide those other ones. These are interactive. This is security findings by tool monthly to get an idea of what month over month your tools are showing you. And then this is that chart I mentioned earlier.

11:55
about average EPSS and you can tell these are obviously programmatically created data elements inside of this testing instance. We don't have a product called My Test Type or a test called My Test Type. And then program insights. So findings fixed per quarter, number of products tested by year, number of tests performed, the efficiency increase, which is how many...

12:23
What percentage of noise was reduced by DefectDojo automatically handling? Like I said earlier, those dedupe the false positive or the false positives. And then the reimport diffing it does. Here's a more graphical representation of that, where you can take these off and on to get an idea of what the percentages are. This is that noise reduction. Just another view similar to this of how the noise has been reduced for your team and the hours saved.

12:49
And then cost savings, right? This is your cost savings based on also your team not having to work those issues that DefectDojo automatically handles for you. So this is a rough idea of what these metrics will look like. They may not look exactly like this when they get embedded into the thing. This is just sort of a template we're using to prove out each of these graphs before we embed them into DefectDojo itself, but at least you get to see what they look like and how they interact with real data.

13:19
Technically, I guess, QE data, but you get the idea.

13:25
So besides what we're doing in the new UI that I just showed you with those three dashboards, we're also doing UI improvements in the pro version of DefectDojo as well in the existing or current UI. So product metrics have been redone. All of those metric menu items on the left bar menu, this is the dashboard, critical products, product type metrics have all been redone with new charts. Sort of get a little teaser example right there. So what are these current UI improvements and what do they look like?

13:53
Well, here's an example of the new chart that's on the dashboard that shows you historical findings where you can in fact switch between the various severities and sort of redraw the chart in real time, depending on what you really care about, as well as hover over it and see the specific counts in those different sections of the pie like this. As well as, oops, there we go.

14:22
As well as this is the chart of reported severities by month. Same thing. You have the mouse over ability to look at exact values. And I can also filter this stuff in and out based on the criticality or the severity of the findings. I can turn off or on things. What does this look like for Realsy? Well, I actually do have this in DefectDojo. So here's an example. I'm going to refresh this right quick. You can see I can hover over these, get exact numbers.

14:52
These all work nicely. You can also tell when I added all the data for this demo. The critical products, same thing. Let me refresh this guy. You get to see them in action. So you can actually label products as critical and DefectDojo. And then if they are critical products, they will show up in this chart that gives you an idea of how good or clean they are versus having high and critical severity findings. And then the metrics dashboard as well is here.

15:21
All right, severity by month. These also work by turning off and on different aspects of these. So I can just look at all the highs and criticals, let's say, if I don't care about those others. They're all interactive. You can interact with these, do whatever you need to do. Things like top products by severity. If I only want to see what products have criticals and highs, I can filter that down like that. Nice stuff with these new charts. Very interactive, very cool.

15:52
And so what's available today? Well, that is the current UI, the charts and metrics. Like I just showed you, all of those chart and metrics pages, that's as a version 2.35.2, the pro version, that's up on our SAS. These are all the different things you'll see. I just kind of quickly did a fly by through those. What's next? We're going to be rolling out the dashboards into the new UI. That's the next.

16:17
That should happen in the next couple of releases. We'll have those rolled out and available to all of our customers. And then future ideas. So just as an aside, I'm the CTO. I'm, I do a lot of the R and D for DefectDojo. I've been, shoot, I was around when it was on a whiteboard at rack space. So I've, I've kind of been thinking about Dojo for going on 11 years, I guess, based on Greg's last LinkedIn post. Um, what are we doing with this? So, so we've got this metrics 2.0 idea. We've got an MVP.

16:47
that we're about to launch with the new UI. And we have the existing graphs in the current UI. What's next? What else are we thinking? So once we sort of get the MVP out the door, we're looking at other area-focused dashboards. We don't really have a short list yet, but we realize there's probably something there. And we're kind of waiting on customer feedback to let us know what are the things that they think should be there. It's one of the reasons why we do these MVP launches is to get the...

17:12
something in front of the customers as soon as possible so they can give us feedback and we can understand like what what are the other things that are of interest to the to the user base of DefectDojo. And then customizable dashboard so maybe of those three major dashboards there's two in this one and two in that one and one of the other one that are really important to you and your program. Give the user the ability to sort of combine and pick pieces of charts and graphs and numbers and whatever and put them into their own customized dashboard.

17:41
Maybe even do that personalized dashboards, right? To take this up a level. And not only have a customized one for the Dojo instance, but maybe a personalized one for each user. And then we've looked at even adding more ability to sort of do filter driven dashboards. I've showed you some of the interactivity where I can select and deselect things. Maybe also provide some other filter capabilities across, say I only want to see this chart for these three products versus all products, that kind of a thing. But like anything we do with DefectDojo,

18:11
We like to get the stuff out quickly as we can to customers and then get feedback because we're always listening to our customers. We've had an incredible amount of great feedback from our customers. And I love hearing from them because I think the team here is pretty darn clever, but we're not as clever as the group of all of us together, getting together and putting our heads together on what is the best next best thing for DefectDojo. So those are sort of our future vision is get this out in front of the customer base and then see what feedback we get.

18:42
And then bam, that's all I got except for questions. So please hit me up a few of questions. I'm happy to answer. We do have a question from Charlie. And Charlie, I can open this up if you want to, to let you talk, if you want to ask yourself, but Charlie likes the interactive charts. Are we, and we might've just entered this. Are we able to drill down to get more specifics in the interactive chart?

19:06
So to an extent, yes today, but not an infinite drilling down, so to speak. So I, depending on the chart in the particular dashboard widget, a lot of them allow you to do some drill down and actually do cross filtering. Like I showed you where I could click on that one product and it redrew the others based on that product filter selection. So yes. Um, but it's certainly not like an infinitely drill downable. I can't today started an endpoint and sort of drill down our product.

19:34
our data model, which is what would be endpoint finding, test, engagement, product, product type. We can't quite do that level of drill down yet. That's sort of the next evolution of this is find out where customers want more drill down that we have at launch and then add that in. Great question. Does that answer your question, Charlie? You're off mute if you want to talk. Yes, thank you very much. Absolutely.

20:06
Um, any other questions about either the new metrics or anything else? This is your time, uh, to get, pick the CTO's brain about anything you're working on in, in Dojo. Wait, we do. Um, okay. Wait, thanks for the presentation. All right. Is it already known when this will be released? Is it B3? Uh, it is sooner than B3.

20:36
Like I said, we've got all of the current UI charts redone. The dashboards and the new UI will happen as soon as we can get them done we're working on them today. Ideally in the next couple of weeks we should have these rolling out, I don't have a good beat on the exact timeline.

21:03
And so I think we figured out all of the sort of wrinkles of dealing with the library we're using to draw these charts. And it's just a matter of rinsing and repeating those things because we already have the charts picked out. We have the data for them. We know how they work because we have all these POC charts done. Now it's just a matter of taking these widgets and dropping them into a place in the new UI. So it should be soon. I don't know, next couple of weeks, next couple of releases, keep an eye on things. We're going to get these out as quick as we can because I know people want them.

21:35
And we'll definitely post when everything is live. So you'll see it if you're in the, if you follow us on social or get our newsletter or in the Slack channel, you will know.

21:56
Any other questions? As I said about this or anything else? Yeah, one just dropped. Let's see, unrelated to the new enterprise, new metrics features. What are the main differences between pro and enterprise? Uh, okay. So pro and enterprise, um, I haven't done a demo in a while to, to be able to just roll, have this roll off my tongue, but there's several things. Um,

22:19
You want to give it to Greg or Zach or you want to do it? Yeah, Greg or Zach, they want to charm in. You do, you do a lot more demos than I do, to be honest with you. I've been back in the back office working on getting metrics out the door.

22:34
Yeah, either one of you, Greg or Zach, want to take that? I can allow both of you to talk.

22:44
There you go.

22:46
All right, Greg's talking. I'm the lucky winner. OK. You're the lucky winner. Of course. You were first. Happy Wednesday. Yeah, so generally, there are a lot of differences between the Pro and the open source. So on the Pro side, we do have the brand new UI with a ton of additional insights. We make automation also extremely easy with Pro, with the new universal importer.

23:16
draw the line between open source and pro from a philosophical standpoint is around automation and insights. That's tends to be how we think about it. We want people to be able to get incredible value out of open source and even the ability to achieve some automations. Pro is just out of the box ready to go with all those things like the new metrics, like the universal importer, like tunable deduplication.

23:45
Those are the primary key differences from a philosophical perspective. That's how we think about though, in terms of how we develop open source and pro and deciding what should go where.

24:00
Zach, do you have anything you want to add?

24:07
Yeah, I think Greg, I think Greg captured it. I mean, I think really the new UI. Customizer customizable dashboard along those same kind of philosophical lines that Greg was talking about. Um,

24:24
There's just a lot of things engineering wise we can do for our pro customers that are harder to do an open source. Like we have a, if you're one of our, our, our SaaS customers, we have a job that goes every 24 hours, new EPSS, uh, um, stats are released and we pull those in, cash them locally, and then have a job on the pro SaaS, this is an update. All of your findings that have the CV that matches that EPSS that's something we can do for our pro customers. Cause we.

24:53
like there are SaaS customers, it's very easy to cache them for them, right? You're a little more DIY if you're using the open source version. Yep, I think some of the other things to highlight might be the deduplication algorithms that are available in the platform in general. We can help tune those algorithms. That's really helpful when it comes to tuning across similar tools. So, two of our SaaS tools.

25:22
Hey, wow, we all answered the wrong question for Charlie. He was meaning the difference between Pro and Enterprise license. You wanna get that Zach? Cause that's totally in your homework. Okay, Pro and Enterprise. The features and functionality are largely the same. It really comes down to what level of work we're gonna do on the agreement level. So not a huge amount of difference between, no difference in the platform itself.

25:57
Thank you, Charlie, for clarifying the fact that we took the wrong beat on that question. I appreciate that.

26:08
All right, thank you. Anyone else have a question? Well, we have all the minds here.

26:18
All right, well, I will just then thank you and I'll close by saying if anybody on the call is attending the OWASP Global AppSec Conference in Lisbon, we will, Matt will be speaking there and then we will also have a booth in the startup area. So we'd love to meet you in person and talk more dojo things with you. And if you wanna keep up again,

26:47
with anything that we're doing and anywhere we're going to be and what the topics of office hours are. You can obviously follow us on social, but you can also visit our community page and sign up for our newsletter, and that way you will get all of our events, both virtual and in person.

27:06
So if there's nothing else, thank you, Matt. And thank you everybody. All right, take care everybody, have a good day. Thanks for the great questions, I love them.