Transcript

00:07
Hi everyone, happy Thursday. Like Dawn mentioned, my name is Greg Anderson, and I'm the creator and CEO at  DefectDojo.  Happy Thursday. Thanks for taking the time to join our office hours. I'm excited to show you the latest and greatest on what we're doing in the AI space.  Before I start the presentation, just a  quick heads up, there is a massive thunderstorm going on outside my office. So if you hear some banging or whatnot, it's.

00:34
It's just thunder. If I disappear, it's not because  I don't want to be with you, lovely people. Just, probably lost power. But with that caveat,  let's dive into our agenda for today.  We like to have, we like to split our office hours into two sessions. One talking about what's new and exciting and commercial, and then what's new and exciting and open source. And so on the pro side of the house today,  we have our new MCP feature, and we'll talk about

01:03
Why we created it and what's next, and how we've seen AI working well in security, and where maybe AI is not working so well in security. And then some of the open-source updates that are coming as well.  So the new feature in dojo pro is, is MCP. And so if you're not familiar with MCP, it's short for model context protocol.  It's essentially a way for.

01:29
AIs to connect to and utilize APIs or a translation later for AIs to access platforms. And so  I have been a very big skeptic of AI and security until MCP arrived.  I really think MCP is really transformational when it comes to security.

01:56
I went from a non-believer and a skeptic to honestly a total believer in what MCP can do for security.  We have spent a lot of time playing with AI to understand how it could help security professionals.  And one of the things that we consistently saw with any technique that we tried to use an AI smarter is that it  failed to understand nuance that is so, so important for security data.

02:24
And so,  like two different cross-site scripting vulnerabilities, for example, like  DOM-based cross-site scripting versus traditional stored cross-site scripting.  When one of these types of issues would get fixed, AI consistently failed to understand the nuance.  And the magical line in the sand for us always seemed to be 10 megabytes. I don't know why that is.

02:47
But things got consistently worse with regard to hallucinations, specifically around 10 megabytes.  And so we first started playing with rag pipelines to try and make AI more intelligent with regard to security data, vulnerability triage,  vulnerability intelligence, et cetera.  And it just didn't go super well.  We also tried traditional prompt engineering, et cetera.

03:16
but nothing was really fruitful for us until MCP came along.  AI has some great applications in security, specifically with anomaly detection, but with regard to security intelligence, vulnerability management,  we were very unimpressed to begin with.  But then MCP came along, and this greatly changed our perspective.

03:41
MCP, we see as is kind of the best way so far to get data into AI that is meaningful and useful and able to correctly separate context, so far at least. And so we talked about the challenges that we saw with rag pipelines or just generic prompt engineering. But what we've seen is in hooking up Dojo to these different AIs, we've typically been able to make that AI.

04:10
As performant as Dojo is or as smart as enriching, able to make concise and conclusive decisions on what you should do in security or what you should fix or how you should prioritize your time, or what training should your employees receive. And we'll actually demo that shortly, so you can see it in action.  It's one thing to talk about it; it's another thing to prove it. And so  the other thing that we're thinking about is

04:38
As we watch AI agents get developed, the thing that seems to be emerging is agents really need workflows to perform well.  And a big part of what Dojo is is just a workflow. So the more we build this out, the easier it will be for agents to be accurate and perform accurately because our workflows essentially keep them on the rails rather than trying to.

05:05
you know, create workflows out of memory or  keeping the agents on track, essentially. So,  yeah, I mean,  before MCPs again, we just  saw AI is just wasting a ton of money, a great way to burn up a lot of infrastructure.  And  with  MCPs, just the focus that we've been able to add  and what we'll show you shortly has been...

05:31
we believe just totally transformational in terms of what's possible with regard to having a conversational approach to security or using AI for security intelligence to make decisions about your vulnerability management program or really anything in security.  So what we're rolling out  is it's very, very important that these things be secure. I think that's the other just really big hesitation with AI and security.

06:00
is that we see all these vulnerabilities and all these new attack techniques coming out every single day. And so I think this is one of the largest challenges to AI adoption insecurity is the innate security of the usage of AI. so to protect customers, we're rolling out the MCP on a per customer basis, and it will initially just be accessible for super users. And so this is just to keep  everyone's data

06:29
Um, completely separate.  I think the other caveat that I want to mention with MCP is so for this initial launch, we think this is likely best for people that are developing their own models to bring the intelligence and decision-making capabilities of dojo to an AI of a customer's choice. Uh, but primarily organizations that have their internal models to date, just so you're not, uh, trusting a third party.

06:57
With all that  enriched security data, essentially.  And so there's still a couple other security considerations to  work through, but we think this is the best way to protect customers. We have a lot of organizations that are  turning away from OpenAI or some of the larger providers because they have had security incidents. They're building their own models to keep everything internal, to not expose that data to a third party.

07:25
And so we see that as the greatest and initial application. You can hook this up to open AI. You can hook it up to anthropic. You can  hook it up to any AI that supports MCP. It's just not something that we're recommending or endorsing currently because we don't want to introduce that third party to your data.

07:46
So with that said, I will flip over to our demo. So for my demo, in terms of what I've set up, this is connected to Claude for simplicity. It is exclusively using test data for that reason. Let's see, what are the other caveats? Our testers really like Super Mario, so it's Super Mario themed. And yeah, we'll just walk through a couple of the examples of things that you can ask Dojo to help it make.

08:16
decisions with regard to  your security program data prioritization, et cetera. So let me flip over to that. Oops, we skipped ahead.  But yeah, so  what I've already done is just set up, I've tied the MCP to Claude. And so now Claude can use the data from DefectDojo to  make decisions related to security.  So.

08:43
You could ask it things like what products  may need more testing.

08:52
And so when we ask this, it'll go off and think, and then it'll pull the DefectDojo data.  And  I think this is a really interesting analysis. So it's essentially looking at the number of records that are available and looking at the number of findings  and making a decision on say,  if that makes sense relative to the importance of a product, you can ask it  say to tell you

09:22
What findings you should spend time on.

09:31
Some other prompts that we've seen are incredibly successful are  like what training should our employees spend time on? And so we'll take a look at that one next. The other thing that I don't like about Claude specifically is it does show the requests. And so we've tried to prompt it a multitude of different ways with Claude specifically to not show that, but we haven't had success with that.

10:01
The other thing that we're thinking about as we roll this out to customers is if there are  responses that you think doesn't conform with what the answer should be,  there are things we can do  on the dojo side to actually tweak those things to  equivocate two prompts or questions together and then annotate how those responses should be addressed.

10:38
Yeah, so really you can ask it any question about your vulnerability data. We've seen it struggle with ownership. Actually, that's something we're working to address through annotation, but it will give you information based on the type of findings for the type of training you should recommend. You can also ask it to generate reports. Sometimes we've had trouble with generating the size of the report, but we can go ahead and try that.

11:11
Using all my finding data on this instance typically gets it to time out, but sometimes it'll self correct. So we'll give it a go.

11:40
The other thing we've seen people do successfully with this and kind of it's very early previews  is  also generate reports that are specific to your format. So we've seen the ask a lot for custom reporting, which can be  kind of challenging, but you can also we've seen  feed it sample templates of reports and it will actually generate in your preferred formats.

12:26
But yeah, so there it goes, generating away. think  as we continue to enhance this, it can probably replace  DefectDojo's current  report generation capabilities. You can ask it to do graphs.  Really, you can ask it anything about your data. And so,  as we roll this out to customers, I think  the thing that we're asking for your help on is just noting  anything that you experienced that  you didn't get an answer that you were happy with.

12:54
essentially, but we found this to be  much more accurate than the other forms of  AI manipulation that are  available.  And so with that said, I'll flip back over, but hopefully that gives you a taste. Again, we don't recommend cooking this up to Claude just because we don't want to expose that information. What I'm showing you today is exclusively for demo purposes.

13:19
Again, we think the greatest application for this out of the box and why we built this is primarily for companies that are building their own internal models. But we also realize that not everyone is doing that, especially smaller organizations. And so  the next step from this  is making those things accessible to small organizations. And so to flip over to how we're going to do that, what we're here to do at Dojo at the end of the day is just always make the lives

13:49
of security professionals easier. The job security professionals have to do is so, hard. So, you know, whether it's, it's AI or, know, the native platform that we've built, we want to make  your job easier, be a force multiplier, make the job of implementing and keeping us a company secure actually possible. And so I'm looking over to the future, what we're going to build next.  We are,  we are taking a real dual approach.

14:18
to  AI. So I think we're still a little skeptical of  the overall direction in terms of how these things come to fruition.  But to make a really good  AI agent, we have to build the workflows anyway. So we'll continue to build, I think, the key features that security professionals are looking for, the feedback that we've received on additional functionality that Dojo can have,  while simultaneously

14:47
updating and annotating the MCP for great AI usage. So you'll essentially get to choose, you do you want to use the traditional platform or do you want to take an AI first and conversational approach to security? And so this  is a pretty cool update, but if you're not building your own model, maybe it's not that exciting, but  I'm trying not to share like too much in terms of like what is going to be next or how, but

15:14
We've talked about a Genic a lot. And so I don't think it's particularly a secret that  many, many companies are building AI security agents, but what will be, think really unique  about ours  is the way that is being done. It'll be  totally different from  what people have seen so far. And I can't, I don't want to share the underlying infrastructure of how it's being achieved, but  I think it's one of the most exciting things.

15:43
about DefectDojo is the way in which we've developed the platform and how we approach features. We want to make sure that you have a trust and confidence to use these things. And so  I know these are two like really simple statements, but  the truth is, is that there's a lot of flash around AI today  and many of the things don't work as advertised.  And the security component for our industry specifically is just so key. And so

16:13
when we bring an AI agent to market,  everyone can be confident that it will accomplish these two things and we'll have the data to prove it. But we also want to make these things  accessible in secure ways to smaller organizations. And so  that means doing something pretty different in the space compared to how others are building on open AI or anthropic. They've essentially taken horizontal approaches  and that is not our approach, but.

16:40
I expect we'll have an update on that in our next office hours. Dawn, does it make sense to pause there before we go to OS updates for questions? Or do you want me to just, or should we just keep rolling? we do have, we have one question. Do you have a curated list of AI providers that you'll be targeting or development testing?

17:10
Great question. So we're testing against all the major providers today with regard to the MCP, but we don't recommend  hooking the MCP up to those just because it shares data. Like we can't stop you from  hooking it up to those providers, but it isn't recommended today just because of the security implications. We recommend internal models  just because if OpenAI has a security incident.

17:38
We don't want your data to be impacted by that. And that's one of the things that we are trying to keyly protect against with regard to how we're approaching  AI is that we don't want you to  indirectly incur a security incident, which is what is going to happen with  everyone else that's building horizontally, that's building on top of open AI,  et cetera. So  MCP gives us a universal interface to connect these to.

18:07
any AI, whether it be, you know, your own or a third party or something that we're developing and publishing.

18:15
Okay, and then another, oh, okay.

18:21
We use Gemini Vertex as our provider that ensures our data security by project.

18:30
So not, and then there's another question.  Do you have a wishlist guide or guide for people testing some of these new features? So,  you know, to keep them from  hurting themselves or their results. No, no, as long as  you trust the AI that it's hooked up to, I would say go crazy.  The thing we're acutely looking to understand is

19:00
where this can fall down and where it can get data wrong. And so one area we've seen it struggle with is ownership because ownership requires essentially a table jump in Dojo. And so sometimes it is not great at making those associations, but that's something that we're  aware of and looking to address. think,  you know, everything we do at Dojo is extremely collaborative with our community and customers.

19:26
And so the other things I would mention is we're a little behind on getting this deployed in full transparency.  The infrastructure rollout is happening eminently, but I think customers can really realistically expect to see this likely  in the next three weeks. And so  just in that spirit of collaboration, if this is something you're looking to access sooner rather than later,  please let our support team know just so we can track that.  otherwise, go crazy.

19:55
We pride ourselves on being extremely collaborative with both community and customers to deliver  what you actually want,  working hand in hand to make sure that these things hit the mark for what you're actually trying to solve in security.

20:13
Okay, great. Thanks. You can move on to the next part. No more questions right now.

20:19
Wonderful. Okay, so  turning our focus to the open source side of the house.  Uh, on our last office hours, we had a pretty big update  and you know, if you didn't have a chance to attend, I just wanted to quickly recap our quarter update on the open source side of the house. So, uh, open source has, I mean, just steadily grown. I think one of the things that's so crazy about DefectDojo is open source space is that

20:47
We've never really seen a fall off of growth in our history. It's just consistently grown healthily, which is really great to see. You know, at the end of the day, we want to  do something great and security that isn't just about money, to be honest with you. The commercial side allows us to do these things in open source, but just to recap, Dojo has crossed 450 plus  contributors. Yay.

21:12
If you are interested in contributing to open source or customizing the open source version of DefectDojo,  the OAS Slack channel is a great resource around that to get help. Again, we're very friendly, we're very inviting. If you submit a PR and it isn't perfect, we're not going to be upset.  We are making a couple of key updates to the open source platform just with how some of our dependencies have aged. And so I think that's probably...

21:39
the key update for this office hours on the open source side of the house. And so we're aware that  some of the dependencies related to report generation and the global calendar have aged essentially and they've reached end of life. so again, always thinking about that security mindset,  we've  commissioned a contractor to get those updated. Essentially there was breaking changes so we couldn't just  immediately update and use the latest and greatest there.

22:08
But I believe that work is close to being done.  Matt would know better than myself, but I expect we'll see this change eminently. I don't think it will have a significant impact on the actual user experience or what you may be used to seeing in open source, but it was just something that was important to get done for the sake of security. And then this also extends to some of the  UI libraries in open source. So  the pro versus open source UI is incredibly different.

22:36
At this point, one is  a reactive app and one is a little more  classical, shall we say. And so  we've been also updating a couple of the dependencies there. I don't believe that is live yet there either. It's just, it's just a little bit of polish. It's not  a new UI experience in open source or anything like that, but similarly to the.

22:59
report generation dependencies and things of that nature, they've also aged and just needed some updating. And so it gives us an opportunity to do a tiny bit of sprucing. I don't want to oversell what those changes are, but also continue to make sure that dependencies stay in a good state as those  updates have changed. And then finally, this is something that we've talked about in previous sessions, but with regard to

23:26
The V3 version of DefectDojo, we're starting with the API and consolidating things that  we've received feedback on, essentially. So  the V3 portion of  the API, we're building it in this way so that we don't break V2 calls, essentially. We don't want to  just update the underlying API and  cause someone's stuff to break in open source.  It's incredibly important for us that...

23:55
You know, you continue to have a great experience with whichever version of Dojo that you're using. And so this is essentially how we're approaching it. And I think you'll see this here soon. Not mentioned in the previous presentation. I know we're also very close to supporting  CVSS4 on both the open source and pro sides of the house. So all  great updates.  And I think that concludes the presentation. I'm happy to take any questions anyone has about

24:25
Pro,  open source,  questions about the platform. I'm not as technical as I used to be.  You know, really, I push paper these days, but  I'll do my best if you do have questions on either.