Talking DevSecOps, SolarWinds, & Diablo with Keith Hoodlet | AppSecEngineer Podcast

Keith Hoodlet is a leading DevSecOps thinker and passionate advocate for Secure Software Development. He currently works as the Senior Manager of Application Experience at Thermo Fisher Scientific. He talks about the evolution of DevSecOps in conjunction with the rise of DevOps, and observes that security is often deprioritised at companies that most need it.

Talking DevSecOps, SolarWinds, & Diablo with Keith Hoodlet | AppSecEngineer Podcast

AbhayHi  everyone. Welcome to another episode of AppSecEngineer podcast. Today, I'm talking to Keith Hoodlet. Now Keith is somebody that I've been following for a while now. In fact, we met in the OWASP AppSec, Melbourne days , and it was very interesting to hear his talk on DevSecops and Scaling Application Security.Now Keith has been working in application security for quite a while I think. And , today, we're going to have a discussion with him about, of course, application security like this podcast does, outside of that we're going to be talking about things that he loves to do, which is essentially DevSecOps, AppSec, Scaling Applications Security, maybe a little bit of Red Team stuff and Supply Chain stuff as well.Let's get started. I'm going to open up the stage to Keith. Welcome to the podcast, Keith. KeithThanks. Okay. I'm really glad to be here and great to catch up with you. It's been almost three years now since , since OWASP AppSec day 2018, back in Melbourne, Australia. It was an awesome conference and opportunity to get to know one another.I loved to talk there by the way. I thought it was really great to see what you folks are doing, so really cool to catch up with you now and kind of see what the world has done to everything in the AppSec space since I gave that talk. So, I guess briefly on my background. So I did have a role change back in April of last year, but at the time when we first met, I was the senior manager. At that time I was actually the manager of DevSecOps at Thermo Fisher Scientific, eventually went on to become senior manager of Global DevSecOps, built out a team here in the Americas and then out in India.And then, we had plans to actually form a team in Europe prior to all of the COVID-19 stuff. So that, of course, got delayed, but and now I am the senior manager of Application Experience at Thermo Fisher Scientific. So I'm responsible for all of the office conference and service technologies within the company, which is, as you would think , Office 365, Microsoft Exchange, some E-mail. And then, from a conferencing perspective, teams, WebEx, zoom, all that fun stuff and service now as the major service technologies platform. I'm sure there are others that were also working pretty heavily in, but that's the big one that I pay attention to a lot. So it's a very different world being on this side of the business.I'll tell you, it's kind of like a security guy from inside the business, like I'm in [Laughter]. AbhayYeah, in fact, that was interesting because.. I mean, I always pegged you for a ... the AppSec was, I mean, that's what I saw in your title before. So this seems like a pretty big zoom out, right, in terms of what you used to do and so what do you, what do you think? I mean, usually it's the other way round. People, you know, zoom in but you have zoomed out and you've gotten a different perspective of the world.  What has changed? I mean, what do you, what do you see as some of the major changes that you've seen in the zoomed out view of the world. KeithPriority is maybe the big one to talk about, but yeah, it's, you know, the way that I often reflect on it's kind of like climbing different mountains, right. So in the AppSec space, you are very focused on what's on your mountain top and what are the challenges that you're trying to overcome and the space as a whole is continuing to grow and evolve. But now that I'm in such a, a larger IT operations perspective, it's almost like I've, I've stepped foot onto a much larger mountainthat's part of a much larger range and because of that, it's, it's given me this perspective of an appreciation for I should say really, what's critical and why and what does the business really care about and how to actually get people to take action on things that are a challenge in ways that are a little bit different than, than a security perspective.I mean, a really good example is email. Right. Everybody uses it for better or worse. Right. This meeting could have been an email. Well, you know, all the emails still go through and at the end of the day, if it doesn't, if it stops for whatever reason or people don't get things delivered appropriately, the world freaks out.And it's just like, I wish that would be more the case in security, we're regularly around the world. But when email stops, everybody is calling you and being like, why is this broken? You need to fix this. And so , it's definitely changed my perspective, but I also have built a great appreciation for the challenges quite frankly. I think that when it comes to all of the business demands that you see going into a service management platform , they have great ideas in terms of what they want to do to automate, what they want to do to streamline their workflows and their process. And the challenges, of course, you never have enough time to do all of it.And so prioritization is key to everything that I do within my organization now, because at the end of the day, there's just so much demand that we can't possibly fulfill it all. Not to mention, we've got to keep the ship running, right. Like, so on top of actually doing all of these other things that the business wants us to do, we also have to make sure that the things that we've been doing don't break . In AppSec, it's almost the opposite, right.It's an equal and opposite thing where you have business as usual operating and everybody thinks that it's fine, but you found something that is broken and you have to convince people that it's broken and then you have to convince them that they should fix it. Whereas on this side of the house, it's no, if this thing is broken, you absolutely need to fix it right now because it's impacting business operations and everybody cares about it. And so that's, that's been, my great appreciation is almost I've got to go out and, and talk to everyone about why this is something we need to work on. And now it's like, yeah, I have to pull people back, and it's like okay guys, like we know it's broken, we gotta fix this. Like calm down. And , and the crazy part for me has really been that I was put into this role as part of a bit of a reorganization where the corporate infrastructure and security organization was previously just security. And, so now it's a situation where in April of last year, middle of the global pandemic, as things really were starting to kick off , I was basically taken out of my DevSecOps focus because that team has and continues to be a very smooth operating, a very efficient organization based on all of the foundational stuff we did together. And they said, Keith, there's some challenges in this space and we just, we need you to solve this. And it's like, Oh, by the way, everyone's going remote like next week.So good luck. And so that's been, that's been a great opportunity for me as a whole, it's given me that appreciation for the work that these teams do. Whereas before I didn't have a lot of , you know, just knowledge of the challenges that they face. And quite frankly, at the time as a security professional, I kind of didn't care.I was like, this is a vulnerability and we need to fix it. I don't care that you have all these other problems. We need to fix this thing. And I see that as one of the great downfalls of security professionals in general as we fail to see the forest for the trees, right. We fail to recognize that there are problems that people are experiencing in their own operations teams or in their own infrastructure teams that aren't security related, but that people are screaming at them about because they need to get fixed or they need to be built. And then, security comes in and also starts screaming and yelling. And it's like, guys, like at the end of the day, this is important. Like we need to fix the security problems and we recognize that, but we also have to make sure that the email keeps flowing appropriately so that business keeps operating because if it doesn't, then doing security doesn't matter at that point. Right. It's like if we can't do business, then being secure, irrelevant . And that's something that a lot of security professionals, I think, miss and quite frankly, I was one of them.And now, it drives me a little bit mad when I see other security professionals whether inside or outside of our organization basically put up roadblocks all over the place without stopping and realizing like we have to work together. We have to prioritize together. We have to have respect for the things that they're bringing forward from a security organization perspective and that the operations team is bringing forward from a, you know, traditional or technical their perspective of just something that needs to be fixed or something that needs to be built because it will mean more business revenue or , you know, higher throughput of our operations mechanisms. So that's a long way of saying, yeah, it's a lot more priorities, a lot more stakeholders, a lot more tasks that we need to get done and asks that come of us.But still super important, phishing is still the number one problem that you see from a security perspective out there, that you still got to worry about. Of course, then, when you're looking at other things like just really old systems and the infrastructure that you got to go fix, well, those things tend to be exposed to the internet for better or worse because they're exposed to be internet,you gotta fix them. So it's, it's an ever going challenge or ever , ever enduring challenge. But as I like to say, keeps the day busy , job security. [ Laughter ]AbhayYeah, absolutely. So how did you get your start in security Keith? How did you actually start off ? Did you start an AppSec or did you do something else and come to security or how did you actually come into this space?KeithIt's a great question.  From the... I came in through the window. [Laughter] So, you know, I'll ..I'll go back. It's like to say there's a longer history here but for me, I think , everything changed in 2017 at DerbyCon7 when I gave training for offensive web security with my mentor and friend Casey Dunham.So I'll go back though because, I think, that there's some important build-ups in terms of just my experiences in life that led me into security and then eventually led me into Thermo Fisher and that eventually led me into a higher-level role and leadership of the organization. So when I was young and, and to give you folks an idea I'm, you know, in my mid thirties, right.So if that gives you some idea of what the internet was like when I was growing up; well, we didn't have cell phones , that was a pipe dream. Cable internet was like not really a thing until I was in my early teens. And even then, you know, a lot of the internet hadn't been built out in the way that it exists today. Social media was not even really a thing.I mean, I remember Angel fire and MySpace and all that stuff way back when, but that was like late teens. And so where I grew up is a part of, it's called New England , here in the United States. It's basically everything that's North and East of New York. And so , as you can imagine it's, it's very rural depending on where you are.I mean, areas like Boston are , you know, huge mega cities but where I grew up, it was a farming town and there were more cows than people. And , and so, as you can imagine I had a very small class of friends growing up and I was maybe, I don't know, a 10 or 15-minute car ride from most of my friends.I had one good friend, Tom , who just lived down the street from me. And we'd always, you know, hop on our bikes and do stuff over the summer. But as I turned about 10 or 11 , we had our very first desktop computer in the household and it was a family computer. And right around, I dunno, 96 or 97 , right around, my time that I got, I think it was Diablo , Battle.net came out.It was a blizzard entertainment game for Diablo and, you know, playing online was, it was something that I was super excited about and, you know, that I learned pretty quickly as I was getting deeper into computers because my friends who were far enough away, where I was only really able to interact with them online.As I learned that I could connect to battle.net back when it was like IRC via the Diablo trial account, which I thought was kind of weird, but then I could connect via Telnet with the Diablo trial account key, which was also kind of weird. So I started playing around with network protocols and all that fun stuff.And the most interesting thing that I had discovered is that you didn't need a password when you were using a Diablo trial account on battle.net. You could literally put in any username that wasn't currently logged in and it would allow you to assume that username, which I thought was great because then I could impersonate my friends is, is, you know, pulling pranks on them.But then, I took it a step further and I recognized that I could probably connect more than one Telnet connection to battle.net using this trial account method. And so, I used visual basic. I had taught myself some very simple visual basic skills to then take my 2088 dial up and this was kilobits per second here.Folks, not megabits per second like super slow. And, I built this bot that would allow me to connect as many connections as my 533 MHz PC could actually support 2088 dial up to battle.net. And the, the funniest thing for me was I could basically have all of my bots and I could get like, I don't know, 10 or 11 bots connected and I could have them all message a friend at the same time. And for me, it looked like one message based on the way that I had built the interface, but for them it flooded their screen. And basically because they were on 2088 dial up on super slow computers, they would basically just fall right off battle.net.So I could actually win games by playing like StarCraft or Diablo and just drop someone off line by spamming them with a bunch of messages. And that was kind of my first foray into security as being a cheeky kid on the internet. Lucky for me and, and quite frankly, I have theories around security related to this.But a lot of this stuff wasn't locked because disc space back then was really expensive. And, you know, I'm convinced that the reason that the security industry or most industries we know as technology industries today exist is because disc space is now cheap. It's incredibly cheap. If you go out to the cloud or if you go out to , I don't know, your local store and buy a terabyte drive like a thumb drive or what have you, it's crazy. But that's kind of how I first got into computing. I thought it was just a lot of fun. I was this punk teenager doing all sorts of stupid stuff. I followed the folks at the loft down in the Boston area so they were, you know, maybe a few hours South of me, and so I was following all the things that they were doing online at the time.But I was too young to actually like, you know, travel down there and meet any of them and kind of get involved. So through high school and early college, I spent a lot of time learning how to program but ultimately when I went to university, I decided, you know, you can throw me in a dark room with pizza and mountain dew and I can write this great code and I can do all sorts of fun stuff on the internet, but I wasn't good at interacting with people.And so I actually ended up pursuing a degree in psychology, because human beings are the people that are on the other end of the computer. And I wanted to understand myself and other people better. So that's a long way of saying eventually I graduated in the, the middle of the    downturn of the market after the housing market crash and the global financial crash . No one was hiring someone that had technical capabilities or technical knowledge but a psychology degree.I was, maybe, four years too early for Facebook because they were still just at that phase where they were allowing more colleges to join Facebook. And , nowadays, they're like the number one employer of psychology degree holding technologists, because all they're doing is basically psychological manipulation.Like that's their, that's their thing. And that's all the sell ads of course. So, so at the end of the day, for me , I followed a lot of things as I worked through various jobs. But back in about 2014, I recognized that I really hated the job that I was in. I wasn't really doing anything with my technical skills that I had tried to keep sharp and kind of still played around with things on the internet from a programming perspective.So I actually went back to university for a couple of years to look at a degree in computer science. And that's that's, I think, where things in 2014 started kind of really turning the corner for me. I joined the cyber security club and took them to a national competition for Digital Forensics and Incident Response, which was really cool.Of course, I competed in a lot of the National Collegiate Cyber Defense Competition stuff at the regional level , which is very Blue team defense oriented and and, and really, I think, found my passion or found the professional passion that I wanted to pursue at that point in my life. And for me, I'll never forget asking my dad to help out with financial assistance when I wanted to get back to the university.And, we were working at the same company at the time. And ,and he said like, no, like there's no future in security. Like, what are you doing? You've got this opportunity to have this career at this company that we're at and all this opportunity ahead of you, like, what are you thinking? And I said, well fine.I'm just gonna do it anyway with or without your help. And, you know, took all the loans and kind of all that burden on. And I remember , you know, I don't know, it might've been a few years ago at this point, but I remember my dad saying to me after the fact like as security started being in the news all the time, he's like you were right. Like you, there's absolutely a career in security and it's real and they need people like you. So it was very, I felt vindicated, but after a couple of years at university, two years out of a four-year program , I got an offer to work at a small , Managed Security Services Provider to do what they were calling Pentesting.It ended up really being just installing splunk instances at small , you know, small government organizations and small, medium businesses. And it was like this isn't Pentesting at all. Like, I know what Pentesting is and this isn't it. So I very quickly jumped ship to Rapid7, which was a great opportunity for me down in the Boston area. I got to meet and interact with a lot of really super talented people from the Medisplay project. Jay Radcliffe, who's a colleague here at Thermo Fisher now, was someone that I got to meet and interact with, is kind of famous for hacking his insulin pump and then talking about it at Black Hat, I think back in like, I dunno, 2016, maybe earlier . And , and what was awesome about that is, if they were as Rapid7, they were at that point public, they were kind of turning the corner on the AppSec space and they started doing the whole insight platform that they now have.And it was awesome just interacting with companies and talking about, you know, how to set up a vulnerability management program, how to use all of these application security things effectively. And , and that was, that was a great experience for me to get perspective on the challenges companies are facing.What was nice about all of that experience was I was also very involved in the BSides Boston Organizing Committee. So I volunteered, I think, in 2015 or maybe 2016 and then I joined the organizing committee in 2017. And , and so at that point I had, I had started getting some mentorship from my buddy Casey and we built a training class for, for BSides Boston and basically, I was like an adjunct professor sort of, you know, just an assistant that was helping Casey out with all of the training. And it went pretty well. I mean, we had about 20 students . It was held at, I think, Harvard that year. And it was, it was just a lot of fun, you know, sitting down and training people on AppSec and talking about all the challenges.And , I remember when Casey said let's propose this to DerbyCon and I was like, okay, like, sure, we can propose it. It'll never, never get accepted. But let's do it, whatever. And then, I remember a few months later after kind of all of the, you know, closing interviews for training , Casey reached out to me and said, our training's been accepted to DerbyCon and I can't, I still can't believe it to this day.It was one of those things that for me that we can't here in the US is, is a super small, it was a super small conference. Tickets sold out in a matter of minutes, if not seconds. And here I was, you know, right at the beginning of what I really felt was my professional career in security, giving a training at a conference that was just so highly regarded by people that I had been paying attention to or listening to for so long.And it was, it was just unbelievable. In fact, I still have on my wall the, the kind of styrofoam board poster that they put outside of your classroom, you know, it's, I don't know, maybe two or three feet, like a meter, meter tall and like, I don't know, two-thirds of a meter wide and featherweight, super light.I ended up shipping it home via FedEx. That was like next door to the conference center for like $170 or something like that. And I'll tell you, worth it, totally worth it because that for me back in I don't know, I think it was September or October 2017, was just the turning point for everything for me.At that conference, I remember talking to a good friend of mine, Paul Asdourian, who runs the security weekly podcast, a series of podcasts now but he's most famous for his main podcast, security weekly. And , and I remember Paul, who I knew from the BSides Boston scene and he was a sponsor there. We had talked quite a bit . He said, Keith, you should start an application security weekly podcast and I was like okay Paul, cool, like totally on board, I'd love to do this. Couldn't believe he was actually asking me to do this. And so a few months goes by, and I didn't really hear anything from him. And, I think, in December of 2017, he said, here's the contract for the podcast, you know, the sort of sponsorship payments that you would get if, if we had sponsors on the podcast, etc., and I was like, this is happening. Like this is real. So, in January of 2018, we kicked off episode 00, because we came from zero in the development in the security world for application security weekly. And then, I ended up being the main host of that show from episode 00 to 55.So about March of 2019, I stepped away from that to focus more on some of the challenges faced at Thermo Fisher and the career that I have here. But that really changed things for me. Somewhere in the middle of that, I think, it was June of 2017, I ended up moving from Rapid7 over to Bugcrowd , which is fairly well-known for the bug bounty platform that it is.But I was super fortunate to work for a guy named Jason Haddix, who I think many in the application security space know and respect . He is probably one of the best human beings that I know. And just in terms of his passion for the space, willingness to teach people things , just over, you know, all-around he is a good guy and a good leader.And , and so I was a Trust and Security Engineer at Bugcrowd. You know, I was working with companies to try and set up bug bounty programs, understand, you know, application security vulnerabilities, talking about it all the while on my podcast. And , and at one point or another , I ended up crossing paths with, with Thermo Fisher.And what was interesting is my boss now, Brian. I remember, you know, a few episodes into the podcast, he's like, Keith, you need a new co-host. And I was like, okay Brian, like we've, maybe, talked two times, like you're a good guy but like Paul Asdourian has, has done podcasting forever and that's his network, I can't possibly ask for a new co-host. And he's like, you're clearly like showing up to this and you, you are stealing all of the spotlight, you need someone that can balance you out. And , and I took it as a really great compliment and, you know, lo and behold, a few months later , Thermo Fisher ended up running a bug bounty program briefly and it didn't go as well as they would've liked as for most companies that's the case, you know, they ended up finding a lot more vulnerabilities than they expected to find.And thankfully, they've resolved all of those, but it was one of those situations where it was eye opening and Brian reached back out to me and we were talking back and forth. You know, what static analysis vendors should I look at? How would I prioritize the vulnerabilities that that come in, you know, justtalking about the things that I was talking about on the podcast. And in April of ...  it might have been April or March of 2018, Brian basically said, okay, time to put up or shut up. We have an opening for someone to, to run our application security program here at Thermo Fisher. You talk big on your podcast.Can you actually build a program to do this? And I was like bring it on, like, let's do this. So that's how I ended up here at Thermo Fisher which, which has been an absolute dream. The CISO, Erik Winebrenner, is incredibly supportive and engaged with regards to all of the different things happening in the security space and all the things that he pays attention to . Over the course of joining , you know, I started as kind of the first employee focused specifically on application security.And then they hired a few other folks , some new leaders , some, some new engineers. And I remember, shortly after I joined, like two or three months after I joined , we ended up having a small reorganization. They said, okay, Keith, you need a new title because application security isn't going to be your only focus.We're building this whole product and software security team. And you know, you're going to do way more than just the web... web applications and mobile applications. You're actually going to be part of the process of securing our devices, like the things that go in laboratories. And, I was just like, [sound] like, okay, this is, this is pretty much as close as you can get to the business when it comes to security because the devices that Thermo Fisher makes for those that, that aren't aware, we are pretty much everywhere in the medical and laboratory science space.I mean, we make everything from personal protective equipment, masks, gowns, hand sanitizers to, you know, diagnostic machines that look at blood samples, all sorts of bodily fluid in chemical samples of all different kinds. But then, we also are with COVID-19, super involved in the testing space. We actually make, as, as our CEO calls the gold standard for the PCR tests that they do, and they'll like to swab the nose to determine that people have COVID-19. So we make the test kits, the devices that run the tests , you know, the plastics that some of the stuff gets stored into, the media that this stuff goes into to actually make it to the laboratory and then it'd be run for, for cycling in testing.And so the things that Thermo Fisher does is super important to the world not just from COVID, but all of the other research and diagnostics that were involved in. And for me to be a part of that kind of ecosystem and making sure it was secure for our customers was super important. So I said, well, you know, DevSecOps is this whole kind of new hot term.I don't really believe it exists. I think that it's just DevOps, but it is something that I think will help us get more talent. I think that it's the right thing to be doing. And it's the right way to talk about this problem. So that, that became my title. Ironically, on the podcast application security weekly, I had said in one of the episodes that DevSecOps does not exist.And so it was very, it was silly that it became my title and I was like, okay. So built the program at Thermo Fisher , you know, started with several, several engineers here in the United States and then branched out to some folks in Asia. You know, we were looking to hire in Europe before COVID-19 but then, of course, that slows everything down because, well , you know, travel froze and budgets were freezing just to make sure that, you know, no one knew when COVID started, you know, how, how it's only going to play out from a financials, from a market perspective, from all of our customers being impacted.And so , right around that time is when I switched over into this new role because for the most part of the foundation of what we were looking to do in application security or DevSecOps came to fruition, you know, centralizing your version control so you have perfect visibility into the code base across the different platforms, which also by the way, make sure that the product teams are actually working together on common problems rather than building their own, you know, 7th, 10th, 30th version of the same feature set, but a different set of software with different vulnerabilities that now have to be fixed.It's like no, no like, build one feature set that does the same thing and then share it. And so, that was super, super awesome for us. It's getting, you know, deep into the static analysis space , really looking at centralizing version control, thinking about things like dynamic analysis and also fuzzing because it's not just web applications, but compiled applications that we've got to care about. And then, working into the DevOps process, right, making sure that as teams kick off compilations or builds of, of things or they're getting their scanning done at the right time that we're appropriately kind of inserting ourselves as a security team into that space and stopping things before they go too far.But also then to your point Abhay, in the stuff that you're doing is making sure that developers at least have some semblance of security training like even if it's very... this is what this vulnerability is, this is what it does. This is why it's bad. This is why we should fix it. And here's why it's so critical.That was all part of, kind of the first call it, I don't know, little over two years or year and a half [ish ]of my time here with the company. And now, I'm in this different role where I'm, I'm still engaged a little bit on the security side. I still have people reach out to me and, you know, ask questions about how to make sure that we're delivering things securely.We're allowing remote connections insecurely. But, also... I'm now also, you know, much higher level in terms of, well, gotta care about phishing or got to care about all these other things in security that weren't my purview before, but are very much my purview now. So that's, that's the long way of saying how I got here, you know, where I started and how I got here [Laughter]AbhayNo, absolutely. That was a fascinating, actually a fascinating journey that you took us through. It was really nice to understand how you started off and it was very nice to see that, you know, you've come from a different background. I come from an accounting background and I actually finished my CPA and I realize I wanted nothing to do with finance. and, you know, [Laughter] Yes. So yeah, these are, yeah. These are interesting stories to hear from people as well, because most of us kind of have kind of stumbled upon security, right. It's not usually something we started off with in many cases where I was... some, some people today of course are starting off only in security. So one of the questions I have for you is what would you advise somebody who is kind of getting into security today, especially, I mean, application security as a space, right, because security itself is such a large space. You have so many different silos, but let's look at application security. how would you recommend, How would you advise this person, you know, go ahead with their journey into this space. KeithSo, I think that's always hard because there are so many even fields of application security now that people have to think about. But what I would recommend for sure is learning how to code, learning how to build things is actually more powerful from an application security standpoint than anything else someone could do. And that's, that's I think the thing that's often lost on various traditional security professionals. Now, pentesters that are learning how to code and are, you know, turning that into how to automate exploits are absolutely eating people's lunch in terms of just the way that they're able to get things done.But the same has been done on the security side. You know, I would say, in my experience, building the program here at Thermo Fisher , the best people that I hired weren't security people like in the traditional sense, you know, they were engineers, they were developers and oftentimes they were right out of college.Right. Like, I wasn't looking for like he must have seven years of Kubernetes experience or 10 years of Kubernetes experience even though Kubernetes hasn't been around that long, right. Like you see that all the time and it's ridiculous. But basically, I was looking for people that had an interest in building and automating , you know, solutions and also understanding the security vulnerabilities that they were, they were finding in trying to resolve, but could do so in a way that would scale up. And that's the biggest thing that I think most companies really need is they need to think about that scaling problem very early on and think about how they can address those problems very quickly. So learning how to code , especially if you want to get into the mobile space but more so as well in the, the web development space, because there's so many web frameworks out there . I gave a talk at a Hackfest, Canada, back in, I dunno, 2016, I think.It might've been 2017, but I can't recall. In any event , the whole concept that I had built the talk around was this concept that I coined to the term, but I'm certainly not the first person to do this. So I won't claim that I've invented this new way of learning, but I called it attack driven development and it was this whole concept of build an application but build it in such a way that it is intended to be broken that you are actually intended to be able to attack it in a very specific way, and then go and learn how to attack it. Right. And, it was kind of like this holistic cycle of I want to build a login function that's vulnerable to authentication bypass via a SQL  injection.And so I'd build an application that was very basic login functionality, database connectivity, etc., And then, I had to build in kind of the way that it would be vulnerable and then I'd iterate, right, so then I'd go on to Burp and I'd learn how to do the attack. And I'd, you know, manipulate things and I'd look at the logs to make sure that like certain things were coming across the way that I anticipated and how things landed in the database.And then, what I do is I'd go back to the application and I make it a little bit more secure, not like, okay, use a Regex or regular expressions. You know, make sure that I've got the character limits, like all of these other things that I could have put in place to make it like rock solid, like the most ironclad secure that I could.But by doing those little iterations, I learned, you know, what, what makes this thing vulnerable? How can I end up attacking this thing and be really effective? How can I then go ahead and turn this around and make it more secure? And all the while I was learning , in this case it was Python flask is what I was doing this with.The really nice thing is this whole practice taught me, you know, Docker taught me a lot about nginx, taught me a lot about working with , I think, Postgres databases. It taught me all of these kinds of skills working with Burp Suite Pro and a Proxy, and doing those kinds of inserting into various elements of the, of the attack chain.And at the end of it, I had this really good, full picture of the problem space but I also understood very specifically the vulnerability that I was trying to attack. And then, of course, I do with others, Cross-Site Scripting or we even did something where it's like Sandbox Escaping with the AngularJS framework and kind of the different things you can do there.And for me what was really great about it is it ended up actually leading directly into the BSides Boston training that we gave, because now I understood how to build vulnerable applications, which, Oh, by the way, is a skill that you need to have and you're going to give a training because you need to have vulnerable applications that people can attack.And, of course, you could go out and look at, I don't know , the OWASP Juice Shop Application as one example . There are any number of other , you know, vulnerable applications that are available via OWASP and other providers, but this is a way that I really understood not only how it is built, how does it break?How do you fix it? What those fixes look like? And now, I can make recommendations on the most secure ways to do some of these things and also the kind of second and third order challenges. So that's the method that I used, it worked well for me.  I was very focused on specific vulnerabilities to kind of learn as many of them as I could.And there's more and more out there today. And it's, it's a growing space, as you mentioned, like supply chain security is like a whole other set of problems to deal with. And so starting with it, you have to learn how to build things. If you know how to build things, you will be exceedingly effective in the security space when it comes to application security.And then on top of that, you'll have respect from the developers that you're working with. And respect for the developers that you're working with because you understand how challenging it is to build things, especially, you know, fully robust enterprise applications, right. Those are, those are things that have a lot more scrutiny than your little , you know, silly app that you're attacking.So that's where I'd start, is learn to build things. JavaScript is a great thing to learn today in general because it's in the compiled space, it's in the web space. You could start with ECMAScript or ES6. You could start with a node. Definitely, I strongly encourage people to learn Python. It's a great tool as well, but that's where I'd get started if it were me.And I don't know that I would necessarily say a degree program is right for everyone. I think that you know, certain individuals benefit from that sort of rigid, structured approach. I don't know how well those programs really keep up with the changes and the pace of change that we see in the, in the real world.But that being said, you know, having a, having an online presence, building a brand, you know, doing things like this where you've got some sort of podcasts even if it's out to YouTube or Twitch or whatever and doing those sorts of things will take you so much further in your career.Then you know, just sitting around in the classroom and kind of playing with Java, which is, you know, what you often see in traditional development or computer science programs today. So that's my opinion. But, you know, different serves for different folks. People learn different ways.Sometimes it's the right thing to do to get that rigid structure if you can afford it, if it's available to you and that's the best way for you to learn. AbhaySo, right. Absolutely. No, I think one of the things that you mentioned, I liked the phrasing of the attack driven development, because that's what we have done right. So in, in AppSecEngineer, when we built our courses, the whole thing is essentially, hey, this is an app that has a JavaScript prototype pollution, fix it and see how you can get into fix, how it's broken and why it's broken this way and what kind of impact it has. So that's, that's really effective because, and this, I think extends to everything, right. So you have this Kubernetes cluster, it's vulnerable this way, you attack it this way. And the same thing with the cloud where AWS, you have this and, you know, you have these kinds of misconfigured systems. So I think that makes a lot of sense simply because , you know ,you get to understand it and I really liked the point where you said build to break better, right. So if you know how to build stuff, you definitely know how to break it much better than that. And you can think about breaking it much better than... I completely resonate with that school of thought. Now, I think, one of the things you mentioned, which I also agree with is around application security itself not being this one thing today, right. Today, with the cloud, you have a completely different paradigm that we need to deal with. You have Kubernetes, which is another paradigm, which is now on top of the cloud. Sometimes not on the cloud. So where do you, I mean, how have you seen this evolve and where do you see this going? I mean, especially as part of your own work in your own company, but even outside in terms of industry trends and so on, where do you see it this way?Where do you think, what do you think we're headed towards, especially today where, I mean, this is definitely incase but where do you think this is going? KeithYou know, it's hard to say, I think, definitively but you kind of like a, almost like a range, right. And the further out you go, the less you're accurate  but what I will say is near term what I see a lot of is, as you've already identified, right. Things like containerization and orchestrating those containers whether it's Kubernetes or, you know, another set of tools that allow you to do that is going to be that near-term space that it's the, you know, microservice orientation where you can build something that plugs into this ecosystem that you can update that thing independently and it doesn't have downstream impacts or negative impacts to the rest of the ecosystem in a way that is really impactful. I think, resilience at the end of the day is, is really the name of the game. It's not that you expect to be perfectly secure, it's that you expect to be able to fix the problem that you've identified as quickly as you can with the least amount of disruption and then bringing it back into the production ecosystem in a way that now makes it more secure and is continuously and rapidly evolving effectively, to do that at scale is really hard because you have a lot of different teams and a lot of different services interacting with one another. And you really need to have infrastructure as code; meaning, you have to have a central place where you're version controlling all of this. And then, of course, you need all of the tool sets that plug into the version control to make sure that the things you're pushing are in fact secure. That's. I think, the near term, right.We're going to see a continued drive toward automation, configuration, resilience in microservices. What I would say is that it's not always on the cloud. I think that there are a lot of companies today that are realizing that just lifting and shifting to the cloud for their ecosystem is not the right thing to do from a cost perspective. A lot of the time, you know, companies need to first be able to do that sort of thing in their own data center before they can then realize the benefits of the cloud and all of the security and the resilience that they get from the cloud ecosystem then being able to go in that direction.But what I see happening after that is, I think, where things get more interesting because I strongly suspect that as more companies start to really learn and adopt the whole idea of function as a service. So  AWS Lambda, Azure functions. Google has its own iteration of this. And, I think, that this is something that you can probably emulate in at your own data center if you really wanted to.But not only getting the service down to a microservice, but literally going down to just the function. Right. And being able to say, you know, I, I want to just perform this one action entirely separate of the operating system ecosystem, you know, the web transfer ecosystem in terms of whether it's nginx or Apache or something else , you know, totally separate from the database.And, and so I think that's where we're going. Right. I think that eventually what will happen is you'll start to see companies that are going to the cloud will actually instead of taking their entire ecosystem or even the microservice that they've built, they'll start taking functions that they can offload out to the cloud because they're widely used in the ecosystem.They're fairly stable and to the greatest extent possible, it's actually cheaper to then run it in a way where you're just paying for what you're actually using and not paying for the ecosystem, compute power, storage space, etc., that you might otherwise need. And so that's where I think the next step is, is kind of going serverless as a lot of people call it.After that, I think things get really interesting though. Right. And, this is where I'm telling a lot of people to plan for this over the next decade; which is two things, one quantum computing and two general artificial intelligence or artificial general intelligence.... I always forget how they organize that.But we are seeing, especially with like GP3 where they've got this, this kind of bot or this general intelligence that can take a few words and write like an entire entire book or news article or what have you fairly natively and convincingly. I think that we are on the cutting edge of really having artificially intelligent systems that work as a concert of artificial intelligence systems as opposed to just a single, large brain of artificial intelligence. And, and that will really change the game both from a security perspective, a development perspective , communication perspective, like almost all of it, because at that point the human interaction becomes a lot different and the security interaction becomes a lot different, especially if you have an artificial intelligence system that is now maintaining your infrastructure, your ecosystem, your, your, you know, anything, right, your functions is able to patch on the fly based on what it's seeing or what's coming down from, you know, from the attackers or what have you.And so I think that, you know, at that point what then happens is after artificial intelligence gets a little bit more robust and quantum computing starts to become a kind of a broader feature set that becomes available not just in supercomputers out in major data centers, but, you know, on people's general systems that they have at home, probably not to this size anytime soon but definitely , I think, down to the desktop level. You'll need artificial intelligence to write the programs that can run on quantum computers because the quantum computers themselves are both true and false at the same time. Or, it's every iteration of the variable that's possible when it's running things and human minds think in a very serial way.We know that multitasking is just not something that we're good at and artificial intelligence will be required to write the programs for computers that are running things in parallel that are able to multitask effectively. Even today, our computers are all serial based at one point or another.I mean, they might have multiple cores for example but each core is running one thing at a time as opposed to , you know, each core kind of or each processor running just one thing. That. where I think we're going, right. Like the horizon for that is much further out maybe a decade, maybe more, maybe a little less, but start learning, you know, learn good artificial intelligence theory now, because I guarantee you if the foundations are fairly solid, it will change the game in security, especially as we get to quantum computing and suddenly, I mean, I think, it was like in the last year or two years, two or three years ago, DEF CON had the entire like, you know, artificial intelligence systems hacking for, for the one of its CTFs and like governments are doing that now.Like, soon it will be more generally available. AbhayNo, absolutely. In fact, a couple of interesting points because, yeah, I think , I saw this, I think it was the 2019 DEF CON. There was a talk by the Bishop Fox folks on how they used machine learning to actually automate AppSec, vulnerability scanning. And outside of that, one of the things, of course, around quantum computing is that I think, I mean, at least I'm not a quantum computing or crypto expert but from what I know, it's going to definitely have huge implications on the cryptography because elliptic curves, which used to be the, like the established best practice in terms of generating the large primes are now going to be kind of rendered. I have thought useless, definitely a lot less, a lot to a lot more toothless if not useless. So yeah, that's going to be interesting times where, you know, you're not going to be able to necessarily rely on current cryptographic primitives to be able to actually do. KeithYeah. We're going to get to a point where we're going to have the, you know, immovable or unmovable object versus the, you know , the unstoppable object, right. It's just like a perfect crypto meets perfect security. Like, I don't know,  it's going to be a weird, a very weird dynamic at that point when we get there. And, I suspect it's not as far off as any of us would like. I wonder what will happen first though Abhay. The 2038, you know, problem with all of the , you know, system time stuff happening where we run out, it goes all back to zeros or quantum computing and AI taking over the world.I don't know which will happen first, but we're heading for another Y2K event in 2038, one way or another. [Laughter] AbhayI think we're getting into territory much beyond , you know, the industry that we're in. I think the industry itself is not remotely ready for, I don't think, we are as an industry ready for this stuff yet.So, yeah.. that will be interesting to how we cope with it , you know, in terms of jobs, in terms of the way , you know, we do things itself. So it's, it's interesting. KeithFor sure. AbhayYeah. So one of the things that obviously , I think, you need to be living under a rock to not know this, but I think one of the things that you and I both spoke about before we started was the whole supply chain thing and the SolarWinds we had, I mean, we had 2020 kind of ending on a pretty crappy note on this stuff.KeithYeahAbhayWhere do you see this? I mean, first of all, is it.. I mean , first of all, is it solvable in the near term? And second, do you see this people actually taking it seriously or people, you know, like this, brushing it off as another, Hey, you know what this is another breach that happened and that's fine, let's move on. What kind of a response do you see this eliciting, especially from the kind of impact that it had, which is pretty huge.. KeithYeah. SolarWinds changed the game, right. The whole Sunburst attack completely changed the way that people think about this problem. You know, it's, it's been a problem that has existed for a while, right. Since open source has become a thing and since we all started pulling hundreds or thousands of dependencies into our nodeJS applications , it's gonna be a problem. It's, it's a, it's an ecosystem thing that is going to catch people off guard and in really bad ways. And, I think, that ultimately it gets back to that, that resilience thing.Right. How quickly can you, you know, identify that you have a problem, resolve that problem, get it out into production so that you no longer have it in production. And then, of course, your, your incident response activities to determine how vulnerable you are and what sort of problems you need to report on from a legal standpoint, a shareholder standpoint, you know, customer standpoint, etc..,But I think it really will , I don't know if it will be the death of open source, because I don't think that's realistic at this point given how much open source is now a part of the development process and is helping companies move faster. But, I also think that companies need to be much more aware of this in ways that they just haven't been up until now.And , and so I think that what you'll start to see is, it's almost like the microservices of things that you pull in, right? You need to, to become less dependent on a specific library or a specific product in your tool chain and you need to abstract away from that and say, I need a feature set that I can buy or utilize or pull in from multiple providers and be able to switch from one provider to the next so seamlessly that your customers don't even know that you've done it. And then, go back and do all of your incident response and identification to determine if this was ever really a problem in your ecosystem.So, I think that will be the biggest thing that we see investments in from a company perspective is getting to that point where companies are centralizing their version control on platforms that are scalable enough to support that for their organization will help. Right. Having a well-defined set of libraries, whether they're open source or closed source that you've paid forthat your company utilizes and then having your team keep a very close eye on those and then centralizing those services is going to make a very, very big difference in terms of the company's ability to respond to this problem. And so, I think, that it's, you know, we've seen the tip of the iceberg. We know it's an iceberg given how bad the SolarWinds thing was.And that's just the first, right, eventually we're going to see more of this. We've seen it a lot in the Bitcoin space where people are still in cryptocurrency or wallets or all of that stuff via all sorts of attacks in this chain. I strongly suspect that the Magecart crew that has gone and got like, I think, it was British airways or something is probably playing in this space already.And, a lot of people just don't know that they're just polluting this, this tool chain or the supply chain in ways that people don't fully understand yet. But , what I will say is whether or not it's application security, pentesting, you name it, right. Any form of security out there, it's a predator and prey relationship.And what I mean by that is the predators, the people that are performing criminal activity and are getting out there to get ahead of all of the things that they're hunting will always evolve faster than the prey, right. In order for them to eat, they need to catch something. The things that need to be caught, they're doing other things too to live off the land and so they don't have to be able to run as fast. They don't have to be able to hide better. So I think that what will happen is as more regulation comes out, as companies are held to account for these things, as stocks are impacted as , you know, the whole ecosystem starts to change. Security will be more invested in but I also think that security is going to becomepart of the business in a way that it just hasn't been traditionally. Right. I think even now you still see very siloed security organizations compared to the infrastructure teams, compared to the product teams. But thankfully in, in, in certain pockets, you're starting to see this evolution where security is getting closer to the development organization with DevSecOps. They're getting closer to the product organization, whether it's a website, mobile app, physical device that sits on a, a lab space, they're getting closer to the business.And I often say or I've often referred to this, this whole problem as the hospital model. Right. Traditionally speaking, and I say this with empathy and love for my security peers. Security people have either been ambulance chasers or ambulance drivers. We keep the patient alive and we get them to the hospital and then it's up to somebody else to actually deal with that.But where we actually need to be is, we need to be specialists in the hospital, like the neurosurgeons or the doctors, or, you know, or heck even the nurses, right. Like you, you have this, if you treated a hospital except for, you know, get rid of all the administration and financial aspect and legal aspect for a moment and think about just medicine.If medicine was software development and we were just another practitioner of that. Everyone in a hospital from a nurse to a pediatrician to a surgeon has a general understanding of medicine. I think at some point everyone will need to have a general understanding of security and have professionals in that space that are specialists.But I think in the same way, security professionals will need to have a general understanding of software development because that's how they become part of this ecosystem in an effective way. Otherwise, we continue to be ambulance chasers or ambulance drivers, and all we can do is find the problem, make sure the person gets to the hospital and hope that it gets fixed.AbhayYeah, no, absolutely. I think, yeah, I think that's a very valid point simply because the way , yeah, DevSecOps is definitely driven security teams closer together, if done well right now... if done well is a pretty big caveat but sometimes it's not and people still want to stick to that but, yeah, I get what you're saying.I think, yeah. I think security.... one of the points that you mentioned was very nice, which is essentially having a baseline security knowledge, like something like security and safety, right. People understand safety pretty, pretty well. So security as safety but security as a specialization maybe something that , you know , I think we need to kind of chase even with open source or whatever not this, this was not, of course, the only open source, this was not a closed source stuff as well. I think we're almost out of time, Keith, any additional points that you wanted to talk about related to AppSec or DevSecOps or something that you wanted to put out there?Any projects that you want to mention that you're working on? KeithSo no major projects. The one thing I would recommend for everyone in the security space is go read the DevOps handbook. You know, it's funny. I literally. I have it literally on my desk , because as you can see, like, I refer to it all the time and the nice part is part six.It was all about security. And , and so I tell people, you know, whether they're in Dev, Operations, Security, I say, read the DevOps handbook. It has case studies in there that are still relevant today for a vast majority of companies. It will teach you things about the process that you might not understand , you know, in a, kind of a natural way that they'll give you an appreciation for.And that starts the conversation with your peers in development and operations in really important ways. So I don't... I'm not an author of the book. I get no royalties from it. [Laughter]  I mention it everywhere, because I think it's super important for people to read and enjoy. And then yeah, hit me up on Twitter.I'm @securingdev on Twitter. So drop me a message. I'm always happy to have a conversation. I think, you know, Abhay about these things, so thank you for having me though. This has been awesome and I really appreciate it. AbhayThanks Keith. It was great. I think our listeners are going to have a lot of value added points to take home from this. Thank you very much for being a part of this. KeithCool. Thank you. Cheers!!