AI in Fuzz Testing

AI is a hot topic within and outside of the workplace. In this episode, Zachariah (Zane) Pelletier, Service Manager for Enterprise and ICS Security Services in the Americas at ETAS is joined by Irina Nicolae, Research Scientist at the Bosch Center for Artificial Intelligence, to talk about the use of AI in fuzz testing. After taking us through an overview of fuzz testing and challenges associated with it, Zane and Irina talk about the use of AI in fuzz testing. 

The two then take us through the history and current status of a project using AI in fuzz testing, including models used and how they were chosen, considerations in programming the language/libraries, upfront work that was required before getting started, the importance of data, fine tuning once the testing was underway, and a peek into their results. 

In the ever-evolving world of automotive and testing technologies, this is a valuable discussion on one of the key trends. 

Thanks for listening!

• Email us at: contact.us@etas.com
• Learn more about ETAS on our website 
• Follow us on LinkedIn: @ETAS

Transcript

00:00:02 Voiceover

Welcome to the Empowering Tomorrow's Automotive Software Podcast, brought to you by ETAS, a single source of cutting-edge software and hardware solutions that make automotive embedded systems safe, smart, secure, and sustainable.

00:00:15 Voiceover

Each episode, we'll be joined by ETAS and industry experts to discuss how electrification, automation, and connectivity are impacting the automotive industry.

00:00:25 Voiceover

Now, sit back and enjoy the discussion.

00:00:33 Zane Pelletier

Hello, everybody, and welcome back to the Empowering Tomorrow's Automotive Software Podcast.

00:00:37 Zane Pelletier

I'm your host, Zane Pelletier, and today we're talking about AI-guided embedded fuzz testing for automotive modules.

00:00:45 Zane Pelletier

Today I'm joined by Irina Nicolae.

00:00:48 Zane Pelletier

She is a research scientist with the Bosch Center for Artificial Intelligence.

00:00:54 Zane Pelletier

And I'm going to pass it off to her to tell you a little bit more about her background.

00:00:58  Irina Nicolae

Hi, Zane.

00:00:59  Irina Nicolae

Hi, everyone.

00:01:00  Irina Nicolae

Thanks for having me today.

00:01:01  Irina Nicolae

Yeah, about my background.

00:01:03  Irina Nicolae

My background is in computer science, and I am a research scientist now with Bosch Research.

00:01:08  Irina Nicolae

My research mainly focuses on AI and cybersecurity, going into both directions.

00:01:14  Irina Nicolae

So kind of how do we solve security issues of AI models and AI deployments, but at the same time, also how do you solve

00:01:23  Irina Nicolae

classic cybersecurity tasks with AI.

00:01:26  Irina Nicolae

And I guess the latter is going to be part of our conversation today as well.

00:01:31 Zane Pelletier

Perfect.

00:01:31 Zane Pelletier

Thank you for joining, Irina.

00:01:33 Zane Pelletier

So jumping right into it, I'd like to introduce the topic a little bit.

00:01:36 Zane Pelletier

So we're going to be talking about embedded fuzz testing, which has been a topic here several times before.

00:01:41 Zane Pelletier

But for anybody who doesn't have a background in this area, essentially what we're doing is we are talking to embedded devices, in this case, mostly automotive controllers.

00:01:53 Zane Pelletier

And what we're doing is we're sending carefully crafted pseudo-randomized input to the protocol interfaces that are on these embedded devices.

00:02:02 Zane Pelletier

And what we're doing is we're trying to monitor for adverse events, whether that is a failure to respond to some known good message, maybe some other program that's running on the device that has some kind of watchdog timer that's getting triggered, or there's some other adverse condition based on

00:02:21 Zane Pelletier

what the application of that device is.

00:02:23 Zane Pelletier

We're trying to monitor it while we're sending these inputs, and we're trying to see if we can trigger some kind of software bug or hardware bug on the device, which could indicate either an issue with functionality, safety, or in many cases, a security issue.

00:02:40 Zane Pelletier

So that's the goal of fuzzing.

00:02:42  Irina Nicolae

So Zane, kind of by now I think of fast testing has quite a little bit of a tradition and has had quite a bit of impact.

00:02:49  Irina Nicolae

when it comes to kind of application testing and PC software, I want to say, with many bugs in open source or closed source programs being found this way.

00:02:59  Irina Nicolae

What's the main difference when you're trying to fast-test embedded devices versus PC software?

00:03:07 Zane Pelletier

Yeah, definitely.

00:03:09 Zane Pelletier

That's, I think, a big topic.

00:03:11 Zane Pelletier

A lot of people ask that initially.

00:03:14 Zane Pelletier

So

00:03:16 Zane Pelletier

I have done fuzz testing of both embedded devices and traditional, more traditional IT software and network devices.

00:03:23 Zane Pelletier

And I would say for embedded controllers, the primary difference is the reliance of the data that the controller is sending on safety functionality.

00:03:35 Zane Pelletier

So for a lot of these controllers, they are embedded within a vehicle and they're responsible for certain actions that the vehicle takes.

00:03:41 Zane Pelletier

For instance, braking functionality.

00:03:43 Zane Pelletier

Sometimes it's

00:03:46 Zane Pelletier

other features that have to do with acceleration of the powertrain.

00:03:50 Zane Pelletier

So a lot of these things, if there is an adverse event with the controller that is responsible for those things on the vehicle, it usually becomes a safety issue.

00:03:59 Zane Pelletier

And the difficulty here, I think, is the general abstract nature of some of these diagnostic requests that are being made on the controller level.

00:04:08 Zane Pelletier

There are a lot of

00:04:11 Zane Pelletier

proprietary protocols and messages that are being sent that are not well-defined within the standards that these devices are using.

00:04:19 Zane Pelletier

So fuzz testing it is especially difficult because we're looking for an even smaller needle in the haystack.

00:04:25 Zane Pelletier

There's a lot less that is known about these devices generally, and they're usually very dependent on these processes to operate.

00:04:35  Irina Nicolae

Got it.

00:04:35  Irina Nicolae

And are there any kind of technical challenges to fuzz testing?

00:04:41  Irina Nicolae

Embedded software.

00:04:42 Zane Pelletier

Yeah, definitely.

00:04:43 Zane Pelletier

So, I would say that every controller is different, and that is a huge problem that we come up with, because every time we start doing a new embedded fuzz test, we're basically starting from level 0 again, so we are building up our understanding of how the communication protocol is working.

00:04:59 Zane Pelletier

And that's essentially, that's very difficult because of the nature of fuzzing.

00:05:04 Zane Pelletier

It's pseudo-random.

00:05:05 Zane Pelletier

So you essentially just have to start the test and wait until it finds something randomly, essentially.

00:05:11 Zane Pelletier

So you're relying on entropy to find the issues for you, which is not a very efficient way of finding things, I would say.

00:05:19 Zane Pelletier

But it's the only way to have comprehensive coverage in some cases where a lot of these

00:05:24 Zane Pelletier

Inputs are not defined by a standard structure, so this idea of finding bugs in the basic software stack, for instance, of a controller module is very difficult, and you have to consider time constraints and also hardware constraints of the physical constraints of the testing.

00:05:43 Zane Pelletier

usually you have to run these on multiple devices.

00:05:45 Zane Pelletier

And sometimes these devices are prototypes or one-offs.

00:05:49 Zane Pelletier

So having lots of hardware available to you to perform this type of testing is also difficult.

00:05:55 Zane Pelletier

So there is a lot of issues there with regards to efficiency overall.

00:05:59  Irina Nicolae

Right, and kind of resource constraints it sounds as well.

00:06:02 Zane Pelletier

Certainly.

00:06:03  Irina Nicolae

You mentioned testing brakes or other modules that are on cars or from cars, right?

00:06:08  Irina Nicolae

And you mentioned safety implications.

00:06:12  Irina Nicolae

What kind of failures can one expect to find in embedded and kind of automotive software?

00:06:19 Zane Pelletier

Definitely.

00:06:19 Zane Pelletier

So that's a great question.

00:06:21 Zane Pelletier

I would say the most common type of failure that we see is some type of denial of service.

00:06:26 Zane Pelletier

So completely taking the diagnostic functionality of the module offline, rendering it unresponsive to other modules in the vehicle is a very, very common one that we see, at least for a small period of time.

00:06:38 Zane Pelletier

So this could take the form of either sending a message that crashes

00:06:42 Zane Pelletier

trashes some piece of the communication stack, or sending a legitimate diagnostic message that isn't supposed to be able to be processed without security authentication on the module.

00:06:56 Zane Pelletier

So here you can see if there are some security flaws in the way that, for instance, diagnostic unlock is implemented.

00:07:04 Zane Pelletier

You can send a message such as ECU reset, which causes the ECU to

00:07:09 Zane Pelletier

Do exactly that to completely reset and reboot through the boot process, and that will render it unresponsive on the network as well, so you kind of have to think about what the module is doing on the vehicle, what it's supposed to do in its normal functionality.

00:07:24 Zane Pelletier

But essentially, you can kind of start to see how this could be an issue, especially on a network where a lot of these modules are supposed to respond in a very short amount of time, sometimes as low as 5 milliseconds to 20 milliseconds.

00:07:36 Zane Pelletier

Having even a second of downtime is a very, very long time on a network like this, especially when you're dealing with automated driving, ADAS, and things like braking systems.

00:07:46  Irina Nicolae

Right, that indeed sounds very dangerous, and I wouldn't want it, you know, happening on my car, right?

00:07:52 Zane Pelletier

Definitely not.

00:07:54  Irina Nicolae

So then, can you tell us a little bit about how you came to the idea of using AI for ECU testing?

00:08:02 Zane Pelletier

Definitely.

00:08:02 Zane Pelletier

Yeah, for that, we're going to have to go a few years back.

00:08:05 Zane Pelletier

I was originally working on the fuzz testing tool that ETAS develops, CycurFUZZ.

00:08:12 Zane Pelletier

we were running into a lot of common issues that we saw on a lot of different modules that we would work with.

00:08:19 Zane Pelletier

So we would see the same issues across the board.

00:08:22 Zane Pelletier

And usually we would find some of these same issues in modules that were in the same vehicle architecture or even supposed to be on the same vehicle.

00:08:30 Zane Pelletier

So what this led us to believe is that

00:08:33 Zane Pelletier

The bugs that we were finding were so common due to their origin, I would say, in a base layer of the software that was being used everywhere throughout the vehicle, so all the different modules, regardless of supplier in some cases.

00:08:48 Zane Pelletier

So the idea that we had initially was to try to map out the issues that we were seeing on a certain vehicle architecture, a certain set of modules, or even a certain supplier,

00:09:01 Zane Pelletier

and try to train some type of model.

00:09:04 Zane Pelletier

We tried various machine learning methods originally to try to train a model that could be aware of common historical issues that we've seen before.

00:09:15 Zane Pelletier

But not only that, also to have knowledge of being able to understand the threat model itself for these types of, in this case, CAN controllers on a CAN vehicle network.

00:09:26 Zane Pelletier

And so the idea here was to build a heuristic that we were able to use to apply to the future

00:09:31 Zane Pelletier

so that the system wasn't actually fully random.

00:09:34 Zane Pelletier

We were still fuzzing pseudo-randomly, but we were guiding it through the process using historical data.

00:09:40  Irina Nicolae

Yeah, I think it's worth also mentioning that pretty much for fuzzing, what one would want in a test case is a test case that follows the protocol closely enough so that it can actually be processed, right, and parsed.

00:09:58  Irina Nicolae

but you don't want all perfect and kind of like usual messages, right?

00:10:02  Irina Nicolae

You do want them to be a little bit faulty or incorrect and just incorrect enough that they can actually explore those border, right, and the cases, right, those age cases in your system and find, right, those failures.

00:10:16 Zane Pelletier

That's an excellent point.

00:10:17 Zane Pelletier

Yeah.

00:10:18 Zane Pelletier

We're looking for, I've heard it explained, like the gray zone.

00:10:21 Zane Pelletier

You have completely white test cases that are 100% valid and good and can be processed by the system normally.

00:10:26 Zane Pelletier

And then the black test cases, those that are not, that are completely wrong, right?

00:10:31 Zane Pelletier

And yeah, exactly as you said, we're looking for those that won't get thrown out immediately when they're being processed.

00:10:37 Zane Pelletier

So

00:10:38 Zane Pelletier

Yeah, that's an excellent point as well.

00:10:39 Zane Pelletier

And so our idea here was to develop a system that could get close.

00:10:42 Zane Pelletier

It could try to mimic some of the same messages it was perhaps already seeing on the network, but modify and manipulate slightly some of the fields, but in a strategic way so that we could start to map out, there was a larger issue, can we refine what messages are causing that and in what circumstances essentially?

00:11:00 Zane Pelletier

So with that, I think that's a really good background on fuzzing in general and why we need to use it for automotive controllers to be able to find issues, especially in a security context.

00:11:10 Zane Pelletier

So I know that we've been working for a while, Irina, on this project, but let's go back to maybe our initial discussions on what the architecture was to look like for such a system.

00:11:21 Zane Pelletier

You know, I know that you have more expertise than I do in artificial intelligence and being able to formalize these systems.

00:11:28 Zane Pelletier

What considerations when I first brought this project to you, did you have, and I know that we went through several different iterations of this trying different things.

00:11:36 Zane Pelletier

I know I started with the general adversarial network and then we started to lean towards other implementations of this.

00:11:43 Zane Pelletier

And then of course in that time also, I would say that the technology and the systems available to us have changed a lot in the last two years.

00:11:52 Zane Pelletier

So yeah, can you walk me through what you were considering when we brought this to you?

00:11:58 Zane Pelletier

Yeah, for sure.

00:11:59  Irina Nicolae

One of the most common ways, I want to say, of using AI for fuzzing is indeed to do the test case generation step that the fuzzer would otherwise do on its own.

00:12:12  Irina Nicolae

And again, under these constraints that you want test cases that are good enough, but not great, so that you can actually reach the depth of your software and the trigger crashes.

00:12:24  Irina Nicolae

But

00:12:25  Irina Nicolae

kind of people have tried to use AI in other steps of the fuzzing process previously, and maybe not all of them, all of these approaches were as promising.

00:12:34  Irina Nicolae

So I think together we've mainly focused on this idea of generating new test cases.

00:12:40  Irina Nicolae

As you mentioned, basically initially you had considered using GANs, right?

00:12:45  Irina Nicolae

So generative adversarial networks, which pretty much do a cat and mouse game, right, between two models, right?

00:12:53  Irina Nicolae

you're training a model that will actually become your test case generator, and you're training a secondary model that is trying to catch the first model, or rather distinguish real test cases from test cases generated by the generator model.

00:13:09  Irina Nicolae

And you kind of know that the generator has won when the second model, the discriminator, is actually not able to distinguish them from real test cases anymore, right?

00:13:20  Irina Nicolae

And then kind of you get this generator that is indeed

00:13:23  Irina Nicolae

in a way capable of speaking, I want to say, the protocol language that you're aiming for, right?

00:13:29  Irina Nicolae

So in this case, CAN protocol.

00:13:32  Irina Nicolae

But what happens with GAN models is that they're sometimes kind of a little bit difficult to train.

00:13:40  Irina Nicolae

And as you mentioned, also with like recent advances in neural network architectures, we've also kind of tried to switch from GANs to the more recent transformers.

00:13:52  Irina Nicolae

Yeah, transformer is just a fancy name for the model architecture that's basically used for models such as ChatGPT and the GPT family in general these days.

00:14:03  Irina Nicolae

For our case, it was the most adapted to use what's called a decoder transformer, which is exactly the family of models used by GPT.

00:14:13  Irina Nicolae

But I think in our case, we're going for a smaller size transformer than what's used in large language models just because

00:14:22  Irina Nicolae

the CAN protocol does not have as much complexity as, you know, human like natural language, right, with.

00:14:30  Irina Nicolae

So I think with that, we were already kind of aiming for the most promising use or one of the most promising usages of AI for fuzzing.

00:14:41  Irina Nicolae

And with the use of transformers, basically that put us in, I think, in a good position to be relatively state-of-the-art,

00:14:49  Irina Nicolae

or actually state-of-the-art in terms of a few things.

00:14:52  Irina Nicolae

Transformers, for example, do not have the same training issues that GANs would have, which means that you are able to kind of train models relatively faster and more reliably.

00:15:04  Irina Nicolae

So kind of pretty much you save also some of your development time.

00:15:09  Irina Nicolae

Now, in this idea that kind of you train a model to speak the protocol that you want it to speak, remember that we're trying to

00:15:17  Irina Nicolae

speak that protocol almost correctly, but not super correctly either.

00:15:22  Irina Nicolae

And in our case, to achieve that, basically, if you train a transformer strictly on clean CANn messages, like the regular communications, then this is what that model will generate in the end.

00:15:37  Irina Nicolae

So then for our case, what we did is that we used data that contains some vulnerabilities, right, from previous findings.

00:15:46  Irina Nicolae

So in that way, basically the model is able to speak can, but just a little bit faulty and is a little bit biased towards types of faults in the messages that have proven to work well in the past to find failures in modules.

00:16:03 Zane Pelletier

Yeah, that's an excellent overview.

00:16:05 Zane Pelletier

I really like the comparison that you're making to ChatGPT, I think, because almost everyone listening to this

00:16:12 Zane Pelletier

most likely has had experience with that.

00:16:14 Zane Pelletier

So that's a really great way of framing it.

00:16:16 Zane Pelletier

It's essentially the ChatGPT of canned models that also has context on vulnerabilities that we've found.

00:16:23 Zane Pelletier

So that's, I think, easy way to think about what we're attempting to achieve here.

00:16:28 Zane Pelletier

I guess my next question for you is, so I know we went through quite a process of trying to pick

00:16:34 Zane Pelletier

I know that you have developed this actually specifically for this application.

00:16:38 Zane Pelletier

So when you went through and you tried to pick, I know there are several libraries available here.

00:16:42 Zane Pelletier

I think in this case, we were using Python for building this out.

00:16:46 Zane Pelletier

Why did you select certain libraries for this?

00:16:48 Zane Pelletier

And maybe could you talk about the selection process there?

00:16:52 Zane Pelletier

I know initially we used TensorFlow and then we moved to PyTorch.

00:16:55 Zane Pelletier

So maybe you could go through some of the considerations that we had to take into account when actually programming the system.

00:17:02  Irina Nicolae

Right.

00:17:02  Irina Nicolae

So I think it's fair to say for a lot of machine learning applications, kind of Python is now, and definitely for machine learning engineers and data scientists, Python is pretty much the language of choice these days, you know, because it's easy to use, but also because a lot of the toolboxes and libraries, the ecosystem for machine learning, it has been developed for Python.

00:17:24  Irina Nicolae

So that's where you'll get the best support.

00:17:26  Irina Nicolae

And at the same time, I want to say TensorFlow and PyTorch are two of the most

00:17:31  Irina Nicolae

commonly used libraries and most popular when you're working with neural networks.

00:17:38  Irina Nicolae

And it's true that kind of initially we were mainly using TensorFlow.

00:17:42  Irina Nicolae

And at some point we proceeded to migrate the entire code base from TensorFlow to PyTorch.

00:17:49  Irina Nicolae

And this is pretty much to account for the fact that I think these days there are indeed, so the community has shifted towards PyTorch.

00:17:58  Irina Nicolae

Personally, I don't necessarily have a preference between the two.

00:18:02  Irina Nicolae

I kind of like them both.

00:18:03  Irina Nicolae

And they're, I want to say, relatively equivalent in terms of what they offer.

00:18:08  Irina Nicolae

But for our case, we were also considering including some external dependencies and additional libraries that existed.

00:18:18  Irina Nicolae

And those were mainly developed for PyTorch, and that was, for us, the reason to switch.

00:18:25  Irina Nicolae

Outside of that,

00:18:27  Irina Nicolae

in order to not have to do any neural network architecture kind of manual selection, because this is quite lengthy and can take quite a little bit of manual effort if you do it by hand.

00:18:40  Irina Nicolae

We've also used a ray tune to do automated hyperparameter tuning.

00:18:45  Irina Nicolae

So now pretty much your neural network architecture will adapt a little bit to the training data set that you have.

00:18:53 Zane Pelletier

That makes a lot of sense.

00:18:54 Zane Pelletier

I do really see, for certain applications, I think using certain libraries is a good idea just for the tools available to you, but that is something that I don't think that I consider very often when I use specific libraries to...

00:19:11 Zane Pelletier

to either build out tooling or use it for one-off applications is how supported is this going to be by the community in the future, especially if it's an open source project?

00:19:22 Zane Pelletier

Are there things that I'm going to have to go back in and manually implement or change if I want to come back and use this?

00:19:28 Zane Pelletier

And how is its ability to, I would say, adapt over time with changes to our understanding of the best ways to implement things?

00:19:36 Zane Pelletier

So that's definitely, I think,

00:19:39 Zane Pelletier

very difficult to predict, but I definitely saw an improvement there when we made that shift to be able to adapt to an account for some of those things.

00:19:51 Zane Pelletier

So at this point, I think we have a good understanding of what it looks like from a technical perspective, how this thing was built.

00:19:59 Zane Pelletier

Maybe now we can go into a little bit of what we tried it on

00:20:03 Zane Pelletier

and how we were able to evaluate if this was actually helpful for us in this kind of context.

00:20:10 Zane Pelletier

So just to give some background on this, we actually initially had to develop a threat model for the CAN network within the vehicle, and also specifically the application layer of that in-vehicle network, which is known as Unified Diagnostic Services, or UDS.

00:20:28 Zane Pelletier

At this point in time, this actually wasn't well established in the literature.

00:20:32 Zane Pelletier

We had gone through several other papers that detailed both fuzz testing, evaluating the security of this diagnostic protocol, and we weren't really able to find anywhere a unified threat model that showed all of the threats to this type of network in a way where it divided out each of the layers of the stack in terms of the OSI model and looking at different

00:20:55 Zane Pelletier

facets of the protocol itself.

00:20:58 Zane Pelletier

So that was actually some of the upfront work I think we had to do that had nothing to do with actually implementing the model itself, right?

00:21:06 Zane Pelletier

And then following that, we actually had to go out.

00:21:09 Zane Pelletier

And I think a lot of people nowadays, especially I think management level people, usually just say, oh, we have to implement AI.

00:21:17 Zane Pelletier

We got to add AI to it.

00:21:19 Zane Pelletier

It's like the new buzzword, essentially, to a lot of times it's synonymous with having efficiency increases and improving the way that we're doing things.

00:21:28 Zane Pelletier

But I think that part of that is not usually taken into account is the fact that all of these systems rely on a lot of really good quality data that is pre-labeled by people that are collecting it and categorizing it and looking at what the data actually means prior to even implementing and training some of these models.

00:21:49 Zane Pelletier

So I think that process in and of itself took a very long time to achieve for us at least, because we had to go through, we had to do many, many fuzz tests, and then we had to go in and look at what was wrong.

00:22:01 Zane Pelletier

We had to categorize it, we had to label it, and we had to do this for a ton of different components.

00:22:07 Zane Pelletier

I think at this point, we've actually collected a total of almost 50 million test cases that have been executed that we had to go look at.

00:22:14 Zane Pelletier

And out of that, there are hundreds of thousands of failures that humans had to go and look at and

00:22:19 Zane Pelletier

Figure out what was going wrong there, so that was definitely a very grueling part of the process, let's say.

00:22:28 Zane Pelletier

So, our idea here was to create a testing setup, and once we had that testing set up, we wanted to prove this out.

00:22:36 Zane Pelletier

So essentially what we did was we went and we found some modules.

00:22:40 Zane Pelletier

These modules we actually acquired usually for researchers in the automotive realm.

00:22:46 Zane Pelletier

Your options are either the company approaches you and asks you to take a look at their modules or their vehicle.

00:22:52 Zane Pelletier

You can go to a junkyard and you can start pulling them out of vehicles that have been wrecked.

00:22:57 Zane Pelletier

Or you can go on eBay and you can try to find modules off of eBay for people that are parting out, again, vehicles that are wrecked.

00:23:05 Zane Pelletier

So that's exactly what we did.

00:23:06 Zane Pelletier

We went on eBay and we found several modules from the same type of vehicle.

00:23:12 Zane Pelletier

It was actually the same vehicle trim, same model year.

00:23:14 Zane Pelletier

And we hooked them all up into kind of a skeleton of the automotive network.

00:23:20 Zane Pelletier

And then we went ahead and we did traditional fuzzing on all of them to try to figure out

00:23:25 Zane Pelletier

what vulnerabilities existed on the system that we could find.

00:23:30 Zane Pelletier

And luckily, we chose modules that did have vulnerabilities, and it turned out that a lot of them had the same types of issues that we were seeing over and over again.

00:23:40 Zane Pelletier

So I think we had our, we kind of lucked out there.

00:23:43 Zane Pelletier

To be honest, it could have not been that way.

00:23:46 Zane Pelletier

But we were able to acquire modules that did have issues and that had very consistent issues at that.

00:23:52 Zane Pelletier

So I think you can attest to this, Arena.

00:23:54 Zane Pelletier

It was a very long process of us going through and collecting data and then labeling it and getting it into place.

00:24:03 Zane Pelletier

And even after we had everything and we had our system set up, we did have to go back in and tune things a little bit because the output, I think, that we were getting was not quite what we wanted.

00:24:15 Zane Pelletier

And I think it was a little bit, in some cases, it was either repetitive or

00:24:22 Zane Pelletier

the input was not riding that fine line, that edge case that we were looking for.

00:24:27 Zane Pelletier

So maybe, Irina, could you tell us a little bit about, I know, hyperparameter tuning and some of the other terminology you used is kind of heavy on the jargon a little bit.

00:24:36 Zane Pelletier

Maybe you could demystify that a little bit for listeners who've never done that before or who are not working at a very deep level with some of these applications all the time.

00:24:48  Irina Nicolae

Sure, but maybe first I want to answer kind of your point on the data collection, right, and the effort that goes into that, right?

00:24:56  Irina Nicolae

So I think with now AI being relatively, you know, more commonplace used in the past decade or so, you know, it's a little bit like the running gag of AI practitioners that, you know, you spend 80% or more of your time doing data collection and cleaning and pre-processing and all of that.

00:25:14  Irina Nicolae

And even with what we know, right, the experience we have collected

00:25:18  Irina Nicolae

right, as humanity over the past decade, I think the effort that usually goes into the data collection is still very much underestimated and often enough kind of underestimated by companies or maybe management teams, right?

00:25:32  Irina Nicolae

And we've seen that in this case as well, pretty much kind of hands on.

00:25:36  Irina Nicolae

But definitely for everything that is AI, kind of the quality of the data and the effort that you put here in the data collection is actually what will pay off because

00:25:47  Irina Nicolae

You know, the amount of modeling or machine learning that you can do can overcompensate for poor data quality, right?

00:25:56  Irina Nicolae

And in our case, it also served as a guidance, right?

00:26:02  Irina Nicolae

Because basically the threat model that you developed pretty much tells us what we should be focusing on at the modeling steps.

00:26:10  Irina Nicolae

So like what it basically defines the requirements of what the model should be able to

00:26:16  Irina Nicolae

to create those test cases and what is the expectation.

00:26:20  Irina Nicolae

But on your other point regarding hyperparameter tuning, what happens is that when you're training a machine learning model, training that basically means finding the right values for certain parameters in an automated way based on your data.

00:26:38  Irina Nicolae

It's pretty much really just fitting mathematical functions to match your data.

00:26:44  Irina Nicolae

But those on top of those parameters that you're kind of fitting automatically, so that's the basically the training part, your model usually and the entire training process has additional parameters that you're supposed to pretty much kind of set by hand.

00:27:01  Irina Nicolae

So these can mean, maybe the size of your neural network model.

00:27:06  Irina Nicolae

This can pretty much tell you, I don't know, how long you're training and things like that.

00:27:10  Irina Nicolae

And those are things that are not always

00:27:14  Irina Nicolae

So basically that you cannot do automatically based on the data per se.

00:27:18  Irina Nicolae

And these are the ones that are called not the parameters of the models, but the hyper parameters.

00:27:23  Irina Nicolae

So they're kind of like the next level of abstraction.

00:27:27  Irina Nicolae

And historically speaking, you know, maybe kind of like 2 decades ago, people were using a lot of time to tweak these models, these parameters by hand.

00:27:36  Irina Nicolae

But luckily by now we kind of have and the knowledge and the tooling to do so automatically.

00:27:43  Irina Nicolae

And basically this is what we used in this case, because I want to say there's no kind of virtue or anything like that to tuning these parameters by hand.

00:27:57  Irina Nicolae

And being all of us in an automation field, I think whatever can be automated should be.

00:28:03  Irina Nicolae

And it pretty much adds to the effort gains, right, that we're hoping to, and the efficiency gains that we want to actually get right from these, back from these models, right?

00:28:13 Zane Pelletier

Definitely.

00:28:13 Zane Pelletier

Yeah.

00:28:14 Zane Pelletier

Well, thank you for that.

00:28:15 Zane Pelletier

I think that's very helpful.

00:28:16 Zane Pelletier

And I, at the end of the day, it always strikes me, I think people put a lot of weight on.

00:28:24 Zane Pelletier

how far this has advanced.

00:28:26 Zane Pelletier

And I think from an external perspective, it kind of looks like a magic almost, right?

00:28:32 Zane Pelletier

But in the end, it's all just math.

00:28:34 Zane Pelletier

In the end, you're all just trying to do some of these same statistical tests of what is most probable to come next in a sequence of things.

00:28:43 Zane Pelletier

And that's ultimately what we're trying to get to here.

00:28:46 Zane Pelletier

So it always helps to kind of take a step back and look at, with all of these terms for things, what does this actually mean?

00:28:55 Zane Pelletier

And why are we trying to do these things?

00:28:58 Zane Pelletier

So I don't want to dive too deep into our results from this because I do think they were promising, right?

00:29:06 Zane Pelletier

I think that after all of this time, us working on this project, it was definitely worth it.

00:29:11 Zane Pelletier

I will say that we actually did

00:29:13 Zane Pelletier

reserve one of the modules that we looked at.

00:29:15 Zane Pelletier

And what we did is we didn't fuzz test it at all.

00:29:18 Zane Pelletier

Actually, we just took the data from the other four that we had set up.

00:29:21 Zane Pelletier

And we went ahead and we trained the model with that data.

00:29:25 Zane Pelletier

We gave it the information from those vulnerabilities that we were seeing.

00:29:29 Zane Pelletier

And we went ahead and did some of that hyperparameter tuning on that model to try to get it to be the best shape it could be.

00:29:37 Zane Pelletier

And when we went ahead and tested it against the 5th module,

00:29:41 Zane Pelletier

Essentially, high level, what we found is that you could find all of the same things that would take you weeks on the modules before, and this is weeks of trying randomized input to try to find these things.

00:29:55 Zane Pelletier

You could get essentially the same coverage from a model that already had knowledge of what issues were historically on the vehicle in a matter of minutes instead of weeks.

00:30:04 Zane Pelletier

So it was almost 100 times increase in the efficiency of finding these things.

00:30:10 Zane Pelletier

And I think that's by virtue of the fact that we had trained this model to know a lot about CAN, and not only know a lot about CAN, but know a lot about the issues that were present on this specific vehicle.

00:30:23 Zane Pelletier

And I know there's also something that we're working on right now.

00:30:26 Zane Pelletier

Irina, I don't know if you want to go into your application of, I always just refer to it as model patching, but we do have kind of an issue here where since these modules are so unique and so proprietary, there are issues that we will most likely find on certain modules that we will have found nowhere else, right?

00:30:45 Zane Pelletier

So what happens when we want to take this base model that we've created that is generically able to find things on the network and apply it to one specific

00:30:53 Zane Pelletier

Could you talk about kind of how that works and what goes into implementing something like that?

00:31:00  Irina Nicolae

Sure.

00:31:00  Irina Nicolae

So what we are planning to do next is what's known also under the name of fine-tuning.

00:31:08  Irina Nicolae

So let's consider that our base model that we trained so far with kind of existing test cases from previous logs and test runs, basically this is what we would call a pre-trained model.

00:31:21  Irina Nicolae

know, kind of a pre-trained model is, you can find these and download them also from the internet when it comes to, for example, natural language.

00:31:28  Irina Nicolae

So it's really just a model that is trained quite generically.

00:31:33  Irina Nicolae

Again, this is what happens with a lot of large language models.

00:31:36  Irina Nicolae

So, you know, for example, even ChatGPT is trained to chat to humans, right?

00:31:41  Irina Nicolae

But it's not necessarily trained for a very specific task, like, I don't know, maybe crafting a workout plan for you or something like that, right?

00:31:50  Irina Nicolae

But then even

00:31:51  Irina Nicolae

If you want to take such a generic model and then adapt it to a very specific task, that's something, well, that's called fine-tuning, and that it can be done provided that you have a good pre-trained, like a good base model.

00:32:05  Irina Nicolae

It's something that can be done, the specialization, with very little data.

00:32:10  Irina Nicolae

So there's a lot to gain in terms of efficiency if you have a great base model that afterwards you're trying to specialize to a given task.

00:32:20  Irina Nicolae

So here the idea was to use the general CAN model as a pre-training and then to kind of adapt it either to different types of vulnerabilities or potentially, as you mentioned, to different types of providers or different types of modules, right?

00:32:37  Irina Nicolae

So naming this pretty much as seeing this as a specialization.

00:32:41  Irina Nicolae

And this, again, is a relatively standard setup for many tasks, so having pre-training and fine-tuning those two separate steps.

00:32:50  Irina Nicolae

But here we tried to, or we want to do this kind of more efficiently.

00:32:55  Irina Nicolae

And for that, we would be using some pre-existing methods.

00:32:58  Irina Nicolae

One of them is called low-rank adaptation, and short for that is LoRA.

00:33:04  Irina Nicolae

And there is also like a newer version of that that's called DoRA.

00:33:08  Irina Nicolae

So pretty much what they do in normal fine-tuning, you would go and take the base model and actually continue training parts of it

00:33:17  Irina Nicolae

to specialize it to the task that you want.

00:33:20  Irina Nicolae

But with LoRA and DoRA, they found ways, also it's back to math, you know, of doing those adaptations in a way more efficient way without kind of changing the base model is really you have more some shortcuts or like some appendices to your base model that represent your specialization.

00:33:39  Irina Nicolae

So then this is something that can be done quite efficiently from a training perspective.

00:33:45  Irina Nicolae

This is something that can be also done with, as I mentioned, with a very small amount of data.

00:33:50  Irina Nicolae

So that means that, for example, if you wanted to specialize to a specific type of vulnerability, you would not need a lot of data, a lot of failure examples from that vulnerability, and the adaptation would still work.

00:34:03  Irina Nicolae

And this is also something that can then be kind of stored efficiently because you would always have your base model that's not modified, and then you can load on top of it

00:34:14  Irina Nicolae

different adaptations or like appendices, right, that allow it to behave in a certain way for fuzzing specific vulnerabilities or devices.

00:34:24 Zane Pelletier

Awesome.

00:34:25 Zane Pelletier

Thank you for that explanation.

00:34:26 Zane Pelletier

Yeah.

00:34:27 Zane Pelletier

Again, I think another moment to kind of demystify a lot of the, you know, processes that I don't think I personally understood in the past and I think a lot of other people maybe struggle with.

00:34:37 Zane Pelletier

But yeah, that's definitely, I think, very necessary, especially with this type of application.

00:34:42 Zane Pelletier

So I know that we're hoping to continue researching this topic.

00:34:45 Zane Pelletier

I know that we're hoping to do some future experiments.

00:34:47 Zane Pelletier

Hopefully, we're able to get accepted into some kind of academic conference and talk about this more.

00:34:54 Zane Pelletier

And hopefully, this also gets adopted into tooling in the future.

00:34:58 Zane Pelletier

But yeah, Irina, I really want to thank you for joining today and really talking about this topic with me.

00:35:03 Zane Pelletier

I know that we were able to go through a lot of things.

00:35:07 Zane Pelletier

There's a lot to cover here, and I really thank you for bringing in your insight here to talk about how we can use artificial intelligence systems to help us perform some of these types of security analysis on devices that are historically so tricky to fully evaluate.

00:35:25 Zane Pelletier

Do you have any closing thoughts about this topic before we wrap up?

00:35:29  Irina Nicolae

So from my side, kind of the work we did together makes me really happy because I think we managed to find a great

00:35:36  Irina Nicolae

use case for AI in fuzzing, where I think kind of we've seen more, you know, kind of state-of-the-art fuzzers are actually very difficult to beat with or without AI.

00:35:47  Irina Nicolae

But I think kind of with this embedded setup, we managed to find like a setup that's actually quite difficult and where AI can really help move the needle.

00:35:57  Irina Nicolae

And as you mentioned, I'm actually really happy with the increase in efficiency, right, that you mentioned, right, from bringing testing time down from weeks to

00:36:06  Irina Nicolae

basically hours, I think that's something that can have real-world impact, right, for the security engineers and testing engineers who need to do these processes usually.

00:36:17  Irina Nicolae

So I'm really hoping that, you know, we're going to try to publish as much of our approach as possible, and I'm really hoping that this will be, you know, adopted and more people can benefit from it.

00:36:30  Irina Nicolae

More generally, I am looking forward to seeing how AI can shape the

00:36:35  Irina Nicolae

the future of security and safety maybe in the automotive domain.

00:36:42 Zane Pelletier

Certainly, yeah.

00:36:43 Zane Pelletier

And I know that both myself and my team are probably going to be utilizing this quite a bit.

00:36:47 Zane Pelletier

And I hope that others are able to get a lot of insight from this and hopefully be able to apply some of these same things to other domains.

00:36:54 Zane Pelletier

Well, thank you all so much for tuning in to this episode of Empowering Tomorrow's Automotive Software Podcast.

00:37:00 Zane Pelletier

We hope that you found our discussion insightful and valuable as well.

00:37:05 Zane Pelletier

If you enjoyed today's episode, don't forget to subscribe on Spotify, Apple Music, or wherever you get your podcasts.

00:37:11 Zane Pelletier

Feel free to share the episode with your network and leave us a review.

00:37:14 Zane Pelletier

We'd love to hear your feedback.

00:37:18 Voiceover

Thank you for joining this episode of the Empowering Tomorrow's Automotive Software podcast.

00:37:23 Voiceover

Please leave a comment or review with your feedback or what you'd like to hear in future episodes.

00:37:28 Voiceover

To learn more about automotive embedded systems and ETAS's capabilities,

00:37:32 Voiceover

Visit our website at ETAS.com.

00:37:34 Voiceover

That's ETAS.com.