External USB fuzzing for Linux kernel

Syzkaller support fuzzing the Linux kernel USB subsystem externally (as it would be done by plugging in a physical USB device with e.g. Facedancer). This allowed to find over 100 bugs in the Linux kernel USB stack so far. This is still in development and things might change.

USB fuzzing consists of 3 parts:

  1. Syzkaller changes that are now upstream.
  2. Kernel interface for USB device emulation, which can be found here.
  3. KCOV changes that allow to collect coverage from background threads and interrupts (the former can be found here, the latter in still in development).

Currently syzkaller defines 5 USB syzcalls (see this and this):

  1. syz_usb_connect - connects a USB device.
  2. syz_usb_disconnect - disconnects a USB device.
  3. syz_usb_control_io - sends or receives a control message over endpoint 0.
  4. syz_usb_ep_write - sends a message to an endpoint.
  5. syz_usb_ep_read - receives a message from an endpoint.

More details can be found:

  1. In the OffensiveCon 2019 “Coverage-Guided USB Fuzzing with Syzkaller” talk (slides, video).
  2. In this email.

A few major things that need to be done:

  1. Collect coverage from interrupts (this is required to enable better fuzzing of USB drivers after enumeration completes).
  2. Add descriptions for all main USB classes.
  3. Upstream KCOV changes.
  4. Upstream the kernel interface for USB device emulation.

Some ideas for things that can be done:

  1. Add a mode for standalone fuzzing of physical USB hosts (by using e.g. Raspberry Pi Zero, see below). This includes at least: a. making sure that current USB emulation implementation works properly on different OSes (there are some differences); b. using USB requests coming from the host as a signal (like coverage) to enable “signal-driven” fuzzing, c. making UDC driver name configurable for syz-execprog and syz-prog2c.
  2. Generate syzkaller programs from usbmon trace that is produced by actual USB devices (this should make the fuzzer to go significantly deeper into the USB drivers code).

Syzkaller descriptions for USB fuzzing can be found here.

Setting up

  1. Checkout the usb-fuzzer branch from https://github.com/google/kasan

  2. Configure the kernel (at the very least CONFIG_USB_FUZZER=y and CONFIG_USB_DUMMY_HCD=y need to be enabled).

    The easiest option is to use the config from the syzbot USB fuzzing instance.

    Another option is to use the USB config generation script. This script allows to extract enabled USB related configs from a set of existing .config files. Right now it extracts configs only from one of the Ubuntu kernel's configs.

    cd ./dashboard/config/
    # Put relevant .configs into ./distros/
  3. Build the kernel.

  4. Optionally update syzkaller descriptions by extracting USB IDs using the instructions below.

  5. Enable syz_usb_connect, syz_usb_disconnect, syz_usb_control_io, syz_usb_ep_write and syz_usb_ep_read syzcalls in the manager config.

  6. Set sandbox to none in the manager config.

  7. Pass dummy_hcd.num=8 to the kernel command line in the maganer config.

  8. Run.

Updating syzkaller USB IDs

Syzkaller uses a list of hardcoded USB IDs that are patched into the syz_usb_connect syzcall by syzkaller runtime. One of the ways to make syzkaller target only particular USB drivers is to alter that list. The instructions below describe a way to generate syzkaller USB IDs for all USB drivers enabled in your .config.

  1. Apply this kernel patch.

  2. Build and boot the kernel.

  3. Connect a USB HID device. In case you're using a CONFIG_USB_FUZZER=y kernel, use the provided keyboard emulation program.

  4. Use syz-usbgen script to update syzkaller descriptions:

    ./bin/syz-usbgen KERNEL_LOG ./sys/linux/init_vusb_ids.go

Running reproducers with Raspberry Pi Zero W

It's possible to run syzkaller USB reproducers by using a Linux board plugged into a physical USB host. These instructions describe how to set this up on a Raspberry Pi Zero W, but any other board that has a working USB UDC driver can be used as well.

  1. Download raspbian-stretch-lite.img from here.

  2. Flash the image into an SD card as described here.

  3. Enable UART as described here.

  4. Boot the board and get a shell over UART as described here. You'll need a USB-UART module for that. The default login credentials are pi and raspberry.

  5. Get the board connected to the internet (plug in a USB Ethernet adapter or follow this).

  6. Update: sudo apt-get update && sudo apt-get dist-upgrade && sudo rpi-update && sudo reboot.

  7. Install useful packages: sudo apt-get install vim git.

  8. Download and install Go:

    curl https://dl.google.com/go/go1.10.8.linux-armv6l.tar.gz -o go1.10.8.linux-armv6l.tar.gz
    tar -xf go1.10.8.linux-armv6l.tar.gz
    mv go goroot-1.10.8
    mkdir gopath-1.10.8
    export GOPATH=~/gopath-1.10.8
    export GOROOT=~/goroot-1.10.8
    export PATH=~/goroot-1.10.8/bin:$PATH
    export PATH=~/gopath-1.10.8/bin:$PATH
  9. Download syzkaller, apply the patch below and build syz-executor:

    diff --git a/executor/common_usb.h b/executor/common_usb.h
    index e342d808..278c2f4e 100644
    --- a/executor/common_usb.h
    +++ b/executor/common_usb.h
    @@ -269,9 +269,7 @@ static volatile long syz_usb_connect(volatile long a0, volatile long a1, volatil
            // TODO: consider creating two dummy_udc's per proc to increace the chance of
            // triggering interaction between multiple USB devices within the same program.
    -       char device[32];
    -       sprintf(&device[0], "dummy_udc.%llu", procid);
    -       rv = usb_fuzzer_init(fd, speed, "dummy_udc", &device[0]);
    +       rv = usb_fuzzer_init(fd, speed, "20980000.usb", "20980000.usb");
            if (rv < 0) {
                    debug("syz_usb_connect: usb_fuzzer_init failed with %d\n", rv);
                    return rv;
    diff --git a/executor/executor.cc b/executor/executor.cc
    index 34949a01..1afcb288 100644
    --- a/executor/executor.cc
    +++ b/executor/executor.cc
    @@ -604,8 +604,8 @@ retry:
                            call_extra_cover = true;
                    if (strncmp(syscalls[call_num].name, "syz_usb_connect", strlen("syz_usb_connect")) == 0) {
    -                       prog_extra_timeout = 2000;
    -                       call_extra_timeout = 2000;
    +                       prog_extra_timeout = 5000;
    +                       call_extra_timeout = 5000;
                    if (strncmp(syscalls[call_num].name, "syz_usb_control_io", strlen("syz_usb_control_io")) == 0)
                            call_extra_timeout = 300;
    go get -u -d github.com/google/syzkaller/...
    cd ~/gopath-1.10.8/src/github.com/google/syzkaller
    # Put the patch above into ./syzkaller.patch
    git apply ./syzkaller.patch
    make executor
    mkdir ~/syz-bin
    cp bin/linux_arm/syz-executor ~/syz-bin/
  10. Build syz-execprog on your host machine for arm32 with make TARGETARCH=arm execprog and copy to ~/syz-bin onto the SD card. You may try building syz-execprog on the Raspberry Pi itself, but that worked poorly for me due to large memory consumption during the compilation process.

  11. Make sure that ou can now execute syzkaller programs:

    cat socket.log
    r0 = socket$inet_tcp(0x2, 0x1, 0x0)
    sudo ./syz-bin/syz-execprog -executor ./syz-bin/syz-executor -threaded=0 -collide=0 -procs=1 -enable='' -debug socket.log
  12. Setup the dwc2 USB gadget driver:

    echo "dtoverlay=dwc2" | sudo tee -a /boot/config.txt
    echo "dwc2" | sudo tee -a /etc/modules
    sudo reboot
  13. Get Linux kernel headers following this.

  14. Download the fuzzer module:

    mkdir module
    cd module
    wget https://raw.githubusercontent.com/google/kasan/usb-fuzzer/drivers/usb/gadget/fuzzer/fuzzer.c
    wget https://raw.githubusercontent.com/google/kasan/usb-fuzzer/include/uapi/linux/usb/fuzzer.h

    Apply the following change:

    diff --git a/fuzzer.c b/fuzzer.c
    index 308c540..68d43b9 100644
    --- a/fuzzer.c
    +++ b/fuzzer.c
    @@ -17,7 +17,7 @@
     #include <linux/usb/gadgetfs.h>
     #include <linux/usb/gadget.h>
    -#include <uapi/linux/usb/fuzzer.h>
    +#include "fuzzer.h"
     #define        DRIVER_DESC "USB fuzzer"
     #define DRIVER_NAME "usb-fuzzer-gadget"

    Add a Makefile:

    obj-m := fuzzer.o
    KDIR := /lib/modules/$(shell uname -r)/build
    PWD := $(shell pwd)
    	$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules

    And build with make.

  15. Insert the module with sudo insmod fuzzer.ko.

  16. Build and test the keyboard emulator program:

    # Connect the board to some USB host.
    wget https://raw.githubusercontent.com/google/syzkaller/master/tools/syz-usbgen/keyboard.c
    # Apply the patch below.
    gcc keyboard.c -o keyboard
    sudo ./keyboard
    # Make sure you see the letter 'x' being entered on the host.
    diff --git a/tools/syz-usbgen/keyboard.c b/tools/syz-usbgen/keyboard.c
    index 2a6015d4..3ebd1e03 100644
    --- a/tools/syz-usbgen/keyboard.c
    +++ b/tools/syz-usbgen/keyboard.c
    @@ -95,8 +95,8 @@ int usb_fuzzer_open() {
     void usb_fuzzer_init(int fd, enum usb_device_speed speed) {
            struct usb_fuzzer_init arg;
            arg.speed = speed;
    -       arg.driver_name = "dummy_udc";
    -       arg.device_name = "dummy_udc.0";
    +       arg.driver_name = "20980000.usb";
    +       arg.device_name = "20980000.usb";
            int rv = ioctl(fd, USB_FUZZER_IOCTL_INIT, &arg);
            if (rv != 0) {
  17. You should now be able to execute syzkaller USB programs:

    $ cat usb.log
    r0 = syz_usb_connect(0x0, 0x24, &(0x7f00000001c0)={{0x12, 0x1, 0x0, 0x8e, 0x32, 0xf7, 0x20, 0xaf0, 0xd257, 0x4e87, 0x0, 0x0, 0x0, 0x1, [{{0x9, 0x2, 0x12, 0x1, 0x0, 0x0, 0x0, 0x0, [{{0x9, 0x4, 0xf, 0x0, 0x0, 0xff, 0xa5, 0x2c}}]}}]}}, 0x0)
    $ sudo ./syz-bin/syz-execprog -executor ./syz-bin/syz-executor -threaded=0 -collide=0 -procs=1 -enable='' -debug usb.log
  18. Steps 19 through 21 are optional. You may use a UART console and a normal USB cable instead of ssh and Zero Stem.

  19. Follow this to setup Wi-Fi hotspot.

  20. Follow this to enable ssh.

  21. Optionally solder Zero Stem onto your Raspberry Pi Zero W.

  22. You can now connect the board to an arbitrary USB port, wait for it to boot, join its Wi-Fi network, ssh onto it, and run arbitrary syzkaller USB programs.