docs/dev/zero_length_packet.md

# How ADB uses USB Zero-Length Packets (ZLP)

## TLDR;
There is no USB mechanism that lets a sender announce the size of a `Transfer`. This is not
a problem when the host side receives packet since is leverage the aprotocol to know what size
of transfer to expect. However, traffic towards the device must include zero length packets to
mark the boundaries since the device does not leverage the aprotocol.

## Introduction

There is an asymmetry in how ADB communicates over USB. While all USB backends on the host side (Linux,
Windows, Mac, and libusb) send ZLPs, the device side never does. Why is that? This document explains
what ZLPs are, how ADB uses them, and why things are designed this way.

## USB Transfer 101

In the context of ADB, USB can be thought of as two unidirectional pipes per device.
One pipe takes care of upload while the other takes care of the download. On the pipe
transit payloads. The maximum size of a payload is set by the pipe buffers located
on the device. These buffers are called the `Endpoint`.

```
  ┌──────────┐                         ┌──────────┐
  │ USB Host │                         │USB Device│
  ├──────────┤                         ├──────────┤
  │          │                         │          │
  │  Pipe ◄──┼─────────────────────────┼ Endpoint │
  │          │        USB              │          │
  │  Pipe ───┼─────────────────────────► Endpoint │
  └──────────┘                         └──────────┘
```

In USB parlance, sending a buffer of data on a pipe and receiving it on the other end is called a `Transfer`.
On the sender side, the USB Controller is presented with a [buffer,size] pair called IRP. On the receiver
side, a similar IRP is provided to the USB controller.

```
       ┌────────┐                             ┌──────────┐
       │ Sender │                             │ Receiver │
       └────┬───┘                             └─────┬────┘
            │                                       │
         ┌──▼───┐                                ┌──▼───┐
         │ IRP  │                                │  IRP │
         └──┬───┘                                └──▲───┘
     ┌──────▼───────┐                        ┌──────┴───────┐
     │USB Controller│                        │USB Controller│
     │              │                        │              │
     │             ─┼───►─DP──►─DP ─►──DP──►─┼─►            │
     └──────────────┘                        └──────────────┘
```

Because of the endpoint buffer size (`wMaxPacketSize`), an IRP is broken down in
several data payloads called `Data Packets` (DP).

Note: On the device, ADB uses `functionfs` which is not based on IRP. However, the logic is the same since received DP
must be re-assembled on the device to rebuild the original IRP. To simplify this document we use the name "IRP"
everywhere in this doc to mean "[buffer,size] pair provided to the USB Controller".

## When does a USB Transfer ends?

If an IRP is broken down in DPs by the sender, how does the receiver reassemble
the DPs into an IRP on the other side?

The key concept to get out of this whole document is that there is no mechanism
in USB for the sender to announce the size of a `Transfer`. Instead, the receiving
end uses the following rules. A `Transfer` is considered done when either of the following condition
is met.

- An error occurred (device disconnected, ...).
- The IRP is full.
- The size of the packet is less than `wMaxPacketSize` (this is a Short-Packet). This is
a different behavior from the usual UNIX `read(3)` which is allowed to return less than required
without meaning that the stream is over.
- Too much data is received. The IRP overflows (this is also an error).

See USB 2 specifications (5.8.3 Bulk Transfer Packet Size Constraints) for additional information.
```
An endpoint must always transmit data payloads with a data field less than or equal to the endpoint’s
reported wMaxPacketSize value. When a bulk IRP involves more data than can fit in one maximum-sized
data payload, all data payloads are required to be maximum size except for the last data payload, which will
contain the remaining data. A bulk transfer is complete when the endpoint does one of the following:

• Has transferred exactly the amount of data expected
• Transfers a packet with a payload size less than wMaxPacketSize or transfers a zero-length packet
```

### Example 1: The IRP is full

For a USB3 bulk pipe, the `wMaxPacketSize` is 1024. The sender "S" wishes
to send 2048 bytes. It creates a IRP, fills it with the 2048 bytes, and gives the IRP
to the USB controller. On the
received side "R", the USB controller is provided with a IRP of side 2048.

```
Traffic:
S -> 1024 -> R
S -> 1024 -> R IRP full, Transfer OK!
```

At this point R's IRP is full. R USB controller declares the `Transfer` over
and calls whatever callback the client provided the IRP.

### Example 2: Short-Packet

Same USB3 bulk as Example 1. The `wMaxPacketSize` is 1024. The sender wishes
to send 2148 bytes. It creates a IRP of size 2148 bytes and fills it with data.
On the received side, the USB controller is provided with a IRP of size 4096.

```
Traffic:
S -> 1024 -> R
S -> 1024 -> R
S ->  100 -> R Short-Packet, Transfer OK!
```

The receiver end detects a short packet. Even though it was provided with a 4906
byte IRP, it declares the `Transfer` completed (and records the actual size
of the `Transfer` in the IRP).

### Example 3: Overflow

Same USB3 bulk as Example 1. The `wMaxPacketSize` is 1024. The sender wishes
to send 4096 bytes. It creates a IRP, fills it with the 4096 bytes. On the
receiver side, the USB controller is provided with an IRP of size 2148.

```
Traffic:
S -> 1024 -> R
S -> 1024 -> R
S -> 1024 -> R ERROR, Transfer failed!
```

On the third packet, the receiver runs out of space in the IRP (it only had 100
bytes available). Without a way to fully store this packet,
it discards everything and returns an error stating that the `Transfer` was not successful.

## Preventing overflow and the need for Zero-Length Packets

There are two techniques to avoid overflows.

### Using a protocol
One technique is to create a protocol on top of `Transfers`.
ADB does that with its "aprotocol" ([protocol.md](protocol.md)).

In aprotocol, the sender creates a `Transfer` containing a header which is
always 24 bytes. Then it sends a separate `Transfer` containing the payload.
The size of the payload is in the header. This way the receiver always knows
what size of IRP to provide to the USB controller: it first requests a 24 byte IRP
read, extracts the size of the payload, then issues a second IRP read request
with the extracted size of the payload.

### Using a multiple of `wMaxPacketSize`

The other technique to avoid overflows is for the receiver to always use a IRP with
a size which is a multiple of the `wMaxPacketSize`. This way a `Transfer` always ends properly.
* A max size packet will exactly finish to fill the IRP, ending the `Transfer`.
* A short packet will end the `Transfer`.

This technique comes with an edge case. Take the example of a USB3 pipe where
`wMaxPacketSize` is 1024. The sender wishes to send 3072 byte. It creates a IRP
of that size, fills in the data and gives it to the USB controller which breaks
it into Packets. The receiver decides to read with a IRP of size 4096.

```
Traffic:
S -> 1024 -> R
S -> 1024 -> R
S -> 1024 -> R
.
.
.
Stalled!
```

After the USB controller on the sender side has sent all the data in the IRP, it won't send anything else.
But none of the ending conditions of a `Transfer` have been reached on the receiving end. No overflow, no short-packet, and the IRP is not
full (there is still 1024 bytes unused). As is, the USB controller on the receiving end will never declare the `Transfer`
either successful or failed. This is a stall (at least until another Packet is sent, if ever).

This condition is entered when the size of a IRP to send is a multiple of `wMaxPacketSize`
but less than the size of the IRP provided by the receiving end. To fix this condition,
the sender MUST issue a Zero-Length Packet. Technically, this is a short packet (it is less
than `wMaxPacketSize`). Upon receiving the ZLP, the receiver declares the `Transfer`
finished.

```
Traffic:
S -> 1024 -> R
S -> 1024 -> R
S -> 1024 -> R
S ->    0 -> R Short-Packet, Transfer is over!
```

## Implementation choices

By now, it should be clear that whether a sender needs to send a ZLP depends on the way
the receiver end works.

### ADB Device to Host pipe communication design

The receiver on the host leverages ADB aprotocol ([protocol.md](protocol.md)). It
first creates a IRP of size 24 bytes to receive the header. Then it creates a IRP
`Transfer`
of the size of the payload. Because the IRPs are always exactly the size of the `Transfer`
the device sends, there is no need for LZP. The USB Controller on the host side will always be able
to declare a `Transfer` complete when the IRP is full and there will never be any overflow.

The drawback of this technique is that it can consume a lot of RAM because multiple
IRPs can be in flight at a time. With the maximum size of
a apacket payload being MAX_PAYLOAD (1MiB), things can quickly add up.


### ADB Host to Device pipe communication design

On the device side, the receiver does not leverage the ADB aprotocol ([protocol.md](protocol.md)).
I suspect this was done to reduce memory consumption (the first Android device had a total RAM size of 192MiB).

The UsbFS connection always requests the same
`Transfer` size. To prevent overflows, the size is picked to be a multiple of the `wMaxPacketSize` (1x would be
valid but the overhead would kill performances). Currently, the value is kUsbReadSize (16384). USB endpoints
have a well known `wMaxPacketSize` so 16384 works for all of them (this list is for bulk transfers only which
ADB exclusively uses).

* Full Speed: 8, 16, 32, or 64 bytes.
* High Speed: 512 bytes.
* Super Speed: 1024 bytes.

When the apacket payload size is a multiple
of the `wMaxPacketSize`, the sender on the host side MUST send a ZLP to avoid stalling
on the receiver end.


## What happens if the host sender has a bug and ZLPs are not sent?

If there is a bug on the host and ZLPs are not sent, several things can happen.
You can observe normal behavior, stalled communication, or even device disconnection.

Because there are many buffers before the USB controller layer
is hit, the issue won't be deterministic. However my experience showed that attempting to
push 10GiB  rarely
fails to bring up instabilities.

```
$ dd if=/dev/urandom of=r.bin bs=1G count=10 iflag=fullblock`
$ adb push -Z r.bin /datal/local/tmp
```

### 1. Nothing breaks

You could be unlucky and not trigger the fault.

### 2. Stall

A payload of a size that's a multiple of `wMaxPacketSize` but of size less than kUsbReadSize (16384) is sent.
This is a stall as previously described.

### 3. Disconnection (due to merged packets)

In real-life usage, there is rarely a single thing happening on ADB. Users often also run logcat, Studio
monitors metrics, or perhaps the user has a shell opened. What happens if a connection goes stalls
but then something else sends an apacket?

The first `Transfer` of the apacket will be an apacket header which is 24 bytes. This will be considered
a short-packet. The previous stalled `Transfer` will be completed with the header appended. This will
confuse UsbFS since the payload will be 24 bytes more than it should be. In this condition, the connection
is closed. The log message is

```
received too many bytes while waiting for payload
```

or

```
received packet of unexpected length while reading header
```

A summary inspection of logs may make it look like a payload `Transfer`
was merged with the next header `Transfer`.