From Key Press to Pixel - The Complete Trace

21.0 What this chapter gives you#

1. You will be able to follow one key press, the letter A, from your fingertip to the light entering your eye, without a single gap in the story.
2. You will be able to name every piece of hardware and every piece of software that touches that press, in order, and say what each one does to it.
3. You will be able to explain why a switch that closes once produces several electrical changes, and what the machine does about that.
4. You will be able to read a raw USB keyboard report by hand and say which keys are down.
5. You will be able to explain why the keyboard does not know what letter you pressed, and where the letter is actually decided.
6. You will be able to describe what a window system does with an event and why the same key goes to a different program depending on where you clicked.
7. You will be able to give a real time budget for the whole chain, stage by stage, and say which stage is costing you the most.
8. You will be able to do the same trace for a mouse click and a touchscreen tap and say exactly what is different.
9. You will be able to argue honestly about input lag, with numbers, instead of with feelings.

21.1 The finger and the switch#

PLAIN21.1.1 in simple words#

1. Under every key on your keyboard there is a switch.
2. A switch is a thing that joins two pieces of metal when you push it, and separates them when you let go.
3. When the metal pieces touch, electricity can flow through. When they are apart, it cannot.
4. That is the whole job of the key. Nothing more. It is a gate for electricity.
5. The key does not know it is the letter A. It has no idea. It only knows “touching” or “not touching”.
6. Different keyboards use different ways of making that touch happen, and they feel very different under the finger.
7. The cheapest way uses a small rubber dome that squashes flat. The most expensive ways use a spring and a sliding plastic part, or a magnet, or a beam of light.
8. All of them are trying to solve the same problem: turn a finger moving a couple of millimetres into a clean electrical signal.

PLAIN21.1.2 a picture in your head#

1. Think of a garden gate with a spring on it.
2. You push the gate, it swings open, and when you let go the spring pulls it shut again.
3. Now imagine the gate has a metal latch that touches a metal post when the gate reaches a certain point.
4. You do not have to push the gate all the way to the wall for the latch to touch. It touches partway through the swing.
5. That partway point is what engineers call the actuation point: the exact moment the key counts as pressed.
6. Now imagine the gate is light and springy. When the latch first hits the post it does not stop dead. It taps, bounces off a fraction of a millimetre, taps again, and only then settles.
7. Anyone watching only the latch would see it touch, break, touch, break, touch, and finally stay touching. All from one push of the gate.
8. That tapping is exactly what a key switch does, and it is the first real problem in this whole chapter.

Where this comparison breaks: a real gate bounces over tens of milliseconds and you can see it. A key switch bounces over a handful of milliseconds and the movements are microscopic. Also, a gate has one latch, while some switch types have no moving contact at all: an optical switch just blocks a light beam, and a Hall-effect switch just moves a magnet past a sensor. Those types cannot bounce mechanically, because there is nothing to bounce.

PLAIN21.1.3 a worked example#

1. Take a Cherry MX Red switch, one of the most common mechanical switches sold.
2. Its published numbers, from the Cherry datasheet, are these.

3. So the key sits at rest. You start pushing.
4. After 2.0 mm of downward movement, the sliding plastic part has let the metal contact leaf spring back into contact. The circuit closes.
5. You are still moving. There are 2.0 mm left before the key hits the bottom. Those 2.0 mm change nothing electrically.
6. This is important. The key fires at 2.0 mm, not at the bottom. If you type by slamming keys to the bottom, you are wasting 2.0 mm of your own time.
7. At the moment of contact, the metal leaf vibrates. For under 5 milliseconds the connection flickers on and off.
8. The keyboard’s own chip therefore sees something like this over those few milliseconds.

time (ms):  0.0  0.4  0.7  1.1  1.5  2.0  2.4  3.0 ... 5.0
contact:    off  ON   off  ON   off  ON   ON   ON      ON

9. If the keyboard reported every one of those changes, you would get “AAAA” from one press.
10. So the firmware waits. It only accepts the change once the signal has held steady for a set period. That waiting is called debouncing.

PLAIN21.1.4 what is really happening inside#

1. Let us go through the four main switch families, because they behave differently and cost you different amounts of time.
2. Rubber dome / membrane. Three flexible plastic sheets are stacked. The top and bottom sheets have printed conductive traces. The middle sheet has holes.
3. A moulded rubber dome sits over each key. Pressing collapses the dome, which pushes the top sheet through the hole in the middle sheet, so the top trace touches the bottom trace.
4. The collapse is sudden, which is why a cheap keyboard has a soft top and then a sharp give. Actuation and bottoming out are almost the same moment, typically at 3.5 to 4 mm.
5. Mechanical. Each key has its own sealed plastic housing with a spring, a sliding stem, and a springy metal contact leaf.
6. The stem has a ramp cut into it. As the stem descends, the ramp lets the leaf spring back into contact with a fixed metal terminal.
7. Because the contact is a springy metal blade hitting a metal terminal, it bounces. Cherry specifies under 5 ms of bounce.
8. Scissor. Used on laptops and thin keyboards. A rubber dome still does the electrical work, but two plastic arms crossed like scissors hold the keycap level.
9. That means the key works even if you press a corner, and it allows travel of roughly 1 to 2 mm instead of 4 mm. Apple’s butterfly design of 2015 pushed this to about 0.5 mm and was abandoned by 2021 because of reliability complaints.
10. Optical. The stem carries a small flag. At rest the flag blocks an infrared beam between an emitter and a photo-transistor. Pressing lifts or lowers the flag and the beam changes.
11. There is no metal-to-metal contact, so there is no contact bounce. The signal still has to cross a threshold, and comparators near their threshold can chatter, so firmware still applies a small filter.
12. Hall effect / magnetic. A magnet in the stem moves past a sensor that measures magnetic field strength. The output is not on or off. It is a number that says how far down the key is.
13. That means the actuation point is a software setting. Wooting’s Lekker switch is specified at 40 gf actuation, 60 gf bottom out, 4.0 mm total travel, and an actuation point adjustable anywhere from 0.1 mm to 4.0 mm.
14. It also allows “rapid trigger”: the key releases the moment you start moving up, rather than waiting to cross a fixed point. That is a firmware behaviour, not a physical one.

TECHNICAL21.1.5 the engineer’s version#

1. Contact bounce is a mechanical resonance of the moving contact against the fixed contact, with the contact modelled as a damped mass-spring system. Bounce duration scales with contact mass and inversely with damping.
2. Cherry’s MX1A-Lxx datasheet, which covers MX Red, specifies bounce time under 5 ms measured during actuation at a closing speed of 0.4 m/s. The MX2A revision keeps the same 45 cN, 2.0 mm, 4.0 mm figures.
3. Typical debounce strategies in keyboard firmware:

4. QMK, the widely used open-source keyboard firmware, ships DEBOUNCE defaulting to 5 milliseconds, with selectable algorithms named sym_defer_g, sym_eager_pk, asym_eager_defer_pk and others. The eager variants report the press immediately and then lock the key out, giving zero added press latency at the cost of possible false presses on a dirty contact.
5. Actuation force is quoted in centinewtons (cN) by Cherry and in grams-force (gf) by most others. 1 cN is 0.01 N; 1 gf is 0.00980665 N. So 45 cN is about 45.9 gf. Treating cN and gf as interchangeable introduces about 2 percent error, which is inside the manufacturing tolerance anyway.
6. Rated lifetime: Cherry MX2A claims 100 million actuations per key. Rubber domes are typically rated 5 to 20 million. Buckling spring keyboards, the IBM Model M design of 1985, are commonly rated at 25 million and many are still working after forty years.
7. IBM patented the buckling spring mechanism in 1978. In it, a coil spring under compression buckles sideways, and the buckling motion flips a hammer that strikes a capacitive sense pad on the PCB. Actuation and the audible click are the same event, which is why the feedback is honest, unlike a clicky mechanical switch where the click and the electrical actuation are produced by different parts and can drift apart.
8. Optical switches used in production include Razer’s opto-mechanical design (2018) and Bloody’s LK Libra series. Vendors claim 0.2 ms actuation latency. That is a marketing claim about the sensor alone and does not include scan, debounce filtering, or USB, which dominate.
9. Hall-effect switches use a linear Hall sensor such as the Allegro A1304 or equivalent, sampled by the microcontroller’s ADC. Resolution of the reported depth is limited by ADC bits and calibration drift, not by the magnet.
10. Tools: an oscilloscope across the switch terminals shows the bounce directly. On a keyboard you control, a logic analyser on the matrix pins shows the scan and the debounce window together.

WORDS21.1.6 remember these#

1. Switch — the thing under a key that joins two wires — an electromechanical or non-contact single-pole momentary contact.
2. Actuation point — how far down the key must go before it counts — pre-travel distance at which the contact closes, typically 2.0 mm on Cherry MX.
3. Travel — total distance the key can move — total actuator travel, typically 4.0 mm full size, 1 to 2 mm scissor.
4. Actuation force — how hard you must press — operating force in cN or gf, typically 45 to 60.
5. Bounce — the flickering of the contact when it first touches — contact chatter caused by mechanical resonance, under 5 ms on Cherry MX.
6. Debounce — waiting until the flickering stops — firmware filtering of contact chatter, typically a 5 ms stability window.
7. Hall effect — sensing a magnet instead of touching metal — a linear magnetic field sensor giving continuous key depth rather than a binary state.

21.2 The key matrix#

PLAIN21.2.1 in simple words#

1. A full-size keyboard has 104 keys. It does not have 104 wires going to the chip inside.
2. That would need 104 pins on the chip plus a common return, and cheap chips do not have that many pins.
3. Instead the keys are arranged in a grid, like squares on graph paper.
4. There are wires running across, called rows, and wires running down, called columns.
5. Each key sits where one row crosses one column, and when pressed it joins that row to that column.
6. With 8 rows and 18 columns you get 144 crossing points, enough for 104 keys, using only 26 wires.
7. The chip cannot look at all crossings at once. So it checks them in turn, very fast, over and over.
8. It energizes one row, reads all the columns, then moves to the next row. Doing all rows once is called a scan.
9. A good keyboard does a complete scan a thousand times a second. A cheap one might do it a hundred times a second.

PLAIN21.2.2 a picture in your head#

1. Think of a block of flats with a caretaker who has to know which flats have someone at home.
2. There are 8 floors and 18 flats on each floor. The caretaker cannot see all 144 flats at once.
3. So the caretaker switches on the light in the corridor of floor 1 only, then walks along and looks at which doorways show light spilling out.
4. Then floor 1’s light goes off, floor 2’s comes on, and the same check happens. And so on to floor 8.
5. One full trip through all 8 floors is one scan. If the caretaker can do it a thousand times a second, nothing gets missed for more than a millisecond.
6. Now the problem. Suppose three flats are occupied and they happen to form three corners of a rectangle. Light can leak from floor to floor through the open doors and make the fourth corner look occupied too.
7. That false fourth key is called ghosting, and it is a real thing that happens on cheap keyboards.
8. The fix is a one-way door on every flat: light can go out but not back in. In electronics, a one-way door is a diode.

Where this comparison breaks: the caretaker is slow and the keyboard is not. A scan of the whole matrix takes tens of microseconds of actual work; the rest of the millisecond is the chip waiting on purpose. And the “light leaking” is not light, it is a current finding a path backwards through two other closed switches.

PLAIN21.2.3 a worked example#

1. Here is a tiny 3x3 matrix. Rows R0 to R2 run across. Columns C0 to C2 run down. A key is at every crossing.

        C0     C1     C2
       |      |      |
R0 ----+------+------+----
       |      |      |
R1 ----+--Q---+--W---+----      Q at R1/C0, W at R1/C1
       |      |      |
R2 ----+--A---+--S---+----      A at R2/C0, S at R2/C1
       |      |      |

2. The chip drives R0 low and reads C0, C1, C2. All read high, so no key on row 0 is down.
3. The chip drives R1 low. C0 reads low, so Q is down. C1 and C2 read high.
4. The chip drives R2 low. C0 reads low, so A is down.
5. Now hold three keys at once: Q (R1/C0), W (R1/C1) and A (R2/C0).
6. Drive R2 low. Current goes out of R2, through A, up C0, back through Q into R1, then out through W into C1.
7. So C1 reads low as well, and the chip believes S (R2/C1) is pressed. It was not. That is a ghost key.
8. Put a diode in series with every switch, all facing the same way. Now current can go from row to column but never from column back to row.
9. The path through Q backwards is blocked. No ghost. Every combination of keys is now readable. That property is called N-key rollover.
10. Some cheap keyboards do not add diodes. Instead the firmware detects the rectangle pattern and refuses to report anything, which is called masking or blocking. You press three keys and get two. That is safer than a ghost and cheaper than 104 diodes.

PLAIN21.2.4 what is really happening inside#

1. The chip’s row pins are outputs. The column pins are inputs with pull-up resistors, meaning each column reads high unless something pulls it down.
2. To scan a row, the chip sets that one row pin low and leaves all the others floating or high.
3. It then waits a short settle time, typically a few microseconds, because the wires have capacitance and the voltage does not change instantly.
4. It reads all the column pins in one operation, usually a single register read, giving 18 bits at once.
5. It stores those bits in a row of a bitmap in memory. After 8 rows it has the full state of all 144 crossings.
6. It compares that bitmap with the previous one. Any bit that changed is a candidate key event.
7. Candidates go into the debounce filter. Only after a candidate has held its new value for the debounce window does it become a real event.
8. Real events go into a queue for the next stage, which is turning a matrix position into a code.
9. The scan rate is a choice. Scanning faster costs power and leaves less CPU time for lighting effects and macros. Scanning slower adds latency.
10. On average, waiting for the scan costs you half a scan period, because your press lands at a random point in the cycle. At 1000 Hz that average is 0.5 ms. At 100 Hz it is 5 ms.

TECHNICAL21.2.5 the engineer’s version#

1. A typical full-size layout uses an 8 x 18 or 6 x 21 matrix. Pin count is rows plus columns, so 26 or 27 general-purpose I/O pins.
2. Diodes are small-signal types, commonly 1N4148 in through-hole builds or BAV70 / 1N4148W in surface mount, chosen for low forward voltage (about 0.7 V) and fast recovery.
3. Orientation is a convention, and both exist. COL2ROW means the diode cathode faces the row; ROW2COL is the reverse. QMK has a compile-time DIODE_DIRECTION setting for exactly this reason.
4. Scan rates in real firmware:

5. Dan Luu’s 2017 analysis computed the Apple IIe’s General Instrument AY-3600 keyboard encoder at a 1.8 ms scan delay and a 6.8 ms debounce delay, under 8.6 ms of internal keyboard logic, using the RC values on the board.
6. N-key rollover over USB requires more than diodes. The 8-byte boot report carries only six key codes. Full NKRO needs a report-protocol descriptor with a bitmap of usages, and most NKRO keyboards expose two HID interfaces: a boot-compatible one for BIOS use and an NKRO one for the operating system.
7. Ghosting terminology: strictly, ghosting is a phantom key appearing; masking or jamming is a real key being suppressed. Consumer marketing uses “anti-ghosting” for both, which is imprecise.
8. Matrix settle time matters at high scan rates. With long traces and no series resistors, a 1 microsecond settle can produce false reads; QMK exposes DEBOUNCE, MATRIX_IO_DELAY (default 30 microseconds) and DIRECT_PINS for keyboards small enough to skip the matrix entirely.
9. Hall-effect boards do not have a conventional matrix scan of digital states. They multiplex analogue sensors into an ADC. The limiting rate is ADC conversion time times the number of keys, which is why 8 kHz analogue boards are recent and expensive.
10. Tools: sudo evtest, wev on Wayland or xev on X11 will show whether keys are being lost. A logic analyser on the row pins shows the scan period directly.

WORDS21.2.6 remember these#

1. Matrix — a grid of wires so a few pins can read many keys — an m x n row/column arrangement needing m+n GPIO pins for m*n keys.
2. Scan — one full pass over all rows — the periodic sequential strobe and sample loop, typically 100 Hz to 5 kHz.
3. Ghosting — a key that appears pressed but is not — a phantom detection from a sneak current path through three closed switches.
4. Masking — keys deliberately dropped to avoid ghosts — jamming, the firmware suppressing an ambiguous rectangle pattern.
5. Diode — a one-way valve for current — a small-signal rectifier per key blocking the reverse sneak path, giving NKRO.
6. N-key rollover — every key readable at once — unlimited simultaneous key detection, requiring both per-key diodes and a suitable HID report format.

21.3 The keyboard’s own computer#

PLAIN21.3.1 in simple words#

1. Inside your keyboard there is a small computer. A real one, with a processor, memory and a program.
2. It is not powerful. It might run at 16 to 100 million cycles a second, with a few kilobytes of memory. Your phone is a hundred thousand times bigger.
3. Its whole job is: scan the grid, clean up the signal, decide what changed, and tell the host machine.
4. When it decides that the key at row 2, column 0 has gone down, it does not send “row 2, column 0”. It looks up a table.
5. The table says: this position on this keyboard corresponds to code number such-and-such.
6. That number is called a scan code, or in USB language a usage code.
7. Here is the part people get wrong. That number is not a letter. It is not the letter A.
8. It is a name for a physical position on the keyboard. “The key that is one to the right of Caps Lock”, roughly.
9. If you switch your computer to a French layout, that same key produces Q on screen. The keyboard sent the identical number both times.
10. So the keyboard does not know your language, your layout, or what you typed. It only knows which physical key moved.

PLAIN21.3.2 a picture in your head#

1. Imagine a hotel with numbered rooms and a bell board behind the reception desk.
2. When someone in room 214 presses their bell, a flag drops on the board labelled “214”.
3. The board does not say “Mrs Patel wants tea”. It says “214”.
4. Reception has a separate list mapping rooms to guests. Guests change every week. The bell board never changes.
5. The bell board is the keyboard. The room number is the scan code. The guest list is your keyboard layout, and it lives at reception, not in the wiring.
6. This split is why a hotel can rewire nothing when new guests arrive, and why your computer can switch between English, French and Devanagari instantly without you touching the keyboard.

Where this comparison breaks: the hotel board reports only “someone pressed”. The keyboard reports both press and release, and reports the full set of keys currently held, so the computer can know that Shift and A are down at the same moment. Also, some keyboards do relabel themselves in firmware, which would be like the bell board printing guest names. That is a firmware remap, and it is the exception, not how layouts normally work.

PLAIN21.3.3 a worked example#

1. You press the A key on a standard 104-key board wired to a US layout.
2. The firmware’s key map is an array. Position [row 2][column 0] holds the value KC_A, which in QMK is the number 0x04.
3. That 0x04 is the USB HID usage code for “Keyboard a and A” on Usage Page 0x07.
4. The name in the specification is literally “Keyboard a and A”. The specification refuses to pick one, because that choice is not the keyboard’s to make.
5. Now compare the three older PS/2 scan code sets for the same physical key.

6. Four different numbers for one physical key, across four standards. None of them is the ASCII code for A, which is 0x41, or for a, which is 0x61.
7. Notice the last row. USB does not send a release code at all. It sends the full list of keys currently held down. A key disappearing from the list is the release.
8. Now hold Shift and press A. The firmware does not combine them. Shift is reported as a separate bit in a separate byte, and A is still 0x04.
9. Combining them into a capital letter is somebody else’s job, several stages later.

PLAIN21.3.4 what is really happening inside#

1. The microcontroller runs a loop, forever, in this order.
2. Scan the matrix into a bitmap. Compare with the last bitmap. Feed changes into the debounce filter.
3. For each debounced change, look up the key map for the current layer.
4. Layers are a firmware idea. Holding a Fn key switches to a second key map array, so the same physical key gives a different code. This is how a 60-key board offers arrow keys.
5. Sort the resulting codes into modifiers and normal keys. The eight modifiers, left and right Ctrl, Shift, Alt and GUI, go into a single byte as bits.
6. Normal keys go into a list, in the order they were pressed, oldest first.
7. Build the report. Byte 0 is the modifier bits. Byte 1 is reserved and always zero. Bytes 2 to 7 are up to six normal key codes.
8. Hand the report to the USB stack, which puts it in a buffer attached to an endpoint and waits for the host to ask.
9. If more than six normal keys are held, the boot protocol has no room. The specification says to fill all six slots with 0x01, the code named ErrorRollOver.
10. There is no auto-repeat in the keyboard. Hold a key and the keyboard just keeps reporting the same list. The repeating you see when you hold a key is generated by the operating system with a timer.

TECHNICAL21.3.5 the engineer’s version#

1. Common keyboard microcontrollers: Atmel ATmega32U4 (8-bit AVR, 16 MHz, 2.5 KB SRAM, native USB) in older custom boards; STM32F072/F103 and RP2040 in newer ones; proprietary parts from Holtek, SONiX and Nuvoton in mass-market boards.
2. The USB HID Usage Tables document defines Usage Page 0x07 as the Keyboard/ Keypad page. Selected values:

3. Note 0xE1. Left Shift has a usage ID, but in the boot report it is carried as bit 1 of byte 0 rather than as a key code. Both representations exist and report-protocol descriptors normally declare the modifier byte as a bitmap of usages 0xE0 to 0xE7.
4. PS/2 scan code set 2 is what AT-class keyboards actually send on the wire. The 8042 keyboard controller on the motherboard translates set 2 into set 1 before the operating system sees it, when translation is enabled. That translation is why so much documentation disagrees about which set is “the” scan code set. Set 3 was defined for the IBM 3270 terminal style and is rarely used, though it has the useful property that every key can be individually configured as make-only, make-break or typematic.
5. USB was published as version 1.0 in January 1996 by a group including Intel, Microsoft, Compaq, IBM, DEC, NEC and Nortel. The HID class specification version 1.11 dates from 2001 and is still the current document.
6. Legacy support: the BIOS or UEFI cannot run a full USB stack early in boot, so the boot protocol exists. SET_PROTOCOL with value 0 selects boot protocol, value 1 selects report protocol. Firmware also historically used SMM-based USB legacy emulation to make a USB keyboard appear at I/O ports 0x60 and 0x64 like an 8042.
7. Firmware options in wide use: QMK (2015 on, from Jun Wako’s TMK), ZMK for wireless, VIA and Vial for runtime remapping, and closed vendor firmware. QMK’s keymap.c is a C array of arrays, one per layer.
8. Latency inside the microcontroller for the lookup and report build is small: on a 16 MHz AVR, a full 8x18 scan plus debounce plus report build measures in the tens of microseconds, so under 0.1 ms. The scan period, not the computation, is the cost.
9. Where the layout is NOT decided, said plainly: not in the switch, not in the matrix, not in the scan code table, and not in the USB report. All of those carry physical position only.

WORDS21.3.6 remember these#

1. Microcontroller — the tiny computer in the keyboard — an MCU with CPU, flash, SRAM and a USB device peripheral, typically 8 to 100 MHz.
2. Scan code — a number naming a physical key — a make/break code from PS/2 sets 1, 2 or 3, or a HID usage ID on Usage Page 0x07.
3. Usage code — the USB name for a key — a 16-bit usage page plus usage ID pair identifying a control, defined in the HID Usage Tables.
4. Modifier — Shift, Ctrl, Alt, Cmd — usages 0xE0 to 0xE7, carried as a bitmap in byte 0 of the boot keyboard report.
5. Layer — a second key map reached by holding a key — a firmware-level alternate keycode array, selected by momentary or toggle actions.
6. ErrorRollOver — the code meaning “too many keys” — usage 0x01 placed in all six slots when more keys are held than the report can carry.

21.4 Getting the data out: USB HID#

PLAIN21.4.1 in simple words#

1. The keyboard now has an 8-byte message ready. It has to get it to the computer.
2. USB does not let a device speak whenever it wants. The computer is in charge of the wire, always.
3. The keyboard cannot interrupt. It can only put its message in a small mailbox and wait to be asked.
4. The computer asks on a fixed schedule. “Anything for me?” Over and over, forever, whether you are typing or not.
5. If the mailbox is empty, the keyboard answers with a tiny “nothing”, which costs almost no time on the wire.
6. If the mailbox has the message, the keyboard hands it over and the computer acknowledges it.
7. The schedule is normally once every millisecond. Some fast devices are asked every 125 microseconds, which is eight times a millisecond.
8. So on average your key press waits half of one asking-interval before it can leave the keyboard.
9. That mailbox has a name: an endpoint. The kind of asking is called an interrupt transfer, which is a confusing name because nothing is interrupted. The device is polled.

PLAIN21.4.2 a picture in your head#

1. Think of a postal round in a village with one postman and many houses.
2. The postman walks the same route every minute, stopping at every house that has asked to be visited.
3. At each stop the postman says “anything to send?” The householder either hands over a letter or says “nothing today”.
4. A householder cannot run down the street and shove a letter into the postman’s bag out of turn. They must wait for the visit.
5. If you write a letter just after the postman leaves, it waits nearly a full minute. If you write it just before he arrives, it goes almost at once.
6. On average it waits half a minute. That is exactly the average USB polling delay.
7. A house can ask to be visited more often, and the postman will agree if the route has room. That is what setting a shorter polling interval means.

Where this comparison breaks: the postman visits in a fixed order, whereas USB visits in a schedule computed by the host controller that can be reordered and that reserves bandwidth in advance. And the postman carries the letter away; USB copies the data and leaves the device’s buffer to be refilled.

PLAIN21.4.3 a worked example#

1. Here is the exact 8-byte boot keyboard report format.

byte 0 : modifier bits
byte 1 : reserved, always 0x00
byte 2 : key code slot 1
byte 3 : key code slot 2
byte 4 : key code slot 3
byte 5 : key code slot 4
byte 6 : key code slot 5
byte 7 : key code slot 6

2. The modifier byte, bit by bit, lowest bit first:

bit 0 : Left  Ctrl     bit 4 : Right Ctrl
bit 1 : Left  Shift    bit 5 : Right Shift
bit 2 : Left  Alt      bit 6 : Right Alt
bit 3 : Left  GUI      bit 7 : Right GUI

3. You press A alone. The report on the wire is:

00 00 04 00 00 00 00 00

4. Read it: no modifiers, reserved zero, key code 0x04 in slot 1, nothing else.
5. You now hold Left Shift and press A. Two reports go out, because Shift lands first.

02 00 00 00 00 00 00 00     Left Shift down, no keys
02 00 04 00 00 00 00 00     Left Shift down, A down

6. Byte 0 is 0x02, which is binary 00000010, so bit 1 is set, which is Left Shift. Byte 2 is 0x04, which is the A key.
7. Notice what is not there. There is no capital A anywhere. There is no character at all. Two independent physical facts: a modifier is held, and a key position is down.
8. You release A but keep Shift. The report becomes 02 00 00 00 00 00 00 00 again. The disappearance of 0x04 is the release event.
9. You release Shift. The report becomes all zeros.
10. Now type A, then B without releasing A, then C without releasing either:

00 00 04 00 00 00 00 00     A
00 00 04 05 00 00 00 00     A then B
00 00 04 05 06 00 00 00     A then B then C
00 00 05 06 00 00 00 00     A released, B and C shuffle down

11. Press more than six normal keys at once and the boot report cannot carry them, so all six slots become 0x01:

20 00 01 01 01 01 01 01     Right Shift held, rollover error

PLAIN21.4.4 what is really happening inside#

1. When you plug the keyboard in, the host asks it to describe itself. This is called enumeration.
2. The device returns a device descriptor, a configuration descriptor, one or more interface descriptors, endpoint descriptors and a HID report descriptor.
3. The interface descriptor says class 3, which means HID. Subclass 1 means it supports the boot protocol. Protocol 1 means keyboard, protocol 2 means mouse.
4. One of the endpoint descriptors says “interrupt IN”, meaning data flows device to host on a polled schedule. It carries a field called bInterval.
5. bInterval sets how often the host asks. On a full-speed device it is a number of 1 ms frames, from 1 to 255. On a high-speed device it is an exponent: the period is 2 to the power of (bInterval minus 1), counted in 125 microsecond microframes.
6. So the fastest possible polling is 125 microseconds, which is 8000 times a second, and that requires a high-speed connection.
7. Once running, the host controller issues an IN token to the device’s address and endpoint number at each scheduled slot.
8. If the keyboard has a new report, it replies with a DATA0 or DATA1 packet containing the 8 bytes, and the host replies ACK.
9. If nothing changed, the keyboard replies NAK, which is 3 bytes on the wire and takes under a microsecond.
10. Bluetooth keyboards do none of this. They use HID over GATT, a profile on Bluetooth Low Energy, where reports become notifications on a GATT characteristic.
11. Bluetooth Low Energy has its own polling clock called the connection interval, and it is much coarser. That is the single biggest reason wireless keyboards feel slower.

TECHNICAL21.4.5 the engineer’s version#

1. USB speeds and their interrupt endpoint limits:

2. Per the USB 2.0 specification, section 9.6.6, a low-speed interrupt endpoint must declare bInterval between 10 and 255 in 1 ms frames; a full-speed one between 1 and 255 in 1 ms frames; and a high-speed one between 1 and 16 as an exponent, giving a period of 2^(bInterval-1) microframes of 125 us.
3. Most plain USB keyboards enumerate as low speed or full speed. Dan Luu’s 2017 measurements list many as “USB FS”. A low-speed keyboard with bInterval 10 is polled 100 times a second and adds an average of 5 ms.
4. Gaming peripherals advertising 1000 Hz are full speed with bInterval 1. Those advertising 4000 Hz or 8000 Hz must be high speed with bInterval 1 or 2, because full speed physically cannot go below a 1 ms frame.
5. The report descriptor for a boot-compatible keyboard is a well-known 63-byte blob. Its meaning, decoded:

Usage Page (Generic Desktop), Usage (Keyboard)
Collection (Application)
  Usage Page (Keyboard/Keypad)
  Usage Min (0xE0) Usage Max (0xE7)      modifiers
  Logical Min (0) Logical Max (1)
  Report Size (1) Report Count (8)
  Input (Data, Variable, Absolute)       -> byte 0
  Report Size (8) Report Count (1)
  Input (Constant)                       -> byte 1 reserved
  Report Size (8) Report Count (6)
  Logical Min (0) Logical Max (101)
  Usage Min (0) Usage Max (101)
  Input (Data, Array)                    -> bytes 2..7
End Collection

6. Note the difference in the two Input items. The modifier byte is Variable, meaning each bit is its own control. The six key slots are Array, meaning each byte holds the index of whichever control is active. That single distinction is why modifiers are a bitmap and keys are a list.
7. NKRO keyboards add a second report, or a second interface, declaring a Variable bitmap over usages 0x00 to 0xF7, which is 31 bytes. Every key then has its own bit and there is no six-key limit.
8. Bluetooth: HID over GATT Profile (HOGP) 1.0 was adopted by the Bluetooth SIG in 2011. Connection interval is negotiated in units of 1.25 ms, with a permitted range of 7.5 ms to 4 s. Apple’s accessory guidelines historically asked for intervals no shorter than 15 ms for most accessories, with exceptions for HID. Vendors using proprietary 2.4 GHz dongles, such as Logitech Lightspeed or Razer HyperSpeed, bypass Bluetooth entirely and can hold a 1 ms interval.
9. Tools: lsusb -v on Linux prints bInterval and the full descriptor tree. usbhid-dump prints report descriptors. On Windows, USB Device Tree Viewer shows the same. Wireshark with usbmon captures the actual reports.
10. Honest note on 8000 Hz polling: it removes at most about 0.44 ms of average wait compared with 1000 Hz. It also multiplies host interrupt load by eight. Whether that trade is worth it is contested, and blind tests have not shown a consistent human benefit above roughly 1000 Hz.

WORDS21.4.6 remember these#

1. Endpoint — a mailbox on the device — a uniquely addressed source or sink of data, identified by number and direction.
2. Interrupt transfer — the host asking on a timer — a guaranteed-bandwidth, bounded-latency polled transfer type, despite the name.
3. bInterval — how often the host asks — the polling interval field of the endpoint descriptor, in frames or as a power-of-two microframe exponent.
4. Boot protocol — the simple fixed 8-byte format — the reduced report format defined by HID subclass 1 so firmware can drive a keyboard without a parser.
5. Report descriptor — the device explaining its own data format — a self-describing byte stream of HID items parsed into a control tree.
6. NAK — “nothing to send” — a handshake packet meaning the endpoint has no data ready, costing about 3 bytes of bus time.
7. HOGP — Bluetooth’s version of HID — HID over GATT Profile, carrying reports as BLE notifications on a negotiated connection interval.

21.5 Into the machine: the host controller#

PLAIN21.5.1 in simple words#

1. The 8 bytes have left the keyboard and travelled down the cable. They arrive at a chip on the motherboard called the host controller.
2. The host controller is the thing that actually runs the USB schedule. It is the postman from the earlier picture.
3. It is not the CPU. It is a separate piece of hardware that works while the CPU is doing other things.
4. The operating system has already told the host controller, in advance, where in memory to put anything that arrives from this keyboard.
5. So when the bytes arrive, the host controller writes them straight into main memory itself, without asking the CPU.
6. Hardware writing to memory on its own is called DMA, direct memory access.
7. Then the host controller writes a small note into another area of memory saying “a transfer finished, here is what happened”.
8. Only then does it poke the CPU, by sending a special message that makes the CPU stop what it is doing and run some kernel code.
9. That poke is an interrupt. This time the word is accurate: something really is interrupted.

PLAIN21.5.2 a picture in your head#

1. Think of a busy kitchen with a chef and a delivery hatch.
2. The chef does not stand at the hatch waiting for deliveries. That would waste the whole day.
3. Instead there are labelled shelves. The chef has already written a list: “onions go on shelf 3, flour goes on shelf 7”.
4. The delivery person puts the goods on the right shelf without speaking to the chef at all. That is DMA.
5. Then the delivery person writes a line in a notebook by the door: “onions, shelf 3, 10:42, all fine”. That notebook is the event ring.
6. Finally, they ring a bell. The chef stops chopping, wipes their hands, reads the notebook, and deals with whatever arrived.
7. The bell is the interrupt. Ringing it costs the chef a few seconds of disruption, so you do not ring it for every single carrot.

Where this comparison breaks: the chef can ignore the bell for a while; a CPU takes the interrupt at the very next instruction boundary unless interrupts are explicitly masked. And the notebook is a fixed-size circular list, so if the chef reads too slowly, the delivery person eventually has to stop and wait.

PLAIN21.5.3 a worked example#

1. Modern machines use xHCI, the eXtensible Host Controller Interface, released by Intel in 2010. It replaced the older UHCI, OHCI and EHCI designs.
2. Before any keys are pressed, the driver has set up these structures in ordinary system memory.

3. Each entry in these rings is a 16-byte block called a TRB, a Transfer Request Block.
4. The driver puts a Normal TRB on the keyboard’s transfer ring saying: “when data arrives on this endpoint, write up to 8 bytes to physical address 0x1A2B3000, then post an event”.
5. The driver then writes to a doorbell register on the controller to say “there is new work on this ring”. Doorbells are the only place the driver pokes the hardware directly.
6. Time passes. You press A. At the next 1 ms slot, the controller polls the keyboard and gets the 8 bytes.
7. The controller performs a DMA write of those 8 bytes to 0x1A2B3000. This goes over PCI Express, through the IOMMU if one is enabled, into RAM.
8. The controller then writes a Transfer Event TRB onto the event ring, saying which endpoint, how many bytes, and a completion code of Success.
9. The controller then raises an MSI interrupt, which is itself just a PCI Express memory write to a special address that the interrupt controller watches.
10. The CPU takes the interrupt. Total elapsed time from the last bit on the USB wire to the first instruction of the kernel handler: roughly 2 to 10 microseconds on a modern desktop.

PLAIN21.5.4 what is really happening inside#

1. Let us be precise about “the CPU is interrupted mid-instruction”.
2. On x86 and on Arm, the processor checks for pending interrupts at instruction boundaries, not in the middle of an instruction.
3. So the current instruction finishes and retires. A few very long instructions, such as x86 repeated string moves, are designed to be interruptible and restartable partway.
4. The core then does the following, in hardware, with no software involved.
5. It saves the current instruction pointer, the flags register and the stack information onto a stack.
6. It reads the interrupt vector number, a value between 0 and 255 on x86, that came with the MSI message.
7. It uses that number as an index into the Interrupt Descriptor Table, a table the kernel built at boot, to find the address of the handler.
8. It switches privilege level from user mode to kernel mode if it was in user code, and switches to the kernel stack.
9. It jumps to the handler. From the program’s point of view, absolutely nothing happened; when the handler returns, execution resumes at the exact next instruction with all registers intact.
10. Whatever the CPU was running has now had its pipeline flushed and quite possibly its caches disturbed. That is the real cost of an interrupt, and it is why interrupt moderation exists.

TECHNICAL21.5.5 the engineer’s version#

1. xHCI 1.0 was published by Intel in 2010; the current revision is 1.2 with errata. It unified low, full, high and SuperSpeed handling in one register interface, which the earlier split of UHCI/OHCI (USB 1.x) and EHCI (USB 2.0) did not.
2. Structures, precisely: the Device Context Base Address Array (DCBAA) is indexed by slot ID. Each device context contains a slot context and up to 31 endpoint contexts. Each endpoint context points at a transfer ring.
3. Rings are circular arrays of 16-byte TRBs. The last TRB in a segment is a Link TRB pointing back to the start. A Cycle Bit is toggled each time the producer wraps, which is how consumer and producer distinguish new entries from stale ones without a separate index.
4. The Event Ring Segment Table (ERST) describes the event ring’s memory. The controller is the producer of events; the driver is the consumer and updates the Event Ring Dequeue Pointer register when it has processed them.
5. Doorbell registers: one per device slot, plus doorbell 0 for the command ring. Writing a doorbell tells the controller to re-examine a ring. This is the only per-transfer MMIO write in the fast path, which matters because MMIO writes to a PCIe device cost hundreds of nanoseconds.
6. Interrupts: MSI was introduced in PCI 2.2 (1998) and MSI-X in PCI 3.0 (2004). Both replace a physical interrupt line with a posted memory write to an address decoded by the interrupt controller, which removes the shared-line ambiguity and the need to read a device register to find out who interrupted.
7. xHCI provides an Interrupt Moderation register per interrupter, expressed in 250 nanosecond units, which sets a minimum gap between interrupts. Raising it reduces interrupt rate and increases worst-case latency. Exact defaults are an implementation detail that differs between the specification’s reset value and what Linux, Windows and macOS drivers actually program.
8. IOMMU involvement: with an IOMMU enabled (Intel VT-d, AMD-Vi, Arm SMMU) the controller’s DMA addresses are I/O virtual addresses translated per device. This costs an IOTLB lookup, normally tens of nanoseconds when hot, and it is what stops a malicious USB controller from reading arbitrary memory.
9. Rough interrupt entry costs on x86-64, order of magnitude:

10. Tools: cat /proc/interrupts shows per-CPU counts per IRQ, including xhci_hcd. sudo perf stat -e irq:irq_handler_entry counts handler entries. sudo trace-cmd record -e xhci-hcd traces xHCI ring activity.

WORDS21.5.6 remember these#

1. Host controller — the chip that runs the USB bus — the xHCI device that schedules transfers and DMAs data, independent of the CPU.
2. DMA — hardware writing to memory by itself — direct memory access, a bus master writing to system RAM without CPU involvement.
3. TRB — one entry in a work list — a 16-byte Transfer Request Block on an xHCI transfer, command or event ring.
4. Ring — a circular list in memory — a producer/consumer queue with a Cycle Bit distinguishing fresh from stale entries.
5. Doorbell — the poke that says “new work” — a memory-mapped register write telling the controller to re-scan a ring.
6. MSI — an interrupt sent as a memory write — Message Signaled Interrupt, a posted PCIe write to an address the interrupt controller decodes.
7. IDT — the table of interrupt handlers — the Interrupt Descriptor Table, 256 gate descriptors indexed by vector number on x86.

21.6 The kernel takes over#

PLAIN21.6.1 in simple words#

1. The CPU is now running kernel code because of the interrupt. This code is called the interrupt service routine, or handler.
2. The handler must be fast. While it runs, other things are held up, so it does the minimum and defers the rest.
3. It reads the event note the controller left, finds out which device and which transfer finished, and marks that transfer complete.
4. It then schedules the real work to happen a moment later, outside the handler. On Linux that deferred half is a softirq or a work queue.
5. The real work starts in the USB core, the part of the kernel that knows about USB in general but nothing about keyboards.
6. The USB core looks at which driver claimed this interface and hands the 8 bytes to it. For a keyboard, that is the HID class driver.
7. The HID driver already read the device’s report descriptor at plug-in time and built a map of where every field lives inside the report.
8. It uses that map to pull out “modifier bits” and “the six key slots”, then compares with the previous report to work out what changed.
9. A key present now but not before is a press. A key present before but not now is a release.
10. Finally it translates each USB usage code into the operating system’s own key code, and hands that to the input subsystem, which is the thing applications can actually read from.

PLAIN21.6.2 a picture in your head#

1. Imagine a large office building’s post room.
2. A courier drops a sack at the loading bay and rings the bell. Somebody signs for it in ten seconds flat and goes back to work. That is the interrupt handler: sign, do not open.
3. Later, a sorter opens the sack. They do not read the letters. They look at the department code on each envelope and put it in the right pigeonhole. That is the USB core.
4. The department clerk opens the envelope. They know this department’s forms, so they know that box 3 on the form is “employee number” and box 7 is “date”. That is the HID driver using the report descriptor.
5. The clerk then rewrites the information onto the building’s own standard form, because every department in the building uses different paper but the building’s computer only accepts one format. That is the translation from HID usages to kernel key codes.
6. The standard form goes into a tray that anybody with permission can read from. That tray is the input subsystem.

Where this comparison breaks: the post room is sequential and slow. The kernel does all of this in a few microseconds, and the “tray” is a shared memory structure with a wake-up mechanism, so readers are woken the instant something lands rather than checking periodically.

PLAIN21.6.3 a worked example#

1. On Linux you can watch this directly. evtest opens a device in /dev/input/ and prints every event.
2. Press and release A on a US keyboard and you see this.

Event: time 1755105143.201933, type 4 (EV_MSC),
       code 4 (MSC_SCAN), value 70004
Event: time 1755105143.201933, type 1 (EV_KEY),
       code 30 (KEY_A), value 1
Event: time 1755105143.201933, -------- SYN_REPORT --------
Event: time 1755105143.298114, type 4 (EV_MSC),
       code 4 (MSC_SCAN), value 70004
Event: time 1755105143.298114, type 1 (EV_KEY),
       code 30 (KEY_A), value 0
Event: time 1755105143.298114, -------- SYN_REPORT --------

3. Line by line. EV_MSC with MSC_SCAN and value 70004 is the raw hardware code: usage page 0x0007, usage 0x0004. The kernel passes it through so tools can remap unusual keys.
4. EV_KEY with code 30 is KEY_A, the Linux input key code for that physical key. It is 30, not 4 and not 65. A third numbering.
5. value 1 means pressed. value 0 means released. value 2 means auto-repeat, generated by a kernel timer, not by the keyboard.
6. SYN_REPORT marks the end of one batch of simultaneous events. Everything between two SYN_REPORT lines happened at the same instant.
7. The gap between the two timestamps is 96.181 ms. That is how long the finger was on the key, which is a completely normal typing dwell time.
8. Notice what is still not present. Nothing anywhere says the letter A. Still no character.

PLAIN21.6.4 what is really happening inside#

1. Order of events inside Linux, in detail.
2. xhci_irq runs as the hard interrupt handler. It reads the event ring, finds the Transfer Event TRB, and calls the completion function attached to the URB.
3. An URB, USB Request Block, is the kernel’s object for one transfer. Its completion runs the HID driver’s usb_hid_irq_in callback.
4. The HID driver calls hid_input_report with the 8 bytes. This is where the parsed report descriptor is used.
5. hid_input_report walks the report’s fields. For each field it extracts the bits and compares with the stored previous value.
6. For each changed usage it calls hid_input_field, which maps the HID usage to a Linux input event through hidinput_hid_event.
7. The mapping is a table. Usage page 0x07, usage 0x04 becomes EV_KEY, KEY_A, which is 30.
8. input_event is called, which puts the event into the input core.
9. The input core hands the event to every registered handler. evdev is one such handler and it appends to a per-open-file circular buffer, then wakes any process sleeping on that file descriptor.
10. Interrupt handlers must not sleep, so all of this after the initial read runs in softirq context or a tasklet. Under a normal load this whole path takes 5 to 50 microseconds.
11. Under a heavily loaded system with the CPU busy, the wake-up of the reading process is at the mercy of the scheduler, and that is where a millisecond can quietly disappear.

TECHNICAL21.6.5 the engineer’s version#

1. Three operating systems, three input stacks, doing the same job:

2. Linux: character devices appear as /dev/input/eventN with a stable alias under /dev/input/by-id/. The protocol is a stream of struct input_event { struct timeval time; __u16 type; __u16 code; __s32 value; }. Key codes live in include/uapi/linux/input-event-codes.h. KEY_A is 30, KEY_LEFTSHIFT is 42, KEY_ENTER is 28.
3. macOS: IOHIDFamily parses the report descriptor into IOHIDElement objects. Events become IOHIDEvents, then NSEvents once they reach the window server. Since macOS 10.15 (Catalina, 2019) reading raw HID input from another application requires Input Monitoring permission, which is why key loggers and remapping tools prompt for access.
4. Windows: HIDCLASS.SYS parses reports. kbdhid.sys translates HID usages into PS/2-style scan codes for backward compatibility, then kbdclass.sys feeds Win32k, which posts a WM_KEYDOWN message to a thread’s message queue. Alternatively an application can call RegisterRawInputDevice and receive WM_INPUT with the raw HID report, bypassing the translation. Games do this, because the message path applies key repeat, focus filtering and accessibility features that a game does not want.
5. The Windows raw input path is genuinely lower latency than the message path, mostly because it avoids waiting for the message pump and avoids the translation layers, not because the reports arrive earlier.
6. Linux key code space: KEY_MAX is 0x2FF (767), so there is room for many more keys than any keyboard has. Codes above 255 cannot be expressed in the older X11 protocol without an offset of 8, which is where the famous “X keycode equals evdev code plus 8” rule comes from.
7. Timestamps: evdev timestamps are taken in the input core when the event is generated, not when userspace reads it. Since Linux 4.4 you can request CLOCK_MONOTONIC timestamps with EVIOCSCLOCKID, which is what compositors do so that input times and frame times are comparable.
8. Tools: evtest, libinput debug-events, sudo showkey -s for raw scan codes on a console, xinput test-xi2 on X11, wev on Wayland, dmesg for enumeration messages, hid-recorder from the hid-tools project for capturing raw reports.

WORDS21.6.6 remember these#

1. ISR — the code that runs when hardware pokes the CPU — the interrupt service routine, running with interrupts partly disabled and forbidden to sleep.
2. URB — the kernel’s object for one USB transfer — USB Request Block, submitted asynchronously with a completion callback.
3. HID class driver — the code that understands report layouts — the driver that parses report descriptors and emits generic input events.
4. evdev — the Linux input file interface — the event device layer exposing /dev/input/eventN as a stream of input_event structures.
5. Key code — the operating system’s own number for a key — a KEY_* constant on Linux, a virtual key on Windows, distinct from both scan codes and characters.
6. SYN_REPORT — the marker for “that batch is finished” — an EV_SYN event grouping simultaneous events into one atomic update.

21.7 From key code to character#

PLAIN21.7.1 in simple words#

1. So far, at every single stage, nothing has known what letter you typed.
2. The switch knew “closed”. The matrix knew “row 2, column 0”. The keyboard knew “usage 0x04”. The kernel knew “KEY_A, value 1”.
3. All of those are names for a position on a slab of plastic.
4. Now, finally, something turns that position into a character.
5. The thing that does it is a table called a keymap, and it belongs to your user account, not to your hardware.
6. The keymap says: for this key, with no modifiers, produce a. With Shift, produce A. With Alt Gr, produce something else again.
7. Change your keymap to French and the same key produces q and Q instead. You have not touched the keyboard.
8. Some keys produce nothing on their own. Press the acute accent key on a French keyboard and nothing appears. Press e afterwards and é appears. That is a dead key.
9. Some systems let you type a sequence: a special Compose key, then o, then c, giving the copyright sign. That is a compose sequence.
10. And for languages with thousands of characters, a whole extra program sits in the middle, taking your keystrokes and offering you candidate words to pick from. That is an input method editor.

PLAIN21.7.2 a picture in your head#

1. Think of a piano with 88 keys and an orchestra behind a curtain.
2. Each piano key sends a number to the conductor. Key 40, key 41, and so on.
3. The conductor holds a sheet that says what each number means today. Today’s sheet says number 40 means a violin note.
4. Tomorrow the conductor swaps in a different sheet, and number 40 means a drum. The piano is unchanged. The keys are unchanged. The sound is different.
5. Now imagine a language with 50,000 sounds and only 88 keys. No sheet can map one key to one sound.
6. So instead the pianist plays a short phrase, and an assistant listens, guesses which of several possible sounds was meant, and holds up a small list to choose from.
7. That assistant is the input method editor. It sits between the keys and the result, and it is a real program with its own memory and its own predictions.

Where this comparison breaks: a piano key really does have a fixed physical string, and a keyboard key genuinely has nothing fixed at all. Also, the conductor’s sheet in real systems is not one flat list. It is a two-dimensional table with a group axis, for which language you are in, and a level axis, for which modifiers you hold.

PLAIN21.7.3 a worked example#

1. One physical key. Four different keymaps. Four different results.

2. The same physical key on a French AZERTY board is where Q lives, so the letter differs even though the code sent was identical.
3. Now dead keys. On a French keyboard, the circumflex key sends its own key code. The keymap marks it as a dead key.
4. Press it: nothing appears. The system remembers “a circumflex is pending”.
5. Press e: the system combines pending circumflex with e and produces ê.
6. Press the space bar instead: the system produces the bare ^ character.
7. Now compose sequences. On Linux, if you set a Compose key, this file controls it:

/usr/share/X11/locale/en_US.UTF-8/Compose

<Multi_key> <o> <c>          : "(c)"   copyright
<Multi_key> <a> <e>          : "ae"    ae ligature
<Multi_key> <minus> <minus> <period> : "--"  en dash

8. Now an input method editor. To type the Japanese word for Tokyo you might type toukyou on a plain US keyboard.
9. The IME converts those letters into kana as you go, showing the five kana to-u-kyo-u underlined as pre-edit text. Nothing is committed yet.
10. You press space. The IME offers a candidate list of kanji spellings that all read “toukyou”: the city name, a rarer compound, and others. You pick one with the arrow keys and press Enter.
11. Only then does the application receive the finished text. Everything before that was a conversation between you and the IME.

PLAIN21.7.4 what is really happening inside#

1. A keymap is not a simple list. It has two axes.
2. The first axis is the group, which means which layout is active. You can have several loaded and switch between them with a hotkey.
3. The second axis is the level, which is chosen by which modifiers you hold. Level 1 is plain, level 2 is Shift, level 3 is Alt Gr, level 4 is Alt Gr with Shift.
4. So a lookup is: this key code, in this group, at this level, gives this symbol.
5. Modifiers themselves are stateful. The system tracks which are held, which are latched (pressed once, applying to the next key), and which are locked (Caps Lock).
6. Caps Lock is not Shift. Caps Lock normally affects only letters, whereas Shift affects every key. That is why Shift-2 gives @ but Caps Lock plus 2 still gives 2.
7. Dead keys are implemented as a small state machine. The keymap marks certain symbols as combining. When one is produced, the system stores it and waits for the next symbol.
8. If the next symbol has no defined combination, most systems emit both separately rather than losing your keystroke.
9. An input method editor is a separate process. It gets keystrokes first, before the application, and decides whether to consume them or pass them on.
10. While it is composing, it sends the application “pre-edit” text, which the application is expected to display inline but not to treat as committed.
11. When you confirm, the IME sends a commit event with the final string, and the pre-edit is cleared.
12. Why this cannot live in the keyboard: an IME needs a dictionary of hundreds of thousands of words, learns from your habits, needs megabytes of storage, and must know what the application is doing. A 2 KB microcontroller cannot and should not do that.

TECHNICAL21.7.5 the engineer’s version#

1. Linux and X11 use XKB, the X Keyboard Extension, standardized in X11R6.1 in 1996. Its data lives in the xkeyboard-config package under /usr/share/X11/xkb/ split into keycodes, types, compat, symbols and geometry.
2. A symbols entry looks like this, and this is the actual syntax:

key <AC01> { [ a, A, ae, AE ] };
key <AD01> { [ q, Q, at,  Omega ] };

3. <AC01> is a position name meaning alphanumeric section, row C, key 01, counting from the bottom-left. The four entries are levels 1 to 4. Position names, not letters, are the identifiers. That is the whole point.
4. libxkbcommon is the modern library that implements this without needing an X server; Wayland compositors and many terminals use it directly.
5. macOS uses .keylayout XML bundles in /Library/Keyboard Layouts/, evaluated by UCKeyTranslate from the Carbon Unicode Utilities. Ukelele is the common editor for them.
6. Windows keyboard layouts are DLLs, historically built with the Microsoft Keyboard Layout Creator. kbdus.dll is the US layout. Applications call ToUnicodeEx to run a virtual key plus keyboard state through the active layout and get characters back.
7. Input method frameworks:

8. Scale of the problem: Unicode 16.0, published September 2024, defines 154,998 characters. A keyboard has around 104 keys. The ratio is roughly 1,500 characters per key. No hardware solution exists, and none can exist.
9. Korean is a special case worth knowing: Hangul syllables are algorithmically composed from jamo, so a 2-set keyboard plus a composition automaton produces all 11,172 modern syllable blocks with no dictionary at all. It is an input method, but a deterministic one.
10. Indic scripts need reordering as well as composition. In Devanagari, the vowel sign i is typed after its consonant but displayed before it, so the input method and the text shaper both have work to do on the same keystroke.
11. Tools: setxkbmap -print -verbose 10 shows the active XKB configuration; xkbcomp $DISPLAY out.xkb dumps the compiled keymap; xkbcli interactive-wayland shows live keysym resolution; ibus engine and fcitx5-remote query the active input method.
12. Honest note on where experts disagree: some argue that firmware remapping, as offered by QMK and VIA, is better because the layout follows the keyboard between machines. Others argue it is worse because the operating system then cannot know what the layout is, breaking shortcut display, accessibility tools and input methods. Both are right about their own failure mode.

WORDS21.7.6 remember these#

1. Keymap — the table turning key positions into characters — a per-user mapping from key code plus modifier state to keysyms and text.
2. Group — which language layout is active — the XKB axis selecting among several loaded layouts, switched by a hotkey.
3. Level — which modifier row of the table applies — the XKB axis selected by Shift, Alt Gr and their combinations, typically four levels.
4. Dead key — a key that waits for the next one — a combining accent that produces no character alone but modifies the following one.
5. Compose sequence — a short recipe of keys giving one character — a multi-key sequence resolved by a Compose table.
6. IME — the helper program for large scripts — an input method editor converting keystrokes to candidate text through a dictionary and a pre-edit/commit protocol.
7. Pre-edit — text you are still composing — uncommitted composition text sent to the application for display but not inserted into the document.

21.8 The window system#

PLAIN21.8.1 in simple words#

1. Your screen has many windows on it. Only one of them should receive what you type.
2. The program that decides this is the display server, or on modern Linux, the compositor.
3. It keeps a note of which window has focus, meaning which window is currently listening.
4. Focus changes when you click on a window, or use a keyboard shortcut to switch, or when a program asks for it and is allowed.
5. So the display server takes the key event from the operating system, looks at its focus note, and sends the event to that one program.
6. For a mouse click there is an extra step, because a click has a position. The server must work out which window is under that point, and then which part of that window.
7. Working out what is under a point is called hit testing.
8. Programs do not sit staring at their input. They sleep. The display server wakes them by delivering a message.
9. A program that is waiting for messages, handling them, and waiting again, is running an event loop. Almost every interactive program on earth has one.

PLAIN21.8.2 a picture in your head#

1. Picture a large open-plan office with a single receptionist and many desks.
2. Post arrives addressed to “the person currently working on the Henderson file”. The receptionist keeps a note of who that is.
3. The note changes during the day. When it changes, the receptionist tells the old person “you are no longer on it” and the new person “you are now on it”. Those are the focus-out and focus-in events, and programs really do get them.
4. Every worker has an in-tray on their desk and is asleep at their desk.
5. The receptionist drops a letter in a tray and taps the sleeper’s shoulder. They wake, read the whole tray, deal with everything, and go back to sleep.
6. That is the event loop: sleep, wake, drain the tray, act, sleep again.
7. A worker who takes twenty minutes on one letter is not asleep and not reading their tray. Their tray fills up and their window appears frozen. That is exactly what an unresponsive application is.

Where this comparison breaks: a real receptionist can be bypassed. On a modern compositor the routing is enforced, and one application cannot read another application’s keystrokes. On the old X11 design it could, which is a real security difference and one of the main reasons Wayland exists.

PLAIN21.8.3 a worked example#

1. Here is what an event loop actually looks like. This is pseudo-code, but it is very close to real code in every toolkit.

for (;;) {
    /* sleep until something happens */
    event = wait_for_next_event(connection);

    switch (event.type) {

    case KEY_PRESS:
        widget = focused_widget(window);
        text   = keymap_translate(event.keycode,
                                  event.modifiers);
        widget->on_key(widget, event.keycode, text);
        break;

    case BUTTON_PRESS:
        widget = hit_test(window, event.x, event.y);
        set_focus(widget);
        widget->on_click(widget, event.x, event.y);
        break;

    case RESIZE:
        relayout(window, event.width, event.height);
        break;

    case FRAME_CALLBACK:
        if (window->damaged)
            redraw(window);
        break;
    }

    if (window->damaged)
        request_frame_callback(window);
}

2. wait_for_next_event is where the program spends over 99 percent of its life. On Linux it is a poll or epoll_wait on a file descriptor.
3. Note there is no loop spinning and checking. The process is genuinely asleep and consumes no CPU until the kernel wakes it.
4. Note also that hit_test is called for the mouse and not for the keyboard. The keyboard goes straight to the focused widget.
5. And note the last two blocks. Drawing does not happen inside the key handler. The key handler only marks the window as needing redraw. The actual painting happens later, once, even if fifty keys arrived.

PLAIN21.8.4 what is really happening inside#

1. Four systems, four architectures. It helps to see them side by side.
2. X11. The X server owns the screen and all input. Applications are clients connected over a socket. The server sends a KeyPress event containing a keycode and a modifier mask.
3. In classic X11 the client then does the keymap lookup itself using XKB data fetched from the server. Any client could also ask to receive all key events through XGrabKeyboard or passive grabs, which is why keyloggers were trivial on X.
4. Wayland. There is no separate server. The compositor is the display server, the window manager and the compositor in one process.
5. At connect time the compositor sends the client the entire keymap as a file descriptor holding XKB text. The client maps it and uses libxkbcommon to translate.
6. Then wl_keyboard.key carries the raw evdev key code and a state, and wl_keyboard.modifiers carries the modifier state separately. Only the focused client gets them at all.
7. Quartz on macOS. The WindowServer process owns the screen. Events become CGEvents, then NSEvents. Each application has a CFRunLoop, and NSApplication pulls events from it.
8. Windows. Every GUI thread has a message queue. GetMessage blocks on it. TranslateMessage looks at a WM_KEYDOWN and, if the key maps to a character, posts an additional WM_CHAR with the character in it. DispatchMessage calls the window’s procedure.
9. That Windows split is worth remembering: WM_KEYDOWN is the physical key, WM_CHAR is the character, and they are two separate messages produced by one press.
10. In all four systems, the moment the event is handed over, a context switch happens: the compositor’s process stops running and the application’s process starts. That switch costs 1 to 10 microseconds when the machine is idle, and can cost milliseconds when it is busy.

TECHNICAL21.8.5 the engineer’s version#

1. History: X Version 11 Release 1 shipped on 15 September 1987 from MIT’s Project Athena, with Robert Scheifler and Jim Gettys as the principal authors of the protocol. The protocol is still wire-compatible today, which is nearly forty years of stability and also the source of most of its problems.
2. Wayland was started by Kristian Hoegsberg in 2008 while at Red Hat. Version 1.0 of the protocol was released in October 2012. GNOME and KDE both default to Wayland sessions on current distributions, and Fedora dropped the GNOME X11 session in Fedora 41 (2024).
3. Input handling on modern Linux normally goes through libinput, which owns pointer acceleration, tap-to-click, palm rejection and gesture recognition, so that every compositor behaves the same way. It reads /dev/input/eventN directly.
4. Security model comparison:

5. Wayland’s frame callback (wl_surface.frame) is the mechanism that ties drawing to the display refresh. A client that draws only on frame callbacks never renders frames that will not be shown, which saves power but does add the wait to the latency.
6. X11 uses an offset in key codes. An X keycode is the evdev code plus 8, for historical reasons dating to the original X server key code range. So evdev KEY_A (30) is X keycode 38.
7. Event delivery cost: on Linux with a Wayland compositor, the path from evdev read to the client’s socket becoming readable is typically 100 to 500 microseconds when idle. Under load with the compositor competing for CPU it can exceed 5 ms, which is why compositors run their input handling at elevated scheduling priority where they can.
8. Tools: wev prints Wayland events including the resolved keysym. xev does the same for X11. libinput debug-events --verbose shows the layer below. On Windows, Spy++ shows the message queue. On macOS, Quartz Debug and the CGEventTap API let you observe the stream.

WORDS21.8.6 remember these#

1. Display server — the program that owns the screen and input — the process holding the DRM master or equivalent and arbitrating input delivery.
2. Compositor — the thing that merges all windows into one image — on Wayland, the same process as the display server.
3. Focus — which window receives keyboard input — the keyboard focus target, tracked as state and changed by explicit focus events.
4. Hit testing — finding what is under a point — geometric resolution of a coordinate to a surface and then to a widget, respecting stacking and input regions.
5. Event loop — sleep, handle, sleep — the blocking dispatch loop at the heart of every interactive program, typically built on poll or epoll.
6. Frame callback — permission to draw for the next frame — a compositor notification synchronizing client rendering with display refresh.

21.9 The application reacts#

PLAIN21.9.1 in simple words#

1. The text editor is now awake with a key event in its hand.
2. It asks: which part of me has the keyboard right now? A window can have many text boxes, buttons and menus.
3. The answer is the focused widget, which in a text editor is the main text area.
4. The text area receives the event and asks the toolkit: what character is this, given the current layout and modifiers?
5. The answer comes back: the character A.
6. The editor then modifies its text. It has a big structure in memory holding every character of the document.
7. It inserts A at the cursor position, and then moves the cursor one place to the right.
8. That change may push words onto the next line, which means the text has to be laid out again from that point.
9. Laying out text means deciding, for every character, which picture to draw and exactly where on the screen to put it.
10. Finally the editor says “this rectangle of my window is now wrong, please ask me to redraw it”. It does not draw yet.

PLAIN21.9.2 a picture in your head#

1. Think of a typesetter in an old print shop, arranging metal letters in a frame.
2. You hand them a new letter to insert into the middle of a line.
3. They do not just drop it in. They push everything after it along.
4. If the line now runs past the edge of the frame, the last word has to move down to the next line, and that might push a word off that line too.
5. This cascade stops when a line still has room. Usually that is one or two lines. Occasionally, adding one letter to the first line of a page reflows the entire page.
6. The typesetter also knows that certain letter pairs sit badly next to each other. A capital A followed by a capital V looks better if they are pushed slightly closer. They adjust the spacing by hand.
7. That adjustment has a name and it is still used today: kerning.

Where this comparison breaks: the typesetter has one physical piece of metal per letter. Modern text has no fixed one-to-one relation between characters and shapes. One character can produce several shapes, several characters can produce one shape, and in some scripts the shapes get reordered relative to the characters.

PLAIN21.9.3 a worked example#

1. The document currently says Hello world and the cursor is after Hello, at index 5.
2. The editor stores the text. A simple editor uses a plain array, which is easy but means every insert copies everything after it.
3. Real editors use smarter structures.

4. The insert happens. The buffer now holds HelloA world and the cursor moves to index 6.
5. Now layout. The editor walks the paragraph and splits it into runs of text that share a font, a size, a direction and a script.
6. Each run goes to a shaping engine. The shaper takes characters and produces glyphs with positions.
7. For HelloA in a normal Latin font, the mapping is nearly one to one:

char 'H' -> glyph 43, advance 722 units
char 'e' -> glyph 72, advance 444 units
char 'l' -> glyph 79, advance 222 units
char 'l' -> glyph 79, advance 222 units
char 'o' -> glyph 82, advance 500 units
char 'A' -> glyph 36, advance 667 units

8. Those advances are in font design units. A typical font has 1000 or 2048 units per em. At 16 pixels per em with 1000 units, one unit is 0.016 pixels.
9. So the A occupies 667 * 0.016 = 10.67 pixels of width. Fractional. That fraction matters and we come back to it when we rasterize.
10. The editor now knows the pixel rectangle that changed, expands it a little for safety, and calls something like invalidate_rect(x, y, w, h).
11. Nothing has been drawn. One flag has been set and one rectangle recorded.

PLAIN21.9.4 what is really happening inside#

1. Let us be precise about characters, glyphs and shaping, because this is where most people’s mental model is wrong.
2. A character is a unit of text meaning. The letter A is a character. It has a Unicode code point, U+0041.
3. A glyph is a picture in a font. It has a number that is meaningful only inside that one font file.
4. The mapping between them is not one to one, and here are the four ways it breaks.
5. Ligatures. The characters f and i can be drawn as one joined shape. Two characters, one glyph.
6. Contextual forms. In Arabic, most letters have four shapes depending on whether they are alone, at the start, in the middle, or at the end of a word. One character, four possible glyphs.
7. Decomposition. In some Indic scripts one character is drawn as several marks placed around a base. One character, several glyphs.
8. Reordering. In Devanagari the vowel sign for i is stored after its consonant but drawn before it. The glyph order does not match the character order.
9. Shaping is the process that resolves all of this. It reads tables inside the font that describe substitutions and positioning rules.
10. Kerning is part of the positioning step. The font contains a table saying that when glyph A is followed by glyph V, shift V left by some units.
11. After shaping the editor has a list of glyph IDs, each with an x and y offset in fractional pixels. That is the layout result.
12. The layout is cached. Editing one line does not re-shape the whole document, which is why a fast editor stays fast on a large file.

TECHNICAL21.9.5 the engineer’s version#

1. Font format history: TrueType was created by Apple around 1991 and licensed to Microsoft. PostScript Type 1 came from Adobe in 1984. OpenType, announced by Microsoft and Adobe in 1996, unified them and added the GSUB and GPOS tables that make real shaping possible. OpenType became ISO/IEC 14496-22 as part of MPEG-4 in 2003.
2. Key OpenType tables in this path:

3. HarfBuzz is the dominant open-source shaping engine, used by Chrome, Firefox, Android, GNOME, LibreOffice and Qt. Its name is Persian for “open type”. Apple uses CoreText, and Microsoft uses DirectWrite with its own shaping engine, usp10’s successor.
4. The hb-shape command-line tool shows shaping output directly:

hb-shape --font-file=/usr/share/fonts/DejaVuSans.ttf \
         --unicodes=0048,0065,006C,006C,006F,0041

5. Text buffer structures, precisely. A gap buffer keeps a movable hole at the cursor, giving amortized O(1) inserts at the cursor and O(n) cursor jumps. A piece table keeps the original text immutable and stores an ordered list of spans into an original buffer and an append-only add buffer, which makes undo nearly free. Visual Studio Code moved to a piece tree, a balanced tree of pieces, in 2018 and documented the change publicly.
6. Damage tracking terminology differs by toolkit. GTK calls it gtk_widget_queue_draw, Qt calls it QWidget::update, Cocoa calls it setNeedsDisplay:, Win32 calls it InvalidateRect. All four mean the same thing: record a dirty region and return immediately.
7. Coalescing is the reason this design exists. If you hold a key and the system generates 30 repeats per second, but the display refreshes 60 times per second, drawing per event would be wasteful. Marking damage and drawing once per frame is strictly better.
8. Latency cost of this section, measured on a modern desktop with a warm cache: widget dispatch under 10 microseconds, buffer insert under 1 microsecond, shaping a single line 20 to 200 microseconds depending on script and cache state, damage marking under 1 microsecond.
9. The honest version: those are the good numbers. In a browser-based editor, this same section can take 5 to 50 milliseconds, because the key event goes through JavaScript, a virtual DOM diff, a style recalculation and a layout pass, and can be delayed by garbage collection. That is a real and measurable difference, and it is why native editors still feel sharper.

WORDS21.9.6 remember these#

1. Widget — one interactive part of a window — a node in the toolkit’s hierarchy with its own geometry, focus state and event handlers.
2. Character — a unit of meaning in text — a Unicode code point such as U+0041.
3. Glyph — a drawing in a font — an indexed outline within one font file, meaningless outside it.
4. Shaping — turning characters into positioned glyphs — the application of GSUB and GPOS rules by an engine such as HarfBuzz or CoreText.
5. Kerning — nudging two letters closer or apart — pairwise or class-based positioning adjustment from the font’s GPOS or legacy kern table.
6. Advance — how far to move before the next glyph — the horizontal advance width in font design units, scaled by point size.
7. Damage — the part of the window that is now wrong — the invalidated region recorded for repainting on the next frame.

21.10 Drawing#

PLAIN21.10.1 in simple words#

1. The editor now knows it must draw the letter A at a certain spot.
2. Inside the font, A is not a picture of dots. It is a description of its outline: a set of curves and straight lines forming the edge of the shape.
3. Turning that outline into coloured dots is called rasterizing.
4. The rasterizer works out, for every pixel, how much of that pixel falls inside the shape.
5. If a pixel is completely inside, it is fully dark. Completely outside, untouched. Half covered, half dark.
6. That partial darkness is anti-aliasing, and it is why text on a screen looks smooth instead of like a staircase.
7. There is a trick that goes further. Each pixel on most screens is actually three tiny lights side by side: red, green and blue.
8. If you treat those three as three separate samples instead of one, you get three times the horizontal detail. That is sub-pixel rendering.
9. Once the letter is drawn, the editor’s window is a finished rectangle of pixels sitting in memory.
10. Then the compositor takes that rectangle, and every other window’s rectangle, and stacks them into one single image the size of your screen.

PLAIN21.10.2 a picture in your head#

1. Imagine drawing a shape on graph paper and then colouring in the squares.
2. A square fully inside the shape gets fully coloured. A square fully outside stays white.
3. A square the edge passes through is the hard case. If you only allow black or white, the edge looks jagged.
4. So instead you shade it grey in proportion to how much of the square the shape covers. Seventy percent covered, seventy percent grey.
5. Step back and the eye blends the greys into a smooth edge. That is anti-aliasing in one sentence.
6. Now imagine each square on your graph paper is actually three thin vertical strips, coloured red, green and blue, that only look white together.
7. If the shape’s edge covers only the left strip, you darken only red. The eye still reads it as a slightly-left-shifted edge, at a third of a square’s precision.

Where this comparison breaks: darkening only the red strip really does tint the edge, and on a large low-density screen you can see coloured fringes on text. This is a genuine trade of colour accuracy for apparent sharpness, and it only works if the software knows the exact stripe order of your panel.

PLAIN21.10.3 a worked example#

1. Rasterize the letter A at 16 pixels per em from a font with 1000 units per em.
2. Scale factor is 16/1000 = 0.016 pixels per font unit.
3. The outline is a list of points. In TrueType the curves are quadratic Beziers, each with one control point. In CFF and PostScript fonts they are cubic, with two.
4. The rasterizer flattens each curve into short straight segments, fine enough that the error is under a fraction of a pixel.
5. It then sweeps down the shape one scanline at a time, finding where the outline crosses that line, and filling between crossings.
6. For the anti-aliased version it computes area coverage per pixel instead of a yes/no fill. The result is a small grey bitmap:

. . 3 9 9 3 . .        digits are coverage
. . 8 4 4 8 . .        0 = empty, 9 = full
. 3 9 . . 9 3 .
. 8 9 9 9 9 8 .        the crossbar of the A
3 9 . . . . 9 3
9 4 . . . . 4 9

7. Sub-pixel rendering triples the horizontal sampling first, producing three coverage values per pixel, then applies a filter across neighbouring sub-pixels to reduce colour fringing.
8. Those coverages are blended with the background. Doing that blend on the raw stored values is wrong, because screen values are gamma encoded. Correct blending happens in linear light, and getting it wrong makes text look too thin on dark backgrounds and too fat on light ones.
9. The glyph bitmap is cached in a glyph cache keyed by font, size, hinting mode and sub-pixel position. Typing a second A costs a cache lookup, not a rasterization.

PLAIN21.10.4 what is really happening inside#

1. There are two places the drawing can happen and modern applications use both.
2. On the CPU. The application writes pixel values into a buffer in ordinary memory using a library such as FreeType plus Cairo, Skia’s software backend, or CoreGraphics.
3. This is simple, predictable and works everywhere. It costs CPU time proportional to the number of pixels touched.
4. On the GPU. The application does not write pixels. It writes commands into a command buffer: bind this texture, draw these triangles, use this shader.
5. Glyphs are pre-rasterized into a big texture called a glyph atlas. Drawing a letter becomes drawing two triangles with the right texture coordinates.
6. The command buffer is submitted to a queue, and the GPU executes it independently. The CPU does not wait.
7. Either way the result is a buffer of pixels representing this one window.
8. Now the compositor. It has one buffer per visible window, plus a cursor, plus wallpaper, plus panels.
9. It composites: for each output pixel, work out which window is on top there, apply any transparency, scaling or rotation, and write the final colour.
10. This is almost always done on the GPU, because it is exactly the kind of massively parallel per-pixel work a GPU is built for.
11. There is a shortcut worth knowing. If a window is full-screen, opaque, and already in the right format, the compositor can skip compositing entirely and tell the display hardware to scan out that window’s buffer directly.
12. That is called direct scanout, and it removes one full copy and often one full frame of latency. It is why full-screen games are faster than windowed ones.

TECHNICAL21.10.5 the engineer’s version#

1. Hinting: TrueType fonts carry a bytecode program per glyph that moves points onto the pixel grid so that stems have consistent widths at small sizes. Apple’s patents on the TrueType bytecode interpreter expired in 2010, which is why FreeType enabled the interpreter by default from version 2.4 onward.
2. FreeType’s own autohinter is used when a font has no useful hints. Full hinting produces crisper but distorted shapes; the current preference on high-density displays is slight or no hinting, keeping shapes faithful.
3. Sub-pixel rendering: Microsoft announced ClearType at COMDEX in November 1998. The work was led by Bill Hill with Greg Hitchcock, and Microsoft published “Displaced Filtering for Patterned Displays” in May 2000. The idea itself is older; Apple’s 1976 Apple II used a related trick on colour artefacts, and IBM described sub-pixel addressing earlier still.
4. Sub-pixel rendering has three real limitations. It requires knowing the stripe order, which is RGB on most desktop LCDs and BGR on some. It breaks on a rotated display. And it is meaningless on OLED panels with non-striped sub-pixel layouts such as PenTile.
5. Apple removed sub-pixel anti-aliasing from macOS in Mojave (10.14, 2018), on the reasoning that Retina-class pixel densities make it unnecessary. This remains contested by people using non-Retina external monitors.
6. Signed distance field text, described by Chris Green of Valve at SIGGRAPH 2007 in “Improved Alpha-Tested Magnification for Vector Textures and Special Effects”, stores distance-to-edge instead of coverage, allowing one atlas entry to scale smoothly. It is widely used in games and in map rendering, and it is slightly less accurate for small body text.
7. Buffer sharing on Linux is by dma-buf file descriptors passed over the Wayland socket, so no pixel data is copied between processes. macOS uses IOSurface. Windows uses DXGI shared surfaces.
8. Compositing cost, order of magnitude at 2560x1440 on a mid-range 2023 GPU:

9. Tools: fc-list and fc-match for font selection; ftview and ftbench from FreeType; GALLIUM_HUD or mangohud for frame timing; weston-debug and drm_info for plane and scanout state; about:gpu in Chrome for its compositing decisions.

WORDS21.10.6 remember these#

1. Outline — the shape of a letter as curves — quadratic or cubic Bezier contours stored in glyf or CFF2.
2. Rasterize — turn a shape into pixels — scan conversion with area coverage computation.
3. Anti-aliasing — grey edges instead of jagged ones — using fractional coverage as an alpha value.
4. Sub-pixel rendering — using R, G and B as separate samples — horizontal tripling with a filter, as in ClearType.
5. Hinting — nudging outlines onto the pixel grid — bytecode or automatic grid fitting for stem consistency at small sizes.
6. Glyph atlas — a texture full of pre-drawn letters — a packed cache of rasterized glyphs for GPU text drawing.
7. Direct scanout — showing a window’s buffer with no compositing — assigning a client buffer straight to a display plane.

21.11 Out to the screen#

PLAIN21.11.1 in simple words#

1. The compositor has produced one image the size of your whole screen. That image is called the framebuffer.
2. A piece of hardware called the display controller reads that image out of memory, continuously, forever, whether or not anything changed.
3. It reads it in a fixed order: the top-left pixel first, then along the top row, then the next row, and so on to the bottom-right.
4. Then it starts again at the top. On a 60 Hz screen it does this 60 times a second. On a 240 Hz screen, 240 times.
5. Between finishing the bottom and starting the top again there is a short gap. That gap is called the vertical blanking interval.
6. If you swap in a new image during the gap, the screen shows one whole image at a time. If you swap in the middle, the top of the screen shows the old image and the bottom shows the new one, with a visible line between them. That is tearing.
7. So the system usually waits for the gap. That waiting is called vsync, and it is one of the largest single delays in this whole chapter.
8. The pixel values travel out of the cable, one after another, as fast electrical signals.
9. At the other end, the monitor has its own computer. It may need to resize the image, adjust the colours, and then drive the actual panel.
10. Finally, tiny amounts of liquid crystal twist, or tiny organic diodes emit, and light leaves the screen and reaches your eye.

PLAIN21.11.2 a picture in your head#

1. Picture a very fast painter repainting a fence, left to right, top plank to bottom plank, over and over.
2. The painter never stops. Even if nothing has changed, the whole fence is repainted 60 times a second.
3. You want to change the design. If you hand over the new design while the painter is halfway down the fence, the top half keeps the old design and the bottom half gets the new one. Half and half.
4. So you wait until the painter reaches the bottom and is walking back to the top. That walk back is the blanking interval. Hand over the design then and the whole fence changes together.
5. The cost of politeness is the wait. If you finish your design just after the painter starts a pass, you wait almost a whole pass.

Where this comparison breaks: the painter’s walk back is short, but the pass itself takes a full frame, and a change to the top of the screen appears before a change to the bottom of the same frame. That means the physical position of your text on the screen genuinely affects when you see it, by up to a whole frame time.

PLAIN21.11.3 a worked example#

1. Take 1920x1080 at 60 Hz over HDMI with standard CEA-861 timings.
2. The controller does not scan 1920x1080 pixels. It scans 2200 x 1125, including the blanking areas where no pixel is shown.
3. Total pixel clock: 2200 * 1125 * 60 = 148,500,000 pixels per second, which is the well-known 148.5 MHz pixel clock for 1080p60.
4. One full frame takes 1/60 second = 16.67 ms.
5. One scanline takes 16.67 ms / 1125 = 14.8 microseconds.
6. Your text is on line 400 of the screen. Once the frame starts scanning out, that line is transmitted at 400 * 14.8 us = 5.9 ms into the frame.
7. Now compare refresh rates:

8. The average wait is half the frame time because your frame becomes ready at a random point in the cycle.
9. That is only the wait. On top of it comes the scanout position, the panel’s own processing, and the pixel’s response time.
10. And if the system is triple buffered, your finished frame may sit behind one or two other finished frames, adding one or two whole frame times on top. At 60 Hz that is another 16.67 or 33.3 ms.

PLAIN21.11.4 what is really happening inside#

1. The display controller, called a CRTC on Linux, holds the memory address of the framebuffer, the timing numbers, and the format.
2. Changing which buffer is displayed is called a page flip. The driver programs the new address and the hardware latches it at the next blanking interval.
3. A page flip is atomic and cheap. Nothing is copied. Only a pointer changes.
4. The pixel data is serialized onto the cable. HDMI and DVI use TMDS, transition-minimized differential signalling, on three data lanes plus a clock. DisplayPort uses packetized micro-packets on one to four lanes with an embedded clock.
5. At the monitor, the received stream first meets a scaler if the incoming resolution does not match the panel’s native resolution. Scaling costs real time, often several milliseconds, and is a common hidden source of lag.
6. Then the timing controller, the TCON, converts the stream into the signals the panel needs, driving gate lines row by row and source lines column by column.
7. On an LCD, the backlight is always on. Each sub-pixel is a tiny cell of liquid crystal between polarizers, with a colour filter. Applying voltage twists the crystal, changing how much light passes.
8. The crystal takes time to twist. That is response time, and it is why a fast-moving object smears on some panels.
9. On an OLED, each sub-pixel emits its own light. There is no backlight and no crystal to twist, so response is under 0.1 ms.
10. Light then leaves the screen. At 60 cm, the photons take about 2 nanoseconds to reach your eye. That is two millionths of a millisecond, and it is the only stage in this chapter you can safely ignore.

TECHNICAL21.11.5 the engineer’s version#

1. Standards and dates: HDMI 1.0 in December 2002; DisplayPort 1.0 from VESA in 2006; HDMI 2.1 in November 2017 at 48 Gbit/s; DisplayPort 2.0 in 2019 and 2.1 in 2022, with UHBR20 giving 80 Gbit/s raw.
2. Variable refresh rate: NVIDIA G-SYNC shipped in 2013 with a hardware module in the monitor. VESA added Adaptive-Sync to DisplayPort 1.2a in 2014, and AMD’s FreeSync built on it from 2015. HDMI added VRR in HDMI 2.1 in 2017.
3. VRR changes the latency picture. Instead of the frame waiting for the display, the display waits for the frame, within a supported range such as 48 to 240 Hz. The average vsync wait largely disappears while the frame rate is inside that window.
4. Linux exposes all of this through DRM/KMS. The atomic modesetting API, merged in Linux 4.2 (2015), lets a compositor commit a whole new display state, including plane assignments, in one transaction.
5. Panel response times, honestly stated:

6. Marketing “1 ms” figures are usually the fastest single transition with overdrive at its most aggressive, not an average, and overdrive causes overshoot artefacts. Dan Luu measured that a monitor advertising 1 ms switching took roughly 10 ms to fully settle a character.
7. Display processing latency is a separate number from response time and is dominated by the scaler and any image processing. Televisions in normal picture modes commonly add 20 to 100 ms; game mode disables most of it and typically brings it under 10 ms.
8. Sample and hold: on an LCD or OLED the image is held static for the whole frame, so your eye, tracking a moving object, smears it across the retina. This persistence blur is independent of response time and is why backlight strobing exists.
9. Tools: modetest and drm_info show CRTC timings and plane usage; xrandr --verbose prints modelines including blanking; sudo powermetrics on macOS and PresentMon on Windows show present and display timings; the Blur Busters test patterns measure real panel behaviour.

WORDS21.11.6 remember these#

1. Framebuffer — the finished image in memory — a region of memory holding one full screen of pixels in a defined format and stride.
2. Scanout — the controller reading the image to the cable — continuous sequential transmission of the framebuffer at the pixel clock.
3. Vblank — the gap between frames — the vertical blanking interval, when no visible pixel is being transmitted.
4. Page flip — swapping which buffer is shown — an atomic change of the scanout address latched at vblank.
5. Tearing — two frames visible at once — the artefact of changing the scanout source mid-frame.
6. TCON — the panel’s own controller — the timing controller driving gate and source lines from the received video stream.
7. Response time — how long a pixel takes to change — the gray-to-gray transition time of the pixel element, distinct from display processing lag.

21.12 The complete time budget#

PLAIN21.12.1 in simple words#

1. Here is the whole chain in one picture. Read it from top to bottom. This is every stage in this chapter, in order.

              FINGER PRESSES THE KEY
                       |
                       |  2.0 mm of travel, 45 cN
                       v
   [1]  SWITCH CONTACT CLOSES     bounce for up to 5 ms
                       |
                       v
   [2]  KEY MATRIX     8 rows x 18 cols, diode per key
                       |  scanned 100 to 1000 times a second
                       v
   [3]  KEYBOARD MICROCONTROLLER
        debounce -> keymap lookup -> scan code 0x04
                       |  builds the 8-byte HID report
                       v
   [4]  USB ENDPOINT 1 IN (interrupt)
                       |  host polls every 1 ms or 125 us
                       v
   [5]  xHCI HOST CONTROLLER
        DMA into RAM -> event TRB -> MSI interrupt
                       |
                       v
   [6]  CPU TAKES INTERRUPT -> kernel ISR
        usbcore -> HID class driver -> input core
                       |  EV_KEY code 30 (KEY_A) value 1
                       v
   [7]  DISPLAY SERVER / COMPOSITOR
        focus lookup -> route to one client only
                       |
                       v
   [8]  TOOLKIT + KEYMAP
        key code + modifiers -> the character 'A'
                       |
                       v
   [9]  APPLICATION
        insert into text buffer -> move cursor
                       |
                       v
  [10]  TEXT LAYOUT
        itemize -> shape (GSUB/GPOS) -> positions
                       |
                       v
  [11]  RASTERIZE
        outline -> coverage -> anti-aliased pixels
                       |
                       v
  [12]  COMPOSITE
        all window buffers -> one framebuffer
                       |
                       v
  [13]  WAIT FOR VBLANK -> PAGE FLIP
                       |
                       v
  [14]  DISPLAY CONTROLLER SCANS OUT
                       |  148.5 MHz pixel clock at 1080p60
                       v
  [15]  CABLE -> SCALER -> TCON -> ROW/COL DRIVERS
                       |
                       v
  [16]  SUB-PIXELS CHANGE STATE
                       |  ~2 ns of flight over 60 cm
                       v
                  PHOTONS -> EYE

2. Now the numbers. Two columns: a carefully built fast setup, and an ordinary badly configured one.
3. The clock starts at the moment the switch closes, not when your finger starts moving.

PLAIN21.12.2 a picture in your head#

1. Think of a relay race with sixteen runners, each handing a baton to the next.
2. The total time is not decided by how fast the runners are. Most of them are very fast indeed.
3. It is decided by the four or five runners who stand still and wait for a whistle before setting off.
4. In our chain the whistles are: the matrix scan, the USB poll, the frame callback, and vsync.
5. Making the fast runners faster changes almost nothing. Making the whistles more frequent changes everything.

Where this comparison breaks: some of our runners genuinely are slow, not just waiting. A browser-based editor doing a layout pass, or a television doing image processing, is real work taking real time, and no amount of scheduling fixes it.

PLAIN21.12.3 a worked example#

1. The good column assumes: eager debounce, 1000 Hz matrix scan, 1 ms USB polling, a lightly loaded machine, a native application, direct scanout, a 240 Hz monitor with no scaling, and a fast panel.
2. The bad column assumes: 5 ms symmetric debounce, 100 Hz scan, a low-speed USB keyboard polled every 10 ms, a busy machine, a browser-based editor, triple buffering, a 120 Hz refresh, and a television doing image processing.
3. The ratio between them is about fifteen to one. Nothing in that table is exotic. Both configurations are extremely common.
4. Notice where the milliseconds actually are in the bad column. Queued frames at 33.3 ms, panel processing at 20.0 ms, and the application itself at 15.0 ms. Those three are more than half the total.
5. Notice what is almost free in both columns. The USB transfer, the DMA, the interrupt and the kernel. All the parts people worry about most are the parts that cost least.

PLAIN21.12.4 what is really happening inside#

1. It is important to compare that table with real measured numbers, because a sum of estimates is not evidence.
2. Dan Luu, in 2017, measured keyboards alone with a logic analyser on a cut USB cable, from the key starting to move to the USB packet leaving.

3. Those are much larger than the first six rows of our table, which sum to about 2 ms in the good case. Why?
4. Because his clock starts when the key starts moving, not when the contact closes. The finger’s own travel to the actuation point is several milliseconds and it is genuinely part of the experience.
5. And because real keyboards use conservative debounce and modest scan rates. A 35 ms gaming keyboard is not a measurement error. It is firmware choosing safety over speed.
6. Dan Luu also measured whole systems, key press to the character finishing on screen, with a high-speed camera.

7. A 1983 Apple IIe beats a 2017 desktop by a factor of nearly seven. That is the single most useful fact in this chapter.
8. For gaming setups, NVIDIA’s Reflex tooling reports click-to-photon system latency. TFTCentral, testing an Asus ROG Swift PG259QNR in 2021, measured about 13 ms of PC-plus-display latency at 360 Hz with Reflex on, plus 0.8 ms of mouse latency, for 13.8 ms total, versus about 26 ms at 60 Hz with Reflex off.
9. So our “good” column of 8.85 ms is optimistic but the right order of magnitude, and our “bad” column of 134 ms sits comfortably inside the range of real measured systems.

TECHNICAL21.12.5 the engineer’s version#

1. Sources of the buffering penalty, which is the largest single controllable item:
2. Triple buffering with a queue depth of 3 lets the GPU render ahead by two frames. Throughput improves, latency worsens by up to two frame times.
3. Dan Luu computed this from his own data: going from 24 Hz to 165 Hz gave a 90 ms improvement, whereas pure vsync mathematics predicts only about 18 ms. The residual implies roughly 2.5 frames of buffering, independent of refresh rate, on that Windows machine running PowerShell.
4. Mitigations that actually work, with their real mechanism:

5. NVIDIA Reflex, announced 1 September 2020, works by having the driver and the game agree to delay the start of CPU work so that the CPU finishes just in time for the GPU, instead of queueing frames ahead. It is a back-pressure mechanism, not a rendering speed-up.
6. Measurement methods and what each one actually includes:

7. Those four methods do not measure the same thing and their results are not comparable. Most arguments about input lag on the internet are two people quoting different methods at each other.
8. Tools: PresentMon on Windows for present-to-display timing; LatencyFlex and Reflex counters in supported games; libinput measure for device-side timing; a 1000 fps phone camera, which most phones now have, for a genuine end-to-end check that requires no trust in software.

WORDS21.12.6 remember these#

1. End-to-end latency — the total wait from press to light — motion-to-photon or click-to-photon latency, measured externally.
2. Queue depth — how many finished frames wait in line — the maximum frames in flight, typically 1 to 3, trading latency against throughput.
3. Back pressure — making the producer wait for the consumer — the technique behind Reflex and anti-lag features.
4. Click to photon — the gamer’s version of the same measurement — latency from a switch event to a measured luminance change at the panel.
5. Jitter — the variation in the wait, not the average — the spread of latency across repeated events, often more noticeable than the mean.

21.13 The same trace for a mouse and a touchscreen#

PLAIN21.13.1 in simple words#

1. A mouse click follows almost the same road. Switch, debounce, matrix or direct pin, microcontroller, USB HID, kernel, compositor, application.
2. Two things differ. First, the mouse also carries movement, and movement comes from a camera, not a switch.
3. Under an optical mouse there is a tiny camera and a light. It photographs the surface thousands of times a second.
4. It compares each photo with the one before, works out how far the pattern shifted, and reports that shift.
5. So a mouse never reports where it is. It only reports how far it moved since last time. The position on screen is the computer adding those up.
6. Second, a click has a place. The system must work out what is under the pointer before it can deliver the click.
7. A touchscreen is different again. It reports an actual position on the glass, not a movement.
8. Under the glass is a grid of transparent wires. Your finger changes the electrical capacity where it touches, and the controller finds the centre of that change.
9. The controller then works out whether you tapped, held, dragged, or made a gesture with two fingers, and reports that.

PLAIN21.13.2 a picture in your head#

1. Imagine walking across a field in fog with a torch and a camera pointed at the ground.
2. Every fraction of a second you photograph the grass. You cannot see where you are in the field, but you can see that this photo is shifted two centimetres left of the last one.
3. Add up all the shifts and you have a path. That is an optical mouse, and it is also why a mouse loses tracking on glass: no grass, nothing to compare.
4. Now imagine instead a giant chessboard where every square reports when somebody stands on it. You do not need to add anything up. You are told where you are.
5. That is a touchscreen. Absolute, not relative.

Where this comparison breaks: the chessboard squares are coarse and a touchscreen is not. The controller looks at how strongly several neighbouring squares respond and computes a weighted centre, giving a position much finer than the spacing of the wires.

PLAIN21.13.3 a worked example#

1. Real optical mouse sensor numbers, from the Agilent ADNS-3080 datasheet of 2005, a design still widely copied:

2. At 6469 frames per second, one frame takes 155 microseconds. That is the sensor’s own contribution to latency and it is tiny.
3. DPI, properly called CPI or counts per inch, is how many counts the sensor reports per inch of movement. At 1600 CPI, moving one inch produces 1600 counts.
4. Higher CPI is not “more accurate”. It is finer granularity. If the sensor’s noise is larger than one count, extra CPI just reports noise more precisely.
5. The USB HID boot mouse report is three bytes:

byte 0 : buttons, bit 0 left, bit 1 right, bit 2 middle
byte 1 : X movement, signed 8-bit, negative is left
byte 2 : Y movement, signed 8-bit, negative is up

6. Left click with no movement gives 01 00 00. Release gives 00 00 00. Moving right 5 and down 3 with no buttons gives 00 05 03.
7. Signed 8-bit caps movement at 127 counts per report. At 1000 Hz polling and 1600 CPI, that caps out at about 79 inches per second, which is enough. At 125 Hz polling it would cap at about 10 inches per second, which is not. That is why real mice use the report protocol with 12 or 16-bit deltas.
8. Click versus drag is decided in software, not hardware. A press, then movement beyond a threshold before release, is a drag. A press and release inside the threshold, within a time limit, is a click.
9. Mouse switches bounce too, and a worn Omron D2FC-F-7N switch is the usual cause of a mouse that double-clicks by itself. It is the debounce window failing to hide a degraded contact.

PLAIN21.13.4 what is really happening inside#

1. Optical mouse tracking, step by step. An LED or a laser illuminates the surface at a shallow angle so that microscopic texture casts shadows.
2. The sensor captures a small grey image. The processor cross-correlates it with the stored previous image, testing candidate shifts, and picks the shift with the best match.
3. Sub-pixel interpolation of the correlation peak gives finer resolution than the pixel spacing. The result is a pair of integers, dx and dy.
4. Frame rate is usually variable. The sensor speeds up when moving fast and slows down when still, to save power.
5. Touchscreen, step by step. Projected capacitive screens have two layers of transparent conductor, usually indium tin oxide, in rows and columns.
6. The controller drives one row with a signal and measures the coupling into each column. Your finger steals some of that coupling.
7. This produces a small grid of numbers, a capacitance image, typically at 60 to 240 scans per second.
8. The controller finds connected regions above a threshold, computes each region’s weighted centre, and assigns each a tracking identifier so that a finger keeps its identity across frames.
9. Then it decides what the touches mean: a tap, a long press, a swipe, a pinch, a rotation. Some controllers do this themselves; on phones, most of it is done by the operating system.
10. Palm rejection, edge rejection and moisture handling all live here too, and they add processing time.
11. Coordinates are reported as absolute values in a device coordinate space, then scaled into screen coordinates. This is the deepest difference from a mouse: no accumulation, no acceleration curve, no pointer to lose.

TECHNICAL21.13.5 the engineer’s version#

1. History: Douglas Engelbart and Bill English built the first mouse at SRI in the mid-1960s; the patent, US 3,541,541, “X-Y Position Indicator for a Display System”, was granted in 1970, and the public demonstration on 9 December 1968 is remembered as the Mother of All Demos.
2. The first optical mice needed a special printed mat. Surface-independent optical tracking arrived with Agilent’s sensors and the Microsoft IntelliMouse with IntelliEye in 1999.
3. Capacitive touch sensing was described by E. A. Johnson at the Royal Radar Establishment in 1965. Multi-touch projected capacitive reached the mass market with the iPhone in 2007.
4. HID reporting: mice use Generic Desktop usages X (0x30) and Y (0x31) with Relative set. Touchscreens use the Digitizer usage page (0x0D) with Absolute coordinates, contact identifiers and a contact count, defined by the Windows Precision Touchscreen and Precision Touchpad specifications.
5. On Linux, touch arrives as multitouch protocol B events: ABS_MT_SLOT, ABS_MT_TRACKING_ID, ABS_MT_POSITION_X, ABS_MT_POSITION_Y, grouped by SYN_REPORT. Protocol A, the older form without slots, still exists on some hardware.
6. Polling rates and scan rates in current products:

7. Dan Luu’s 2017 scroll-latency measurements put the 2017 iPad Pro 10.5 inch at 70 ms with a finger and 30 ms with the Pencil, against 110 to 150 ms for most Android phones of the same period. The Pencil’s advantage comes from the higher scan rate and a shorter dedicated path.
8. Pointer acceleration is a software transfer function mapping counts per report to pixels, and it is a genuine source of disagreement. Games disable it and read raw counts; desktops keep it. libinput implements it in userspace, which is why acceleration behaviour became consistent across Linux desktops after 2015.
9. Tools: sudo libinput debug-events shows both relative and absolute events; evtest shows raw multitouch slots; xinput list-props shows acceleration settings; the mousetester utility plots real polling intervals and reveals a mouse whose 1000 Hz claim is not honoured.

WORDS21.13.6 remember these#

1. Relative input — reporting movement, not position — delta reporting, as from a mouse or trackball, accumulated by the host.
2. Absolute input — reporting a position directly — coordinate reporting, as from a touchscreen or tablet digitizer.
3. CPI / DPI — counts reported per inch moved — sensor resolution, commonly 400 to 26,000, independent of accuracy.
4. Cross-correlation — matching one photo against the last — the block-matching step that converts two images into a displacement.
5. Projected capacitance — sensing a finger through glass — mutual capacitance measurement across a transparent row/column electrode grid.
6. Tracking identifier — the number that says “this is the same finger” — the slot identity that keeps contacts distinguishable between scans.

21.14 Why this matters#

PLAIN21.14.1 in simple words#

1. It is tempting to say that a few milliseconds cannot matter, because a millisecond is a thousandth of a second.
2. That turns out to be wrong, and we can show it with measurements rather than opinions.
3. In a shooting game, the enemy you are aiming at is where they were, plus however long your whole chain took.
4. If your chain is 100 ms and theirs is 20 ms, they see the truth 80 ms earlier than you do. At normal movement speeds that is a body width.
5. In music, the same problem shows up as feel. You hit a drum pad and hear the sound late. Your hands compensate by playing early, and the timing drifts.
6. In plain typing, a slow chain does not stop you working, but it does make the machine feel unresponsive, and there is evidence that it makes people less accurate.
7. The honest part is this: for most people, most of the delay is not in the keyboard. It is in the screen and in the software.
8. Buying a faster keyboard when your monitor adds 40 ms of image processing is spending money in the wrong place.

PLAIN21.14.2 a picture in your head#

1. Think of a conversation over a satellite phone with half a second of delay.
2. Nothing is broken. Every word arrives perfectly. Yet the conversation is exhausting, because you cannot interrupt, agree or laugh at the right moment.
3. Now think of the same delay applied to your own hand. You move it and see it move half a second later. You would immediately feel that something was badly wrong.
4. Your body has an extremely tight expectation for the delay between an action and its result, because that loop is how you learned to move at all.
5. Input lag is that same violation, just smaller. Small enough to be unnamed, large enough to be felt.

Where this comparison breaks: satellite delay is constant and you adapt to it. Input lag on a computer varies from press to press, and variation is harder to adapt to than a constant offset. A steady 40 ms often feels better than a wandering 20 to 60 ms.

PLAIN21.14.3 a worked example#

1. NVIDIA Research ran an aiming study and reported it in 2020 alongside Reflex.
2. They compared PCs at 12 ms and at 20 ms of system latency, an 8 ms difference.
3. The average difference in the time taken to acquire and shoot a target was 182 ms. That is roughly 22 times the latency difference.
4. On a 128-tick server, that is about 23 ticks earlier that your shots land.
5. NVIDIA also stated that most gamers play on systems with 50 to 100 ms of system latency, which puts the 12 ms machine in a different world.
6. Now music. MIDI over the original 5-pin cable runs at 31,250 bits per second. A three-byte note-on message with start and stop bits is 30 bits, so about 960 microseconds.
7. A digital audio interface with a 128-sample buffer at 48 kHz adds 128/48000 = 2.67 ms per buffer, and there are usually at least two, in and out.
8. Real round-trip latency on a well-configured system is 5 to 12 ms. Above roughly 10 ms most players start to feel the instrument is not under their fingers.
9. A useful check: sound travels about 343 metres per second, so 10 ms is 3.4 metres. A 10 ms delay is the same as standing three and a half metres back from your amplifier, which musicians do all the time without complaint. That is why 10 ms is roughly the boundary rather than 1 ms.

PLAIN21.14.4 what is really happening inside#

1. Where do the milliseconds actually go, honestly, for a normal person on a normal machine?
2. The keyboard contributes 15 to 60 ms depending on the model, mostly in debounce and scan rate, per Dan Luu’s 2017 measurements.
3. The USB layer contributes 0.5 to 5 ms, and almost never more.
4. The kernel contributes well under 1 ms unless the machine is loaded.
5. The application contributes anything from 0.3 ms for a native editor to tens of milliseconds for a browser-based one.
6. The compositor and buffering contribute one to three frame times, which at 60 Hz is 16.7 to 50 ms.
7. The display contributes the vsync wait plus its own processing plus pixel response, commonly 10 to 40 ms, and far more on a television not in game mode.
8. So the honest ranking, for a typical desktop, is: display and buffering first, application second, keyboard third, and everything else a rounding error.
9. That is why the 1983 Apple IIe beat a 2017 desktop. It had no compositor, no buffering, no process boundaries, and a display that was written to directly.
10. It is also why the fix is usually free. Turning on game mode, disabling a compositor’s extra buffering, using a native application, and raising the refresh rate cost nothing and beat any hardware purchase.

TECHNICAL21.14.5 the engineer’s version#

1. Human perception limits, from published work rather than folklore:
2. Albert Ng, Paul Dietz and colleagues, in “Designing for Low-Latency Direct-Touch Input” at UIST 2012, built a touch system with about 1 ms of latency and found that people could discriminate latencies down to approximately 2 ms in direct-touch dragging tasks.
3. That result directly contradicts the widely repeated Nielsen Norman claim that anything under 100 ms feels instantaneous. The 100 ms figure describes perceived system responsiveness for discrete actions, not the sensorimotor loop of continuous input.
4. Where experts disagree: whether polling rates above 1000 Hz produce a human-detectable improvement. Manufacturers say yes and sell 8000 Hz products. Independent blind testing has not established a consistent benefit, and the arithmetic caps the possible gain at about 0.44 ms of average wait. Treat “8K polling” claims as marketing until a blind study says otherwise.
5. Established fact: total system latency measurably affects task performance in aiming, per NVIDIA’s published study and per earlier work in human factors going back to Sheridan and Ferrell’s teleoperation studies in 1963.
6. Active research: perceptual thresholds for latency in VR and AR, where motion-to-photon budgets under 20 ms are considered necessary to avoid sickness, and where John Carmack’s 2013 essay “Latency Mitigation Strategies” set much of the agenda.
7. Practical budget for a competitive setup, achievable in 2026 with ordinary parts:

8. Tools that let you argue from data: a 1000 fps phone camera for end-to-end; PresentMon for the present pipeline; Reflex or LDAT counters for click-to-photon; cyclictest to check whether your machine’s scheduling latency is the real problem.

WORDS21.14.6 remember these#

1. Input lag — the delay between acting and seeing the result — end-to-end system latency across the full input and display pipeline.
2. Motion to photon — the VR name for the same thing — the interval from head or hand movement to the corresponding light emission.
3. Sensorimotor loop — the body’s action-and-result cycle — the closed control loop the nervous system uses, sensitive to delays of a few milliseconds.
4. Round-trip audio latency — in through the microphone and out the speaker — the sum of input buffer, processing and output buffer, typically 5 to 12 ms.
5. Tick rate — how often a game server updates the world — the server simulation frequency, commonly 64 or 128 Hz, adding its own delay on top.

21.98 Common wrong ideas#

1. Wrong: the keyboard sends the letter A. Right: it sends usage code 0x04 on usage page 0x07, which names a physical position. The letter is decided by the operating system’s keymap, several stages later.
2. Wrong: a key press is a single clean electrical event. Right: the contact bounces for up to 5 ms on a Cherry MX switch, and firmware must filter that into one event.
3. Wrong: a 104-key keyboard needs 104 wires to its chip. Right: an 8 by 18 matrix needs 26 pins, and the chip scans the grid repeatedly rather than watching all keys at once.
4. Wrong: “anti-ghosting” and “N-key rollover” mean the same thing. Right: anti-ghosting often means the firmware suppressing ambiguous combinations, whereas true N-key rollover needs a diode per key and a HID report format that can carry more than six keys.
5. Wrong: a USB interrupt transfer means the device interrupts the computer. Right: the device can never speak first. The host polls it on a fixed schedule, and the name is historical.
6. Wrong: the CPU is stopped in the middle of an instruction by the keyboard. Right: the keyboard’s data is written into RAM by the host controller using DMA, and the CPU takes the resulting interrupt at an instruction boundary.
7. Wrong: an 8000 Hz polling rate is eight times better than 1000 Hz. Right: it removes at most about 0.44 ms of average wait, which is small next to the frame and buffering costs that dominate the chain.
8. Wrong: one character equals one glyph. Right: ligatures merge two characters into one glyph, Arabic gives one character four contextual shapes, and Devanagari reorders glyphs relative to characters.
9. Wrong: a monitor advertising 1 ms response has 1 ms of latency. Right: response time is only the pixel transition. Display processing, scaling and the vsync wait are separate and usually much larger.
10. Wrong: new computers respond faster than old ones. Right: Dan Luu measured a 1983 Apple IIe at 30 ms end to end and a 2017 Windows desktop at 200 ms. Raw speed went up and latency went up with it, because of buffering and process boundaries.

21.99 Chapter summary in 20 lines#

1. A key switch is only a gate for electricity; a Cherry MX Red closes after 2.0 mm of a 4.0 mm travel at 45 cN and bounces for under 5 ms.
2. Switch families differ in how they close: membrane domes, mechanical leaves, scissor domes, optical beams, and Hall-effect magnets that report depth as a number rather than as on or off.
3. Firmware must debounce, and that choice costs between 0 and 8 ms depending on whether the algorithm fires eagerly or waits for stability.
4. Keys are wired in a row and column matrix so a few pins can read many keys, scanned 100 to 1000 times a second, costing half a scan period on average.
5. Without a diode per key, three pressed keys can create a false fourth, which is ghosting; diodes give true N-key rollover.
6. The keyboard contains a small computer that turns a matrix position into a scan code through a lookup table, and that code is a position, not a letter.
7. USB HID’s boot keyboard report is exactly 8 bytes: a modifier bitmap, a reserved byte, and six key code slots; Shift plus A is 02 00 04 00 00 00 00 00.
8. The host polls the keyboard’s interrupt endpoint every 1 ms at full speed or every 125 microseconds at high speed, so the report waits half a period on average.
9. Bluetooth keyboards use HID over GATT with a negotiated connection interval in 1.25 ms units, which is the main reason they feel slower.
10. The xHCI host controller DMAs the report into RAM, posts an event TRB, and raises an MSI interrupt; the CPU takes it at an instruction boundary.
11. The kernel’s USB core hands the bytes to the HID class driver, which uses the parsed report descriptor to find fields and emits EV_KEY with KEY_A, code 30, through evdev.
12. The keymap, not the keyboard, decides the character, using a group axis for layout and a level axis for modifiers, with dead keys and compose sequences as small state machines on top.
13. Scripts with more characters than keys need an input method editor, a separate program with a dictionary that sends pre-edit text and then a commit.
14. The display server routes the event to the focused window only, and every interactive program is a loop that sleeps, wakes, drains events and sleeps again.
15. The application inserts the character, moves the cursor, re-shapes the affected line into positioned glyphs, and marks a rectangle as damaged without drawing anything yet.
16. Rasterizing turns an outline into per-pixel coverage; anti-aliasing uses that coverage as transparency, and sub-pixel rendering treats the red, green and blue stripes as three horizontal samples.
17. The compositor merges all window buffers into one framebuffer, unless it can hand a single window’s buffer straight to the display hardware.
18. The display controller scans that framebuffer out continuously, so a new frame waits for the vertical blanking interval, costing half a frame time on average plus any queued frames.
19. A carefully built chain totals under 10 ms from contact to light; an ordinary badly configured one totals over 130 ms, and real measurements put a 1983 Apple IIe at 30 ms against a 2017 desktop at 200 ms.
20. The milliseconds live in the display, the buffering and the application, not in the wire, so measure before you spend, and remember that NVIDIA measured an 8 ms latency difference producing a 182 ms difference in aiming time.