The Bitmap

25.0 What this chapter gives you#

1. You will be able to read a sixteen-character hexadecimal string such as F23C448128E09200 and say, without a tool, which data elements the message carries.
2. You will be able to explain why bit 1 is not a field at all, and why getting it wrong shifts every subsequent byte in the message by eight or sixteen.
3. You will be able to tell the three wire forms of the same sixty-four bits apart — raw binary, ASCII hexadecimal and EBCDIC hexadecimal — by inspecting the first sixteen bytes of a link.
4. You will be able to write the receive path for an ISO 8583 message in about twenty lines, and say why it needs no tokeniser, no state machine and no backtracking.
5. You will be able to read the notation n 6, n..19 and ans...999 for what it says about length, and read b 64 as eight bytes rather than sixty-four.
6. You will be able to diagnose a misaligned parse from its symptom — a card number of all zeroes, an amount with two extra digits, a MAC failure — because there will be no error message to work from.
7. You will be able to say why sixty-four bits is exactly optimal in the worst case, gives up about a quarter against an ideal coder on real traffic, and beats a list of field numbers from about the tenth field onward.
8. You will be able to explain why an unknown field is survivable in Protocol Buffers and fatal in ISO 8583, and why that one property makes adding a field to a card network a certification programme rather than a deployment.
9. You will be able to use a bitmap as an instrument: classify traffic before decoding any field, and name the exact difference between a request and its response with a single exclusive-or.
10. You will be able to state where the bitmap runs out — 128 slots, no versioning, no room for meaning — and why the industry answered with nested structures inside elements 48, 62, 63, 124 and 127 rather than with the tertiary bitmap.

There is a small piece of engineering at the front of every card message on earth that almost nobody outside the payments industry has heard of, and that almost nobody inside it can explain properly. It is eight bytes long. It contains no card number, no amount, no merchant, no date, and no information about the transaction at all. What it carries is information about the message — specifically, which of the message’s sixty-four possible fields the sender bothered to fill in.

It is called the bitmap. It was designed in the mid-1980s for terminals with kilobytes of memory talking over telephone lines at 1,200 bits per second. Those constraints are gone. The bitmap is not. It is still there, at the front of authorisations approved in 2026, doing the same job in the same eight bytes, and the reason is not inertia. The reason is that for the problem it solves, sixty-four bits is very close to the smallest number of bits that could possibly work.

This chapter takes those eight bytes apart. By the end of it you will be able to look at a hexadecimal string in a log file and say, without a tool, which fields are in the message.

The plain version#

Imagine a class of sixty-four children who always sit in the same order — the same sixty-four chairs, the same sixty-four names, every day, for forty years. Nobody ever moves.

Every morning the class hands in homework, and there is a rule: nobody writes their name on it. The teacher collects the pile, the pile is in seat order, and that is the only thing identifying whose is whose. Most days most of the class has not done any, so on a typical morning maybe twenty children hand something in and the pile is twenty sheets deep, in seat order, with no names.

How does the teacher know the third sheet belongs to child eleven and not child nine?

Because at the top of the pile there is a register. Not a list of names — a row of sixty-four boxes, one per seat, in seat order. A tick means “this child handed something in”. A blank means “skip this seat”. The teacher reads the register first, learns that seats 2, 3, 4, 7, 11, 12, 13, 14, 18, 22, 25, 32, 35, 37, 41, 42, 43, 49, 52 and 55 have ticks, and deals the pile out in that order. Twenty sheets, twenty owners, no names written anywhere.

That row of ticks is the bitmap. Everything in this chapter follows from it.

Ticks are expensive; symbols are cheap#

Writing out sixty-four boxes takes a lot of space, so we compress it with a trick that takes five minutes to learn. Take the ticks four at a time. Four boxes can be filled in sixteen different ways: all blank, only the last one, only the third one, and so on up to all four. Invent sixteen symbols, one per pattern, and write one symbol instead of four boxes. The symbols are 0 to 9 and then A, B, C, D, E, F. This is called hexadecimal, and the whole of it fits in a small table.

There is a pattern that makes this easy without the table. Left to right, the four boxes are worth 8, 4, 2 and 1; add up the ticked ones. Blank, tick, tick, blank is 4 plus 2, which is 6. Tick, tick, tick, blank is 8 plus 4 plus 2, which is 14, and 14 is E. That is the entire system.

Sixty-four boxes, four to a symbol, is sixteen symbols. So the register — the thing that tells you the shape of an entire card payment — is sixteen characters long. That is shorter than most people’s email address.

One real register, decoded by hand#

Here is a real one, from a chip-and-PIN purchase:

F23C448128E09200

Decode it left to right, four boxes per symbol, keeping a running count of which seat we are on.

Twenty-one seats are ticked: 1, 2, 3, 4, 7, 11, 12, 13, 14, 18, 22, 25, 32, 35, 37, 41, 42, 43, 49, 52 and 55.

Now put the names back. Seat 2 is the card number, 3 is what kind of transaction, 4 is the amount, 7 is when it was sent, 11 is the serial number. Seats 12 and 13 are the shop’s own clock and calendar; 14 is the card’s expiry; 18 is what kind of shop; 22 is how the card was read; 25 is anything unusual; 32 is which bank is asking; 35 is the data copied off the card; 37 is a reference number. Seats 41, 42 and 43 are the till, the merchant and the merchant’s name and town. Seat 49 is the currency, 52 is the encrypted PIN, and 55 is everything the chip said.

You have just read a card payment out of sixteen characters. You know it was a chip transaction with a PIN, in a shop, from a real till, with a real merchant name attached. You have not seen a single value yet.

Seat one is not a child#

Look at that list again. Seat 1 is ticked, but nothing in the description above is seat 1, because seat 1 is not a child. It is a note that says there is a second register. Sixty-four seats was enough in 1985 and it was not enough by 1990, so the designers reserved the very first tick to mean “another sixty-four boxes follow, covering seats 65 to 128”. If seat 1 is blank, the register is sixteen characters and that is all. If seat 1 is ticked, the register is thirty-two characters, and the second half describes a second class of sixty-four children who are only occasionally in the room.

In our example the second register is:

0000000000000001

Fifteen zeroes and a one. Read the same way, the final 1 is the last box of the last group of four, which is seat 128 — the cryptographic seal proving the message was not altered on the way. So the full register for this payment is thirty-two characters:

F23C448128E09200 0000000000000001

and it declares twenty-two things: bit 1 saying “look at the second half”, the twenty fields of the transaction, and the seal at the end.

Some homework is one page and some is forty#

There is one more problem the register does not solve. It tells you whose homework is in the pile. It does not tell you how many pages each piece is.

For most seats it does not matter, because the answer is fixed and everyone knows it. Seat 4, the amount, is always exactly twelve digits. Seat 11, the serial number, is always exactly six. Seat 37 is always exactly twelve characters. You never have to ask. You count.

But some fields cannot be fixed. A card number can be thirteen digits or nineteen. A merchant’s extra notes might be four characters or four hundred. For those, the field carries its own page count on the front. A card number arrives as the two digits 16 followed by sixteen digits. Chip data arrives as the three digits 120 followed by 120 bytes. Two digits of length at the front is called LLVAR, three digits is LLLVAR, and that is the whole naming convention: one letter L per digit of length, then the word “variable”.

So the reader’s job is a loop with no cleverness in it at all. Read the register. For each ticked seat, in order: if it is fixed, take exactly that many characters; if it is variable, read two or three digits of length first, then take that many. Move on. Stop when the ticks run out.

What happens when the register is wrong#

Now the thing that makes this design frightening. Suppose the register ticks seat 25, but the child in seat 25 did not hand anything in. The teacher takes the next two characters and calls them seat 25’s homework — but those two characters were the first two of seat 32’s. Seat 32 is now short by two and reads two characters of seat 35. Seat 35 is variable length, so the teacher now reads a length out of the middle of somebody else’s data, and that length might say 87 when 200 characters remain, or it might say 04.

Every field after the mistake is wrong. Not missing. Wrong. And nothing in the message says so, because there is no name on any of the homework to contradict the register: the whole point was that we stopped writing names to save space.

That is the bargain. The bitmap buys the smallest possible message and the fastest possible reader, and the price is that a single wrong bit does not produce an error where the error is. It produces a plausible-looking message full of nonsense, and the nonsense surfaces somewhere else entirely — often as an amount with two extra zeroes on it.

Where the plain version stops being true#

The register is not a register of children, and the numbering is self-referential. In the standard’s own numbering the bitmap is data element 1, so bit 1 of data element 1 indicates the presence of a second instance of data element 1. That is not a curiosity: it is why you will see published field tables listing element 1 as b 16 and others listing it as b 64, describing the same object. One is counting hexadecimal characters, the other bits, and neither says which. The same ambiguity infects every binary element in the catalogue. When a specification says b 64 it almost always means sixty-four bits, which is eight bytes, which is sixteen hexadecimal characters. Reading it as sixty-four bytes is a classic first-week mistake.

Sixteen characters and eight bytes are both true, and they are not the same number of bytes on the wire. The plain version treats the bitmap as “sixteen characters”. That is one of at least three encodings in real use. The bitmap can travel as eight raw binary bytes, whose individual bits carry the meaning directly, or as sixteen printable characters in ASCII, or as sixteen printable characters in EBCDIC on IBM mainframe links. All three carry identical information, occupy eight, sixteen and sixteen bytes respectively, and look nothing like each other on the link. A parser configured for the wrong one is not slightly wrong; it is catastrophically wrong from the first byte.

A tick tells you a field is present. It does not tell you where it is. The bitmap is not an index. There is no offset table, no pointer, no random access. To find data element 55 you must parse elements 2 through 43 first, because several of them are variable-length and you cannot know where 55 begins until you have measured everything before it. The bitmap makes the message small and the parse deterministic. It does not make the message seekable.

The bitmap says a field exists. It never says what the field means. Bit 62 set tells a parser that data element 62 is present and, from the local field table, how long it is. It says nothing about the contents, because 62 is reserved for private use and is defined by whoever you happen to be connected to. A parser holding the wrong counterparty’s field table will read a syntactically valid message and produce semantically wrong output, silently. Worse: for a private element whose length rule the parser does not have, it cannot even skip the field, because it does not know how many bytes to skip. This is the deep limitation of positional encoding, and no care in the bitmap fixes it.

The technical version#

What the standard specifies#

ISO 8583 defines the message as a message type identifier, followed by one or more bitmaps, followed by the data elements that the bitmaps declare, in ascending order of element number. The third edition, published in July 2023, describes the bitmap component as “one or two message bitmaps, each consisting of 64 bits”, in which “each bit signifies the presence (1) or the absence (0) in the message of the data element associated with that particular bit”.

Three properties follow from that sentence and they are the whole design. The bitmap is fixed width: sixty-four bits always, whether the message carries three data elements or fifty, so a bitmap needs no length prefix of its own. It is positional: bit n corresponds to data element n, with no lookup, no translation table and no negotiation, the correspondence being a property of the numbering itself. And it is ordering: the data elements follow in ascending numerical order, so the bitmap answers “which fields” and “in what sequence” with the same sixty-four bits. That is why no separators are needed anywhere in the message.

Bit numbering, and the convention people get wrong#

Bits are numbered from 1 to 64, left to right, starting at the most significant bit of the first byte. Bit 1 is 0x80 of byte 0, bit 8 is 0x01 of byte 0, bit 9 is 0x80 of byte 1. Formally, for element n the byte index is (n - 1) / 8 and the mask is 0x80 >> ((n - 1) % 8).

This is big-endian bit ordering within the byte, and it is the most common source of a first implementation that almost works. A developer whose instinct is to test byte & (1 << bit) — least significant bit first, the convention used almost everywhere else in computing — will produce a parser that reverses every group of eight, and the failure is not obvious, because the reversed set is still a plausible-looking set of field numbers. Byte 0xF2 correctly declares elements 1, 2, 3, 4 and 7; read least-significant-bit-first it declares 2, 5, 6, 7 and 8. Both look like field numbers. Only one is right.

One bitmap, decoded completely#

Take the primary bitmap from earlier — a 0100 authorisation request for a contact chip transaction with online PIN — and expand it fully, in binary, with the element number each bit governs written alongside:

byte 0  F2  1111 0010   elements  1  2  3  4 |  5  6  7  8
byte 1  3C  0011 1100   elements  9 10 11 12 | 13 14 15 16
byte 2  44  0100 0100   elements 17 18 19 20 | 21 22 23 24
byte 3  81  1000 0001   elements 25 26 27 28 | 29 30 31 32
byte 4  28  0010 1000   elements 33 34 35 36 | 37 38 39 40
byte 5  E0  1110 0000   elements 41 42 43 44 | 45 46 47 48
byte 6  92  1001 0010   elements 49 50 51 52 | 53 54 55 56
byte 7  00  0000 0000   elements 57 58 59 60 | 61 62 63 64

Set bits: 1, 2, 3, 4, 7, 11, 12, 13, 14, 18, 22, 25, 32, 35, 37, 41, 42, 43, 49, 52, 55.

Bit 1 is set, so a secondary bitmap follows immediately, before data element 2.

00 00 00 00 00 00 00 01

The only set bit is the last bit of byte 7, which is bit 64 of that bitmap, which is data element 128.

On the wire, then: MTI, primary bitmap, secondary bitmap, and then elements 2, 3, 4, 7, 11, 12, 13, 14, 18, 22, 25, 32, 35, 37, 41, 42, 43, 49, 52, 55 and 128. Four are variable length — the primary account number in element 2 (n..19), the acquiring institution identification code in 32 (n..11), the track 2 data in 35 (z..37) and the integrated circuit card data in 55. The rest are fixed: n 12 for the amount in 4, n 6 for the system trace audit number in 11, an 12 for the retrieval reference number in 37, ans 8 for the terminal identification in 41, b 64 for the PIN block in 52 and for the message authentication code in 128.

Element 55 is worth a warning. Published tables disagree: the widely circulated 1987 catalogue lists it as ans ...999, while scheme-facing references describe it as binary with a three-digit length, which is what it is in every EMV deployment — a concatenation of EMV tag-length-value objects, which are binary and are not character data. Take the counterparty’s specification, not the general table.

Going the other way#

To construct a bitmap, start with eight zero bytes and set one bit per element: element 2 sets 0x40 in byte 0, element 39 sets 0x02 in byte 4, element 64 sets 0x01 in byte 7. Two published bitmaps make good test vectors. 7010001102C04804 expands to elements 2, 3, 4, 12, 28, 32, 39, 41, 42, 50, 53 and 62 — a response carrying a card number, a processing code, an amount, a local time, a transaction fee, an acquirer identifier, a response code, a terminal, a merchant, a settlement currency, security-related control information and one private field. 7234054128C08000 expands to elements 2, 3, 4, 7, 11, 12, 14, 22, 24, 26, 32, 35, 37, 41, 42 and 49 — a magnetic stripe authorisation from an era when element 24 still carried a network international identifier. If your parser reproduces both, your bit arithmetic is right.

The secondary bitmap#

The secondary bitmap covers data elements 65 to 128. It is present if and only if bit 1 of the primary bitmap is set, and when present it sits immediately after the primary bitmap, before any data element value.

The consequence is that the secondary bitmap is not optional in the way other fields are optional. Its presence or absence changes the byte offset of every subsequent field in the message by eight or sixteen bytes. Getting bit 1 wrong is the highest-blast-radius single-bit error in the whole format.

Elements 65 to 128 hold what a purchase does not need: settlement and reconciliation totals in 74 to 89, the original data elements of a reversal in 90, the file-action elements in 91 to 94, the settlement and receiving institution identifiers in 99 and 100, and private-use space in 120 to 127. Element 96 holds a message security code and 128 a message authentication code.

A 0400 reversal request typically sets element 90, “Original data elements”, a forty-two digit fixed field holding the original message’s MTI, system trace audit number, transmission date and time, and acquiring and forwarding institution identification codes, concatenated. That is how a reversal names the transaction it is reversing when there is no other shared identifier. A reversal carrying element 90 and a MAC in 128 has a secondary bitmap of:

00 00 00 40 00 00 00 01

Byte 3 is 0x40, whose second bit is bit 26 of the secondary bitmap, which is element 90. Byte 7 is 0x01, element 128.

The tertiary bitmap#

Later editions extend the element range to 192, with a tertiary bitmap covering 129 to 192 signalled by element 65. The 1987 catalogue lists element 65 as b 1, an extended bitmap indicator — a single bit whose whole purpose is to say that a third bitmap follows.

You will not see one. Every reference that discusses it describes it as extremely rare or effectively unused on card rails, because the schemes solved the same problem a different way, by defining private elements that contain their own internal structure. Mastercard’s private data subelements inside data elements 48, 124 and 127, and Visa’s proprietary subfields inside 48, 62 and 63, are nested tag-length-value structures with their own subelement identifiers. They gave the industry unbounded extension space without ever touching the top-level bitmap.

Three wire forms of the same sixty-four bits#

The bitmap can be represented as eight bytes of binary data or as sixteen hexadecimal characters in ASCII or EBCDIC. This is not an implementation detail. It is the first thing you must establish about any link, and it is not carried in the message. The same primary bitmap, in all three:

The second row is the sixteen characters F23C448128E09200 written out one byte each. The third is the same sixteen characters in the encoding IBM mainframes use, where the digit 0 is 0xF0 rather than 0x30.

Now the failure. Suppose a parser expects raw binary and receives the ASCII hexadecimal form. It reads the first eight bytes, 46 32 33 43 34 34 38 31, as a bitmap — in binary 01000110 00110010 00110011 01000011 00110100 00110100 00111000 00110001 — and declares elements 2, 6, 7, 11, 12, 15, 19, 20, 23, 24, 26, 31, 32, 35, 36, 38, 43, 44, 46, 51, 52, 53, 59, 60 and 64.

Twenty-five elements, including the MAC in element 64. Bit 1 is clear, so the parser concludes there is no secondary bitmap. Nothing about that set is implausible on its face. The parser then reads twenty-five fields out of what are in fact the remaining eight characters of the bitmap followed by the message body, and produces a card number that is not a card number, an amount that is not an amount, and a confident structured object that is entirely fiction. This is why the encoding is the first question on any new link, and why “it parses” is not evidence that it parses correctly.

Fixed and variable length data elements#

Every data element has a content type — a alphabetic, n numeric, s special characters, an, as, ns and ans for the combinations, b binary, z for track 2 and track 3 code set per ISO/IEC 7813 and ISO/IEC 4909, x+n for a signed amount, p for pad — and a length rule.

The length rule is written after the type. A bare number means fixed: n 6 is exactly six digits, always, and a sender with fewer must pad. Leading dots mean variable, one dot per digit of the length prefix: n..19 is LLVAR, a two-digit length then up to nineteen digits; ans...999 is LLLVAR, a three-digit length then up to 999 characters. A single dot, n.6, is LVAR, which exists in the notation and is essentially unused. For binary elements the declared number is bits, not bytes, so b 64 is eight bytes — a constant source of confusion, made worse by tables that use the same notation for character counts.

Most of the catalogue is fixed length, and of the elements that carry a normal card purchase only 2, 32, 35 and 55 are variable. That is deliberate: fixed fields need no length prefix and no bounds arithmetic, and can be copied with a constant offset and a constant count.

LLVAR and LLLVAR on the wire#

This is the part of ISO 8583 that produces the most interoperability defects, because the notation n..19 specifies the semantics of the length prefix and says nothing about its encoding. There are four independent decisions, and each is a place two implementations can differ.

How the length digits are encoded. ASCII digits use one byte per digit, so an LLVAR prefix is two bytes and an LLLVAR prefix three. Packed binary-coded decimal uses one byte per two digits, so an LLVAR prefix is one byte and an LLLVAR prefix two, with a leading zero nibble. The length 27 is 0x32 0x37 in ASCII and 0x27 in BCD. A parser expecting the wrong one is off by one or two bytes at the front of every variable field.

Whether the length is binary or decimal. Some specifications encode the length as a plain unsigned integer rather than decimal digits, making 27 0x1B. The pyiso8583 library, which exists to model this variety, treats length encoding as a configuration axis separate from data encoding for this reason.

What the length counts. For character data it counts characters, which is also bytes. For packed BCD numeric data it usually counts digits, and the field occupies half that many bytes. For binary data it usually counts bytes. Get this wrong on a packed field and you read half the field or twice it. Serious libraries make it explicit: pyiso8583 offers a len_count setting whose values are bytes and nibbles, with bytes as the default, alongside left- and right-padding options for odd-length packed data. That such an option has to exist tells you how much disagreement is out there.

What happens at the maximum. n..19 means up to nineteen. It does not mean a sender may send twenty. Scheme specifications routinely tighten the maxima below the ISO values, so a message that is valid ISO 8583 may still be rejected.

LLVAR and LLLVAR are therefore not one thing but a family of closely related things, and the specification that governs your link — not the ISO document, not a reference website — is the only authority on which member of the family you are speaking.

The parser loop#

Once the encodings are settled, the entire receive path is about twenty lines.

read MTI (4 characters)
read primary bitmap (8 bytes, or 16 characters if hex-encoded)
if bit 1 set:
    read secondary bitmap (same size)
for n from 2 to 128:
    if bit n is not set: continue
    look up definition[n]
    if definition[n] is fixed:
        take definition[n].length units
    else:
        take definition[n].prefix_digits, decode as length L
        take L units
    store as element n
assert no bytes remain

There is no tokeniser, no state machine, no backtracking, no recursion, no dynamic allocation if the buffers are pre-sized, and no ambiguity about whether a value is a string or a number, because the definition already said. That is why a terminal with sixteen kilobytes of RAM could do it in 1987 and a switch handling tens of thousands of messages per second can do it now without a garbage collector running.

The last line is the only error detection the format gives you for free, and it is worth more than it looks. If the bitmap and the body agree, the parse consumes exactly the message and stops; if they do not, there are bytes left over or the parser runs off the end. Implementations that silently tolerate trailing bytes throw away the one cheap integrity check they had.

Why sixty-four bits is close to optimal#

The bitmap solves a narrow problem: communicate an arbitrary subset of sixty-four known items, as compactly as possible, decodable in constant time by a machine with no memory to spare. It is worth being precise about how good the answer is.

The worst case is exactly optimal. There are 2 to the power 64 possible subsets of sixty-four elements, so distinguishing all of them requires at least 64 bits by counting alone. The bitmap uses exactly 64. No encoding that can express every subset is smaller in the worst case, and the bitmap meets the bound with equality. That is not “close to optimal”, it is optimal, for the problem as stated.

The typical case gives up about a quarter. Real authorisations carry around seventeen to twenty-one elements, not arbitrary subsets. The information content of an arbitrary seventeen-of-sixty-four subset is the base-two logarithm of sixty-four choose seventeen, which is 50.3 bits, or 6.3 bytes. The bitmap spends 8. The overhead against a cardinality-aware ideal coder is about 27 per cent, or 1.7 bytes, and you would pay for that saving with an arithmetic coder in the hot path of every transaction on earth.

The obvious alternative loses above nine fields. The intuitive design is to send a list of the field numbers present, and a field number in the range 1 to 128 fits in seven bits. Seven bits per present field beats a 64-bit bitmap only below about nine fields: at nine the list costs 63 bits and the bitmap costs 64, and from ten onward the bitmap wins and never stops winning. Add a one-byte count so the receiver knows when the list ends and the crossover moves to exactly eight. A card authorisation carrying twenty-one elements would need 147 bits, nineteen bytes, as a field list, against eight as a bitmap.

Against a modern self-describing format the gap is about two to one. Protocol Buffers encodes each present field with a varint tag holding the field number and wire type, which is one byte for field numbers up to 15 and two bytes for 16 to 2047. The twenty-one elements of our example would cost 34 bytes of tags against 16 bytes for the two bitmaps, on framing overhead alone.

Against a fixed layout the gap is three orders of magnitude. The other way to avoid labels is to give every field a fixed position whether it is used or not. Summing the maximum lengths of data elements 2 to 64 gives 12,733 characters, before the secondary range. Even excluding the twelve elements defined as ...999 — the additional-data and reserved fields — the remainder is 745 characters. A real authorisation whose two bitmaps and twenty-one populated elements come to 351 bytes, with the bitmaps 4.6 per cent of that, makes a fixed layout not a design but a joke about telephone bills.

And sixty-four is the right number for the machine, not just the message. A 64-bit bitmap is one register on any processor built since the mid-1990s. Testing presence is a single AND. Counting fields is a single population-count instruction. Comparing two messages’ field sets is a single XOR. In 1987 this meant two 32-bit words rather than one; the committee did not choose 64 because it was a machine word, but it has aged into being one, which is a large part of why nobody has managed to justify replacing it.

The comparison that actually matters#

All of the above is about size, and size is the least interesting axis. The real difference between a bitmap and a tagged format is skippability.

In Protocol Buffers, JSON, XML or ASN.1 BER, a receiver that meets a field it has never heard of can skip it, because the wire type or the delimiter says how many bytes to step over. Unknown fields are survivable and forward compatibility is free. In ISO 8583 a receiver that meets a field it has no definition for cannot skip it, because the number of bytes to skip is in the definition it does not have. It cannot parse the rest of the message either. There is no recovery.

This single property explains an enormous amount about how the card industry behaves. It is why adding a field to a card network is a multi-year certification programme rather than a deployment, why schemes publish mandates eighteen months ahead, and why private data lives inside elements 48, 62, 63, 124 and 127 as nested tag-length-value structures — because inside a ...999 wrapper whose length the parser does know, tags become skippable again, and the schemes rebuilt forward compatibility one level down where the bitmap could not reach. It is also a large part of the argument for ISO 20022, which is self-describing from top to bottom and pays for it in bytes.

Parsing failure modes#

These occur in production, and every one presents as a misaligned parse rather than a clean error. That is the point.

Wrong bitmap encoding. Binary read as ASCII hex or the reverse. Off by eight bytes for the whole message. Diagnosed by checking whether the first sixteen bytes all fall in 30 to 39 and 41 to 46 (ASCII) or F0 to F9 and C1 to C6 (EBCDIC); if they do, it is a hex-encoded bitmap.

Reversed bit order within bytes. Least-significant-bit-first testing. The declared field set is a plausible-looking but wrong permutation: F2 gives 1, 2, 3, 4, 7 correctly and 2, 5, 6, 7, 8 incorrectly. Diagnosed by decoding a known-good bitmap such as 7010001102C04804 and checking you get 2, 3, 4, 12, 28, 32, 39, 41, 42, 50, 53, 62.

Bit 1 set with no secondary bitmap sent. The parser consumes the first eight bytes of data element 2 as a secondary bitmap. If element 2 begins with the ASCII characters 16411111, it reads a secondary bitmap declaring elements 67, 68, 72, 75, 76, 78, 79, 83, 84, 86, 91, 92, 96, 99, 100, 104, 107, 108, 112, 115, 116, 120, 123, 124 and 128 — twenty-five phantom fields, several in private-use space — and tries to read them all out of the transaction body. The most destructive bitmap defect there is, and common in hand-rolled senders.

Bit 1 clear with a secondary bitmap sent. The mirror image. The parser reads the secondary bitmap’s eight bytes as the value of the first declared element. Because a secondary bitmap is usually mostly zeroes, the classic symptom is a card number of all zeroes.

A bit set for a field the sender did not write. Every subsequent field shifts left by the missing field’s length. Fixed fields absorb the shift silently; the first variable field afterwards reads its length prefix from the middle of somebody else’s data, and the parse either overruns the buffer or ends with bytes remaining.

A field written without its bit set. The parser stops early, or reads the unannounced bytes as the next declared field. The trailing-bytes assertion catches it, if the implementation has one.

Length prefix encoding mismatch. ASCII versus BCD on LLVAR and LLLVAR. Off by one or two bytes per variable field, cumulative: four variable fields drift by four to eight bytes by the end.

Length counting mismatch. Digits versus bytes on packed numeric fields. Reads half the field or twice it. The symptom is a PAN of the right length with the wrong digits, or half the expected length.

Scheme-tightened maxima. Valid against the ISO catalogue and rejected by the counterparty. This one at least fails loudly, usually with a format error — response code 30 in the 1987 two-character response code set.

Private element with no local definition. Unrecoverable: the parser cannot determine the length and cannot proceed. The correct behaviour is to reject the whole message rather than guess.

MAC failure standing in for a parse failure. Data elements 64 and 128 carry a message authentication code computed over the message, so any change to the bitmap changes the MAC. A bitmap defect on a MAC-protected link therefore presents first as an authentication failure rather than a parse error, which is confusing until you realise the MAC is doing exactly what it should.

The through-line is that positional encoding has no redundancy. Nothing in the byte stream says “this is the amount”. The claim that byte 47 is the amount lives entirely in the bitmap and the field table, and if either is wrong the message will still parse, still validate, and still be wrong. Every mature payments organisation ends up with the same three defences: assert that the parse consumes the message exactly, range-check every field against its declared type before using it, and keep a golden set of known-good messages that new builds must reproduce byte for byte.

The bitmap as a diagnostic instrument#

Once you can read a bitmap you get several things for free.

You can classify traffic without parsing it. A production switch can read eight bytes and know whether a message carries chip data, a PIN block or a MAC, and whether it is a stripe transaction — routing, fraud-screening and capacity-planning information available before any field is decoded.

You can diff two messages in one instruction. Exclusive-or the two primary bitmaps and the set bits in the result are exactly the elements that differ in presence. Our 0100 request and a corresponding 0110 response, whose primary bitmap is 703800010E808200, differ by 8204448026601000, which expands to elements 1, 7, 14, 18, 22, 25, 35, 38, 39, 42, 43 and 52. The response has dropped the card data and the terminal context and added elements 38 and 39, the authorisation identification response and the response code. The entire logical difference between a request and its answer, in one exclusive-or.

You can specify conformance precisely. Scheme certification is largely a statement about bitmaps: for this message type, these bits must be set, these must not, these are conditional on those. A test plan is, at bottom, a list of required and forbidden bit patterns, which is why it automates so well. And you can read a log over somebody’s shoulder — an engineer who decodes the characters after the MTI on sight can tell in seconds whether a failing message is malformed at the top level or merely carrying a value the issuer did not like, which is usually the first useful question in an incident.

Where the bitmap runs out#

The bitmap’s limits are structural, not incidental, and they are the whole case for the format that will eventually replace ISO 8583.

There are 128 slots, most spoken for by the ISO catalogue and the rest private-use fields that different counterparties have filled with different things. There is no way to add a 129th concept without either the tertiary bitmap nobody implements or a nested structure inside an existing element. The industry chose nesting, which works, but which means the top-level bitmap no longer describes the message — it describes the outer envelope, and the interesting content is inside element 48 or 55 or 127, indexed by a second-level scheme the bitmap knows nothing about.

There is no versioning. The bitmap does not say which field table it was built against. The MTI’s first digit names an edition of the standard, but the standard no longer contains the field definitions — in the third edition they were moved out into a separately maintained register. Two systems can agree on the bitmap and disagree on what element 62 is, and nothing in the message will tell them.

And there is no room for meaning. A bit can say “present”. It cannot say “present but unreliable”, or “present, populated by a stand-in processor rather than the issuer”. Every richer statement becomes a value inside another field, which is how element 48 came to contain dozens of subelements in some scheme specifications.

None of that stops the bitmap working. As of writing, in 2026, the overwhelming majority of card authorisations on earth still begin with a four-digit MTI whose first digit says 1987 and then eight bytes of flags, and they will for years yet. It is a forty-year-old trick that is provably optimal for the problem it was given, and the problem it was given was smaller than the one the industry now has. That gap, not any defect in the bitmap, is what the rest of this volume is about. The next chapter takes the fields the bitmap declares and says what each of them actually holds.

25.98 Common wrong ideas#

Wrong: b 64 in a field table means sixty-four bytes. Right: it means sixty-four bits, which is eight bytes, which is sixteen hexadecimal characters, and reading it the other way is a classic first-week mistake.

Wrong: bit 1 declares data element 1. Right: bit 1 declares a second bitmap covering elements 65 to 128, and its presence or absence moves the byte offset of every subsequent field in the message.

Wrong: the bitmap is an index, so a parser can jump straight to data element 55. Right: it declares presence and order and nothing else; several earlier elements are variable length, so you must measure 2 through 43 before you know where 55 begins.

Wrong: test a bit with byte & (1 << bit), as everywhere else in computing. Right: bits are numbered from the most significant bit of the first byte, so element n is 0x80 >> ((n - 1) % 8) in byte (n - 1) / 8; the least-significant-first version yields a plausible-looking but wrong set of field numbers.

Wrong: the bitmap is sixteen characters on the wire. Right: that is one of three encodings in real use, alongside eight raw binary bytes and sixteen EBCDIC characters, and nothing in the message says which one you are receiving.

Wrong: if the message parses, the parse is correct. Right: positional encoding has no redundancy, so a wrong bitmap or a wrong field table produces a complete, valid-looking, entirely fictional object; only asserting that the parse consumes the message exactly catches it.

Wrong: a parser can skip a private field it does not recognise. Right: the number of bytes to skip lives in the definition it does not have, so it can neither skip the field nor parse the rest of the message, and the correct behaviour is to reject the whole thing.

Wrong: LLVAR means a two-byte ASCII length prefix. Right: n..19 fixes the semantics of the prefix and says nothing about its encoding, which may be ASCII digits, packed binary-coded decimal or a binary integer, and which may count bytes or nibbles.

Wrong: the tertiary bitmap is how you get past 128 elements. Right: it is described everywhere as effectively unused on card rails, and the schemes extended instead by nesting tag-length-value structures inside elements they already had.

Wrong: a message that is valid ISO 8583 will be accepted. Right: scheme specifications routinely tighten the maxima below the ISO values, so a valid message can still come back with a format error.

25.99 Chapter summary in 20 lines#

1. Every ISO 8583 message begins with eight bytes that say nothing about the transaction and everything about the message: which of its sixty-four possible fields the sender filled in.
2. A set bit means the corresponding data element is present, a clear bit means it is absent, and the declared elements follow in ascending numerical order.
3. Because the bitmap gives both the set of fields and their sequence, the message needs no field names, no separators and no offsets anywhere.
4. Sixty-four bits is sixteen hexadecimal characters, and reading them four bits at a time, worth 8, 4, 2 and 1, is the whole skill.
5. The primary bitmap F23C448128E09200 declares elements 1, 2, 3, 4, 7, 11, 12, 13, 14, 18, 22, 25, 32, 35, 37, 41, 42, 43, 49, 52 and 55, which is a chip-and-PIN purchase in a shop, read without seeing a single value.
6. Bit 1 is not a data element: it announces a secondary bitmap covering elements 65 to 128, which sits immediately after the primary bitmap and before any value.
7. Getting bit 1 wrong is the highest-blast-radius single-bit error in the format, because it shifts every subsequent field by eight or sixteen bytes.
8. The bitmap says which fields are present; the field table says how long each is, and fixed lengths are simply counted while variable ones carry a two- or three-digit length prefix.
9. The receive path is therefore a loop of about twenty lines with no tokeniser, no state machine, no backtracking and no allocation, which is why a terminal with sixteen kilobytes of memory could run it in 1987.
10. Its one free integrity check is that the parse must consume the message exactly, and implementations that tolerate trailing bytes throw that check away.
11. The same sixty-four bits travel as eight raw binary bytes, sixteen ASCII characters or sixteen EBCDIC characters, and the encoding is the first thing to establish on any new link because the message does not carry it.
12. A parser configured for the wrong encoding is not slightly wrong; it produces a confident, well-formed and entirely fictional transaction from the first byte.
13. Bits are numbered from the most significant bit of the first byte, and the usual least-significant-first instinct yields a plausible but wrong field set that looks like field numbers either way.
14. For the problem as stated the bitmap is exactly optimal in the worst case, costs about 27 per cent more than an ideal cardinality-aware coder on real traffic, and beats a list of field numbers from roughly the tenth field onward.
15. Against Protocol Buffers the framing gap is about two to one, and against a fixed layout with every field always present it is three orders of magnitude.
16. The property that actually matters is not size but skippability: a tagged format can step over a field it has never heard of, and ISO 8583 cannot, because the length is in the definition the receiver lacks.
17. That single property explains multi-year certification programmes, mandates published eighteen months ahead, and the whole practice of nesting private data inside elements 48, 62, 63, 124 and 127 where lengths are known.
18. Every bitmap defect presents as a misaligned parse rather than a clean error, with symptoms such as a card number of all zeroes, an amount with extra digits, or a MAC failure standing in for a parse failure.
19. Once read fluently the bitmap becomes a diagnostic instrument: it classifies traffic before any field is decoded, and an exclusive-or of two bitmaps names exactly the elements by which a request and its response differ.
20. The bitmap’s limits are structural — 128 slots, no versioning, and no way to say anything richer than “present” — and that gap, rather than any defect in the design, is the case for whatever eventually replaces it.

Chapter sources: ISO 8583:2023, third edition, July 2023, on the message bitmap component, via the publicly available standard preview; the ISO catalogue entry for ISO 8583-1:2003; the Wikipedia ISO 8583 article for the 1987 data element catalogue, the format and length notation tables, the worked bitmap 7010001102C04804 and the 1987 response code table; the ISOParser ISO 8583 reference for the field catalogue with LLVAR and LLLVAR annotations, the worked bitmap 7234054128C08000, the note that binary element lengths are expressed in bits, and Visa BASE I/BASE II and Mastercard IPM private-field practice; the pyiso8583 v4.0.1 specifications manual for length encoding, length-count in bytes versus nibbles and odd-length packed padding; the moov-io/iso8583 bitmap documentation for bitmap sizing; and the IBM App Connect ISO8583 sample documentation for the rarity of the tertiary bitmap. All bitmap expansions, byte counts, encoding tables and size, crossover and information-content calculations were computed from those definitions and verified against the published worked examples.