How Proto Parsing Works

This page explains replay parsing as a pipeline of transformations.

Goal: start with a .dem replay file and end with structured events, entities, and match metadata.

You do not need to know bit math to follow this. We will focus on concepts first.

If you want a quick binary-format foundation first, read Bits & Bytes Primer.

What protobuf is

Almost everything in a replay is encoded with Protocol Buffers ("protobuf"), Google's format for sending structured data compactly. Both sides share a schema, written in a .proto file. Here is part of one from demo.proto:

message CDemoFileHeader {
    required string demo_file_stamp = 1;
    optional int32 patch_version = 2;
    optional string map_name = 5;
}

The number after each field is its tag. The encoded bytes contain only tags and values, never field names. A reader needs the same .proto file to know that tag 5 means map_name. It works like a form with numbered boxes and the legend printed separately.

This has three consequences for replay parsing:

• It is compact. Numbers instead of names keep a replay's millions of messages small.
• Old readers keep working. A reader skips tags it doesn't know. Valve adds fields over time, and gem's generated code still reads new replays; it just doesn't see the new fields until the definitions are refreshed.
• Messages don't say what they are. The bytes of a CDemoFileHeader don't contain the name CDemoFileHeader. Something outside the message has to say which type it is. In a replay, that is the numeric command on each envelope (Stage 1) and the type_id on each inner message (Stage 3).

protoc, the protobuf compiler, turns each .proto file into Python classes:

from gem.proto.demo_pb2 import CDemoFileHeader

header = CDemoFileHeader()
header.ParseFromString(payload)  # payload: bytes known to be a file header
header.map_name

The Proto Files gem Uses explains where gem's 84 .proto files come from and which ones matter.

The mental model

Think of a replay as nested containers:

1. A file-level stream of outer messages.
2. Some outer messages contain a byte blob with many inner messages.
3. Most inner messages are protobuf payloads with named fields.
4. A few inner fields (especially entity deltas) contain custom bit-packed data that is not protobuf.

So parsing is not one decoder. It is a chain of decoders.

The pipeline at a glance

Replay .dem bytes
  -> outer framing decoder
  -> outer protobuf decode (CDemo*)
  -> inner message unpack ({type, size, payload} repeated)
  -> inner protobuf decode (CSVC*/CNET*/game events/user messages)
  -> subsystem handlers (string tables, entities, combat log, events)
  -> structured match outputs

Stage 1: Outer framing (container layer)

Think of a replay as a long stack of envelopes. Stage 1 reads the label on each envelope without opening it.

The file starts with a 16-byte header, before the first envelope:

Clarity uses the first offset to jump straight to the end-of-file summary. Manta and gem skip both, and a truncated replay shows why: its header still points at envelopes that were cut off.

After the header, the envelopes follow one after another. Each one is:

command (varuint32) | tick (varuint32) | size (varuint32) | payload (size bytes)

• command says what kind of envelope this is: an EDemoCommands value from demo.proto. If bit 64 (DEM_IsCompressed) is set, the payload is Snappy-compressed.
• tick is when the envelope was recorded. The game runs at 30 ticks per second. The value 0xFFFFFFFF means "before the game started", and gem stores it as 0.
• size is the payload length in bytes.

binary/stream.py reads these labels, strips the compression bit, decompresses the payload, and yields (tick, msg_type, payload). It knows nothing about protobuf.

Stage 2: Outer protobuf decode (envelope layer)

Now each envelope is opened. command says which CDemo* message from demo.proto the payload is.

Here is every envelope in one real replay (match 8855242704, a 99.6-minute recording). Sizes are after decompression.

The other EDemoCommands values (DEM_ConsoleCmd, DEM_CustomData, DEM_CustomDataCallbacks, DEM_UserCmd, DEM_SaveGame, DEM_AnimationData, DEM_AnimationHeader) do not appear in this replay, and gem ignores them. DEM_Error, DEM_Max, and DEM_IsCompressed are markers, not envelope kinds.

Three things to take from this table:

1. Packets are the game. Almost every envelope is a DEM_Packet, and packets carry about 92% of the bytes. Everything else is setup at the start or a summary at the end.
2. Full packets are checkpoints. A packet only says what changed, so knowing the state at minute 40 means replaying minutes 0–39. A full packet is a saved state every 60 seconds, so a replay viewer can jump around. gem always reads from start to finish, so it opens the packet inside a full packet like any other packet and ignores the string_table snapshot, as Manta does.
3. The summary comes last. DEM_FileInfo is written when the recording ends. gem reads match_id, game_mode, leagueid, and game_winner from it. The players, picks and bans, and team tags come from the entity stream instead. A truncated replay has no summary, so after the loop parser.py reads the same facts from the CDOTAGamerulesProxy entity.

Where this happens:

• ReplayParser._dispatch_outer in parser.py chooses the CDemo* class for the five kinds gem reads and drops the rest.
• CDemoSendTables.data holds another length-prefixed protobuf, CSVCMsg_FlattenedSerializer from netmessages.proto. schema/sendtable/parser.py unwraps it.
• CDemoPacket.data holds the inner messages. That's Stage 3.

Stage 3: Inner message unpack (packet multiplexing layer)

CDemoPacket.data is not one message. It is a bundle of small inner messages packed back to back:

{ type_id, size, payload } { type_id, size, payload } ...

type_id is a ubit_var (a variable-length bit field) and size is a varuint32. read_inner_messages() in gem/binary/packet.py splits the bundle into (type_id, payload) pairs.

Each inner message is what the game server sent to spectators at that moment. A replay is that broadcast, saved to disk. The type IDs come from enums spread across several proto files, so one packet mixes messages from all of them:

Here is what the same replay as Stage 2 (match 8855242704) carries: 89,702 packets holding 2.48 million inner messages.

Voice chat takes almost as many bytes as the whole game state, and gem never decodes it.

Processing order within a packet

The messages in one packet all belong to the same moment, but some depend on others. For example, a combat-log line names a hero by string-table index, so the string table must be up to date first. Before dispatching, gem sorts each packet:

1. First: net_Tick, svc_ServerInfo, svc_ClearAllStringTables, svc_CreateStringTable, svc_UpdateStringTable. These give context, and keep their order in the packet.
2. Then everything not listed here.
3. Then svc_PacketEntities. Entity changes are applied with that context.
4. Last: GE_Source1LegacyGameEvent and DOTA_UM_CombatLogDataHLTV. Events are read after the entities they mention are up to date.

Manta sorts the same way (demo_packet.go). The difference is which messages it puts first: it includes net_SpawnGroup_Load instead of svc_ServerInfo, and it does not delay the combat-log message.

Stage 4: Inner protobuf decode (semantic layer)

parser.py (ReplayParser._dispatch_inner) decodes each inner message whose type_id it knows, using the matching protobuf class. Every other message is skipped without being decoded.

gem decodes inner messages from seven proto files:

The Proto Files gem Uses explains each file: what it carries, what gem reads from it, and what gem does with it.

Stage 5: Subsystem routing (state-building layer)

Each decoded message is handed to the part of gem that owns that kind of state:

Parsing is incremental state reconstruction. No single message says "the state of the game at minute 20". Each one changes what the earlier messages built. The extractors (extractors/) watch that state as it changes, for example sampling every hero's gold and position, and results/assembly.py turns what they collected into a ParsedMatch at the end.

Stage 6: Where protobuf stops

"If everything is protobuf, why does gem need its own bit reader?" Because the most important protobuf fields contain raw bytes in formats Valve designed for size, and protobuf can't decode those.

The main one is CSVCMsg_PacketEntities.entity_data: every entity change, packed at the bit level. For each changed entity it holds:

1. Which entity: the distance from the previous changed entity's index, as a ubit_var.
2. What happened, in 2 bits: update, create, leave (still exists, but out of view), or delete.
3. For a create: the class ID and a serial number. gem first applies the class's defaults from the instancebaseline string table.
4. Which fields changed and their new values. Fields are addressed by field paths, compressed with a Huffman code (schema/field_path/). Each value is decoded according to its field's type in the schema: a quantized float, a varint, a string, and so on (schema/field_decoder/).

Nothing in this blob is labelled. Without the schema from DEM_SendTables, it's just bits. That's why the schema has to arrive before the first entity update.

String tables work the same way on a smaller scale: CSVCMsg_CreateStringTable and CSVCMsg_UpdateStringTable are protobuf, but their string_data is a bit-packed list of entries, sometimes Snappy-compressed (state/string_table.py).

So parsing a replay uses two kinds of decoding:

• protobuf for the structure of messages
• gem's own bit-level decoding (binary/reader.py, schema/, state/) for the packed game state inside them

A worked example: first blood

Here is a real packet from match 8855242704: the one containing first blood, at replay tick 28,757. It is a 4.7 KB DEM_Packet with 56 inner messages. In the order they appear on the wire:

The combat-log death line comes before the clock tick and the entity update. If gem processed messages in wire order, it would record the death before advancing the clock or updating the entities. So gem sorts the packet first (Stage 3):

1. net_Tick advances the clock, and svc_UpdateStringTable updates the name lookups.
2. The chat events are handled: the hero kill and first blood.
3. svc_PacketEntities applies the 170 entity changes, including the dying hero.
4. Last, the combat-log lines are read. The death line's attacker_name and target_name are resolved through the up-to-date CombatLogNames table, and the entities they refer to are already in their state for this tick.

This ordering is why replay parsing is a pipeline, and not a set of independent per-message decoders.

When to use this page vs other docs

• Use this page when you want the big picture and message roles.
• Use The Proto Files gem Uses for what each relevant proto file carries and what gem does with it.
• Use Proto Field Atlas when you need per-message field details.
• Use the API Reference when you need function and class signatures.