> For the complete documentation index, see [llms.txt](https://magiceden.gitbook.io/rune-spec/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://magiceden.gitbook.io/rune-spec/runes-protocol-message.md). # Runes Protocol Message This will detail out all parts of the runes protocol message format religiously. This will hopefully help centralize everyone's efforts in building out a reference indexer whose logic all can come to a consensus around. At the end of this document, there will be suggestions on extensions and/or modifications that could be considered to improve the protocol. ## Overview > Rune balances are held by UTXOs. A UTXO can contain any amount of any number of runes. This means that runes aren't stored as an account balance such as BRC-20, but rather stored directly into UTXOs, which means that access to runes is equivalent to access to the holding UTXO. Thus indexers only need to track rune-holding UTXOs and the runes they hold. > A transaction contains a protocol message if it contains an output whose script pubkey contains an OP\_RETURN followed by a data push of the ASCII uppercase letter `R`. The protocol message is all data pushes after the first. An example of the scriptPubKey that encodes a transfer protocol message correctly would look as follows: ``` OP_RETURN OP_PUSH_1 R OP_PUSH_3 (10, 1, 100) ``` and an example of an issuance message is as follows: ``` OP_RETURN OP_PUSH_1 R OP_PUSH_10 (0, 1, 2_100_000_000_000_000) OP_PUSH_4 (RUNE, 8) ``` More details of how each message type is serialized will be covered below. > After processing all tuple assignments, any unassigned runes are assigned to the first non-OP\_RETURN output, if any. Casey never mentions whether there can be multiple OP\_RETURNs each with its own protocol message. But taking ordinals as inspiration, that situation should be allowed (even though more than one OP\_RETURN output is unlikely given its non-standardness). To handle this, we just process the messages in order, and then handle unassigned runes after all assignments. Also, this will cover the case when there are no protocol messages but there are inputs with runes; they will automatically follow the default rule of being all assigned to the first non-OP\_RETURN output. This will allow for a marketplace experience similar to BRC-20 today. > Integers are encoded as prefix varints, where the number of leading ones in a varint determines its length in bytes. Even though Casey uses the term varint, this is a general term for variable size integers, and not necessarily to same protocol used for Bitcoin. This instead uses leading/prefix ones to determine the total integer size. ``` Number of leading 1s and a stopping 0 equals total byte size 1 byte: 0XXXXXXX (0 length is 1) 2 bytes: 10XXXXXX XXXXXXXX (10 length is 2) 3 bytes: 110XXXXX XXXXXXXX XXXXXXXX (110 length is 3) ... This allows for encoding 7 bits per bytes, with 1 bit/byte used as size indicator. ``` A reference implementation is provided here: ```typescript function readBigIntFromBuffer(buffer: Buffer): bigint | undefined { const bufferBitArray = [ ...BigInt('0x' + buffer.toString('hex')) .toString(2) .padStart(buffer.length * 8, '0'), ].map(Number); let numberByteSize = 0; for ( let bit = bufferBitArray.shift(); bit !== undefined; bit = bufferBitArray.shift() ) { numberByteSize++; if (!bit) { break; } } if (numberByteSize > buffer.length) { return undefined; } const valuesToReturn = this.readBigIntsFromBuffer( buffer.slice(numberByteSize), ); const valueBitString = bufferBitArray .slice(0, numberByteSize * 7) .map(String) .join(''); return BigInt('0b' + valueBitString); } function getBigIntBuffer(integer: bigint): Buffer { const valueBitString = value.toString(2); const totalNumBytes = Math.ceil(valueBitString.length / 7); const bufferBitString = _.repeat('1', totalNumBytes - 1) + '0' + _.repeat('0', totalNumBytes * 7 - valueBitString.length) + valueBitString; const bufferBigInt = BigInt('0b' + bufferBitString); const buffer = Buffer.alloc(totalNumBytes); for (let bufferIndex = 0; bufferIndex < totalNumBytes; bufferIndex++) { buffer.writeUInt8( Number( (bufferBigInt >> (8n * BigInt(totalNumBytes - 1 - bufferIndex))) & 0xffn, ), bufferIndex, ); } return buffer; } ``` Note there are implementations out there that implement Bitcoin varint, which is a different serialization which uses FD/FE/FF in first byte as a special signal that the next 2/4/8 bytes contains the integer value. Another serialization format that is very similar to above is the prefix varint64, which is almost the same except it assumes that the integer to serialize max value is 2^64-1, so it optimizes the max varint size to be 9 bytes instead of 10 by assuming max 8 leading ones, so a stopping zero isn't needed for that case. Unfortunately, that assumption that max int we would want to encode is 2^64-1 is wrong, as quoted below. > An issuance transaction may create any amount, up to `2^128 - 1` of the issued rune, using the ID `0` in assignment tuples. Thus the prefix varint64 is not sufficient to store large values. As to why we need such large values, consider that 2^64-1 is about 1.84e19, which given a decimal value of 18, will effectively only issue 18.4 runes. Whereas 2^128-1 is 3.4e38, so even 18 decimal places allows for more than enough whole runes left over. Finally, the debate of little-endian vs big-endian for the actual integer encoding. Given that we are likely to use some bits in a byte for the integer, little-endian will make actual int byte alignment and varint byte alignment off, whereas big-endian keeps that. For example, little-endian encoding of 200 (hex `0xc8`) is `0x8803` vs big-endian encoding being `0x80c8`. This has the nice property of being more human-readable as well. ## Transfer > If the number of decoded integers is not a multiple of three, the protocol message is invalid. This data push expects there to be a multiple of 3 varints (`ID`, `OUTPUT`, `AMOUNT`), and any deviation of that is an invalid message, which results in a burn of all runes in the transaction. What is curious is that technically a push of 0 integers is technically valid here (e.g. OP\_0). Also, the provided example `[(100, 1, 20), (0, 2 10), (20, 1, 5)]` doesn't explicitly mention whether the output index is 0-based or 1-based. By default we will assume 0-based, but it could be interesting to be 1-based, and allow 0 to be a special value (such as to all non-OP\_RETURN outputs for fanout - this can help with issuers creating a fair launch of their token). > If `SYMBOL` has not already been assigned, it is assigned to the issued rune, and the issued rune receives the next available numeric rune ID, starting at one. > > If `SYMBOL` has already been assigned, or is `BITCOIN`, `BTC`, or `XBT`, then no new rune is created. Issuance transaction assignments using the `0` rune ID are ignored, but other assignments are still processed. This currently makes SYMBOL to be unique, and IDs to be monotonically increasing. Note the way to think about issuance is that it acts as the singular "coinbase" transaction for the created rune, so the only time when there is a rune output without input in the transfer instructions. And this falls back to doing nothing for ID 0 if issuance instructions doesn't create a new rune, since it becomes no new rune to assign, and `excess assignments are ignored`. ## Issuance > The second data push is decoded as two integers, `SYMBOL`, `DECIMALS` > `SYMBOL` is a base 26-encoded human readable symbol, similar to that used in ordinal number sat names. The only valid characters are `A` through `Z`. This is encoded in the following compact way: ``` 0 -> "" 1 -> "A" 26 -> "Z" 27 -> "AA" ... 26**2 + 26 -> "ZZ" 26**3 + 26**2 + 26 -> "ZZZ" The mapping is starting from 0-length string with 0 combinations, then 1-length with 26 combinations, then 2-length with 26**2 combinations, and so forth with n-length with 26**n combinations. ``` Another reference implementation is provided here, courtesy of [@revofusion](https://twitter.com/revofusion/status/1706839841684729942) (slightly modified): ```typescript function getSymbol(x: number): string { let s = ''; while (x > 0) { s += 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'.charAt((x - 1) % 26); x = Math.floor((x - 1) / 26); } return s.split('').reverse().join(''); } function fromSymbol(s: string): number { s = s.toUpperCase() let x = 0; for (const c of s) { if (c >= 'A' && c <= 'Z') { x = x * 26 + c.charCodeAt(0) - 'A'.charCodeAt(0) + 1; } else { throw new Error(`invalid character in sat name: ${c}`); } } return x; } ``` Note any other base26 encodings out there are not actually base26. For example, an encoding of RUST -> 17201304 (decimal) is just actually a wasteful base100 encoding using only 26 values (and doesn't even disambiguate leading As). ## Modification Suggestions ### Non-uniqueness of symbols It is nice to have unique symbols for each rune, so that when we mention RUNE, it is unambiguous. However, in practice this leads to immediate squatting of symbols, and also forces indexers to be aware of global state to confidently state it. Finally, it gives runes two unique identifiers (ID and symbol), which seems redundant. It is better to relax this constraint, which allows for indexers to be simpler, and if really desired, a metaprotocol can be overlaid to provide to runes whether they are the first symbol type or not (at least any errors in determining first doesn't stop the rune from being tracked). ### Rune ID based off of issuance transaction details Rune IDs currently are monotonically increasing sequence numbers. Similar to the arguments above in non-uniqueness of symbols, to support this style of IDs makes the logic more complicated than necessary. This is indeed the most space efficient style of IDs, but being able to construct an ID regardless of other runes will also make indexers simpler, and the goal for a rune indexer is to be the simplest as possible so it can be as bug-free and edge-case free as possible. A proposal for a new ID would be to use enclosing transaction's "compact" txid, which would be 2 varints: (`BLOCKHEIGHT`, `TXINDEX`). This will put a limitation on issuance being only valid when in the first OP\_RETURN envelope, which is generally a given. Note at this writing, this compact txid would be 5 bytes, so very space efficient given no global indexer state needed. The transfer tuple will now look like (`BLOCKHEIGHT`, `TXINDEX`, `INPUT`, `AMOUNT`). This will no longer support ID deltas, but we can tweak the transfer instructions to be even more compact, using the insight that must users will be more interested in lots of assignments of the same rune vs assignments of lots of runes. For issuance, we can use the tuple (0, 0) to represent the newly created rune. ### Compression of transfer instructions If we follow from the new rune ID of (`BLOCKHEIGHT`, `TXINDEX`), then we don't have a good way of reusing the same ID in a compressed manner, since the only way to show delta of two ID tuples is to have to introduce encoding of negative varints, which we would like to avoid. What we can do is rather than mapping a rune ID to just one assignment, we can encode a varint that states the number of assignments related to specified rune to follow. For example: ``` BLOCKHEIGHT, TXINDEX, NUMASSIGNMENTS, (INPUT, AMOUNT) x NUMASSIGNMENTS Concrete values: 809620 1677 5 1 1000 2 1000 3 1000 4 5000 5 9999 Above example assigns rune (809620, 1677): - outputs 1,2,3 1000 count - output 4 5000 count - output 5 9999 count ``` We could allow `NUMASSIGNMENTS` equal 0 be a special assignment type, where it only takes one `AMOUNT` integer after it, of which it assigns each non-OP\_RETURN outputs of that amount. This can be very useful in doing a fair launch of a rune, where each output is just a p2wsh with varying relative block distance timelocks to slowly release the rune for public minting.