From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191] helo=mx.sourceforge.net) by sfs-ml-4.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1YHquk-0000UK-ON for bitcoin-development@lists.sourceforge.net; Sun, 01 Feb 2015 09:33:58 +0000 X-ACL-Warn: Received: from wp059.webpack.hosteurope.de ([80.237.132.66]) by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.76) id 1YHqui-00084l-VL for bitcoin-development@lists.sourceforge.net; Sun, 01 Feb 2015 09:33:58 +0000 Received: from [37.143.74.116] (helo=[192.168.0.100]); authenticated by wp059.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16) id 1YHquc-0005ai-Ao; Sun, 01 Feb 2015 10:33:50 +0100 From: Tamas Blummer Content-Type: multipart/signed; boundary="Apple-Mail=_32EFF217-B018-44CF-A0E9-1DC7C742830C"; protocol="application/pgp-signature"; micalg=pgp-sha512 Message-Id: Date: Sun, 1 Feb 2015 10:33:48 +0100 To: Bitcoin Dev Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-bounce-key: webpack.hosteurope.de; tamas@bitsofproof.com; 1422783237; 7c51f473; X-Spam-Score: 1.0 (+) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [80.237.132.66 listed in list.dnswl.org] 1.0 HTML_MESSAGE BODY: HTML included in message X-Headers-End: 1YHqui-00084l-VL Subject: [Bitcoin-development] var_int ambiguous serialization consequences X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Feb 2015 09:33:58 -0000 --Apple-Mail=_32EFF217-B018-44CF-A0E9-1DC7C742830C Content-Type: multipart/alternative; boundary="Apple-Mail=_EC593710-1FB9-4AAB-92F0-A44D0068DF52" --Apple-Mail=_EC593710-1FB9-4AAB-92F0-A44D0068DF52 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii I wonder of consequences if var_int is used in its longer than necessary = forms (e.g encoding 1 as 0xfd0100 instead of 0x01) This is already of interest if applying size limit to a block, since = transaction count is var_int but is not part of the hashed header or the = merkle tree. It could also be used to create variants of the same transaction message = by altered representation of txIn and txout counts, that would remain = valid provided signatures validate with the shortest form, as that is = created while re-serializing for signature hashing. An implementation = that holds mempool by raw message hashes could be tricked to believe = that a modified encoded version of the same transaction is a real double = spend. One could also mine a valid block with transactions that have a = different hash if regularly parsed and re-serialized. An SPV client = could be confused by such a transaction as it was present in the merkle = tree proof with a different hash than it gets for the tx with its own = serialization or from the raw message. Tamas Blummer Bits of Proof --Apple-Mail=_EC593710-1FB9-4AAB-92F0-A44D0068DF52 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

I = wonder of consequences if var_int is used in its longer than necessary = forms (e.g encoding 1 as 0xfd0100 instead of = 0x01)

This is already of interest if applying = size limit to a block, since transaction count is var_int but is not = part of the hashed header or the merkle = tree.

It could also be used to create variants = of the same transaction message by altered representation of txIn and = txout counts, that would remain valid provided signatures validate with = the shortest form, as that is created while re-serializing for signature = hashing. An implementation that holds mempool by raw message hashes = could be tricked to believe that a modified encoded version of the same = transaction is a real double spend. One could also mine a valid block = with transactions that have a different hash if regularly parsed and = re-serialized. An SPV client could be confused by such a transaction as = it was present in the merkle tree proof with a different hash than it = gets for the tx with its own serialization or from the raw = message.

Tamas = Blummer

Bits of Proof

= --Apple-Mail=_EC593710-1FB9-4AAB-92F0-A44D0068DF52-- --Apple-Mail=_32EFF217-B018-44CF-A0E9-1DC7C742830C Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUzfL8AAoJEPZykcUXcTkcwTAH/1ZnaqjpmLGGtEim0vLPb9D8 b9COVLIoEJ7VKiLIQ8+sSTZcuo6W0k9BCoZbWdZi/NN3Yag2QSzD1AD6bvLhHAuL F600XHuHkFtpl2k2a2zNfYuGM7zn71eChTwiJwld6jEjqsILZi1O7ZMgOWRbEJzm 3HS9tRPmX9pW4CbQZNw38+kKtpDc2WraN66HWz89oW5vRn4EKV6QLbRT4ggDyrvW mLBXn2TV/JQ0qQfB97Q8hmxl6FgOtNrpjAIqcNt+7Pe8A3wshi3SiJYhTTa25vaq RZTZpQKhyOxJy9zF1baLZOPywbSj3CyMouNoohtcBhGBjHIBKtote0I8RWrtA6M= =EoXH -----END PGP SIGNATURE----- --Apple-Mail=_32EFF217-B018-44CF-A0E9-1DC7C742830C-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191] helo=mx.sourceforge.net) by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1YHs1O-0000Ej-Ue for bitcoin-development@lists.sourceforge.net; Sun, 01 Feb 2015 10:44:54 +0000 Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com designates 74.125.82.43 as permitted sender) client-ip=74.125.82.43; envelope-from=laanwj@gmail.com; helo=mail-wg0-f43.google.com; Received: from mail-wg0-f43.google.com ([74.125.82.43]) by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1YHs1O-0001zD-3S for bitcoin-development@lists.sourceforge.net; Sun, 01 Feb 2015 10:44:54 +0000 Received: by mail-wg0-f43.google.com with SMTP id y19so33970393wgg.2 for ; Sun, 01 Feb 2015 02:44:48 -0800 (PST) X-Received: by 10.180.82.137 with SMTP id i9mr13574470wiy.38.1422787488067; Sun, 01 Feb 2015 02:44:48 -0800 (PST) Received: from amethyst.lan (e107003.upc-e.chello.nl. [213.93.107.3]) by mx.google.com with ESMTPSA id vq9sm22829544wjc.6.2015.02.01.02.44.46 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 01 Feb 2015 02:44:47 -0800 (PST) Date: Sun, 1 Feb 2015 11:44:46 +0100 (CET) From: Wladimir To: Tamas Blummer In-Reply-To: Message-ID: References: User-Agent: Alpine 2.10 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: 1.2 (+) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (laanwj[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 2.8 MALFORMED_FREEMAIL Bad headers on message from free email service 0.0 AWL AWL: Adjusted score from AWL reputation of From: address X-Headers-End: 1YHs1O-0001zD-3S Cc: Bitcoin Dev Subject: Re: [Bitcoin-development] var_int ambiguous serialization consequences X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Feb 2015 10:44:55 -0000 On Sun, 1 Feb 2015, Tamas Blummer wrote: > I wonder of consequences if var_int is used in its longer than necessary forms (e.g encoding 1 as 0xfd0100 instead of 0x01) In serialize.h lingo you are talking about CompactSize, not VarInt. CompactSizes indeed have redundancy in their representation, i.e. the same number can be represented as up to four different byte sequences. VARINTs have a different format that (AFAIK) isn't used anywhere in the block chain. See WriteVarInt / ReadVarInt. These were designed to not have any redundancy in their representation. > This is already of interest if applying size limit to a block, since transaction count is var_int but is not part of the hashed header or the > merkle tree. Are you sure that this is a current concern? Non-canonical CompactSizes are forbidden - in serialize.h this is flagged in ReadCompactSize. Wladimir From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192] helo=mx.sourceforge.net) by sfs-ml-1.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1YHsuu-000098-Sp for bitcoin-development@lists.sourceforge.net; Sun, 01 Feb 2015 11:42:16 +0000 X-ACL-Warn: Received: from wp059.webpack.hosteurope.de ([80.237.132.66]) by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.76) id 1YHsut-0007We-4N for bitcoin-development@lists.sourceforge.net; Sun, 01 Feb 2015 11:42:16 +0000 Received: from [37.143.74.116] (helo=[192.168.0.100]); authenticated by wp059.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16) id 1YHsum-000463-Fm; Sun, 01 Feb 2015 12:42:08 +0100 Content-Type: multipart/signed; boundary="Apple-Mail=_EE60C463-12DC-4352-B47E-E8A1FD965FF1"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) From: Tamas Blummer In-Reply-To: Date: Sun, 1 Feb 2015 12:42:05 +0100 Message-Id: References: To: Wladimir X-Mailer: Apple Mail (2.1878.6) X-bounce-key: webpack.hosteurope.de; tamas@bitsofproof.com; 1422790935; c2886a59; X-Spam-Score: 1.0 (+) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [80.237.132.66 listed in list.dnswl.org] 1.0 HTML_MESSAGE BODY: HTML included in message X-Headers-End: 1YHsut-0007We-4N Cc: Bitcoin Dev Subject: Re: [Bitcoin-development] var_int ambiguous serialization consequences X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Feb 2015 11:42:16 -0000 --Apple-Mail=_EE60C463-12DC-4352-B47E-E8A1FD965FF1 Content-Type: multipart/alternative; boundary="Apple-Mail=_07784863-C922-4F18-9704-FD10DFB8BE5D" --Apple-Mail=_07784863-C922-4F18-9704-FD10DFB8BE5D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Thanks for the clarification. Yes, I referred to CompactSize using the = lingo of https://en.bitcoin.it/wiki/Protocol_documentation I am not sure if it is current concern. Apparently an exception is = thrown if non-canonical CompactSize in a transaction s parsed. Is it ensured that transactions are always parsed before computing their = hash? Tamas Blummer On Feb 1, 2015, at 11:44 AM, Wladimir wrote: >=20 > On Sun, 1 Feb 2015, Tamas Blummer wrote: >=20 >> I wonder of consequences if var_int is used in its longer than = necessary forms (e.g encoding 1 as 0xfd0100 instead of 0x01) >=20 > In serialize.h lingo you are talking about CompactSize, not VarInt. >=20 > CompactSizes indeed have redundancy in their representation, i.e. the = same number can be represented as up to four different byte sequences. >=20 > VARINTs have a different format that (AFAIK) isn't used anywhere in = the block chain. See WriteVarInt / ReadVarInt. These were designed to = not have any redundancy in their representation. >=20 >> This is already of interest if applying size limit to a block, since = transaction count is var_int but is not part of the hashed header or the >> merkle tree. >=20 > Are you sure that this is a current concern? Non-canonical = CompactSizes are forbidden - in serialize.h this is flagged in = ReadCompactSize. >=20 > Wladimir >=20 >=20 --Apple-Mail=_07784863-C922-4F18-9704-FD10DFB8BE5D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

Thanks for the clarification. Yes, I referred = to CompactSize using the lingo of https://en.bitc= oin.it/wiki/Protocol_documentation

I am not = sure if it is current concern. Apparently an exception is thrown if = non-canonical CompactSize in a transaction s parsed.

Is it = ensured that transactions are always parsed before computing their = hash?

Tamas = Blummer

On Feb 1, 2015, at 11:44 AM, Wladimir = <laanwj@gmail.com> = wrote:

On Sun, 1 Feb 2015, Tamas Blummer = wrote:

I wonder of consequences if = var_int is used in its longer than necessary forms (e.g encoding 1 as = 0xfd0100 instead of 0x01)

In serialize.h lingo you = are talking about CompactSize, not VarInt.

CompactSizes indeed = have redundancy in their representation, i.e. the same number can be = represented as up to four different byte sequences.

VARINTs have = a different format that (AFAIK) isn't used anywhere in the block chain. = See WriteVarInt / ReadVarInt. These were designed to not have any = redundancy in their representation.

This = is already of interest if applying size limit to a block, since = transaction count is var_int but is not part of the hashed header or = the
merkle tree.

Are you sure that this is a = current concern? Non-canonical CompactSizes are forbidden - in = serialize.h this is flagged in = ReadCompactSize.

Wladimir

= = --Apple-Mail=_07784863-C922-4F18-9704-FD10DFB8BE5D-- --Apple-Mail=_EE60C463-12DC-4352-B47E-E8A1FD965FF1 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUzhENAAoJEPZykcUXcTkcfE8H/2Kc/beKqA4o+Kh3huYz27nt MA9fQCCVSycc1c/3Ph7/SwRQDndG/o/ws9r/Gn+jrh675SiFjOkbOG3So8Gob/Qz wjS4mkSgZIRYGWzoYsAElZW2xzM5SvqaO7CqGspZympJL8y/QuBvZgF9Mla1fYLf CTZ6xgQupVgUwNecRNG7mhBc3X+D6zAWNWQFM4Q4Kb1GQXacDw/Agon5Ib4baWZ+ 8SjyDjMKAZ9W8R7+hiAxj1cV0CmhK1Gz3f82fmFgym72nzyEULJqFswyrdIV8yZv Cc3ODN6g76Ly2/FdbuRRtH1W3+HWQsud6eBp9S+X9OCz+816s11vMqZ4NCTlz+8= =SU3B -----END PGP SIGNATURE----- --Apple-Mail=_EE60C463-12DC-4352-B47E-E8A1FD965FF1-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193] helo=mx.sourceforge.net) by sfs-ml-4.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1YHw10-0006rg-1s for bitcoin-development@lists.sourceforge.net; Sun, 01 Feb 2015 15:00:46 +0000 Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.213.178 as permitted sender) client-ip=209.85.213.178; envelope-from=pieter.wuille@gmail.com; helo=mail-ig0-f178.google.com; Received: from mail-ig0-f178.google.com ([209.85.213.178]) by sog-mx-3.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1YHw0y-0008AY-Si for bitcoin-development@lists.sourceforge.net; Sun, 01 Feb 2015 15:00:46 +0000 Received: by mail-ig0-f178.google.com with SMTP id hl2so12063745igb.5 for ; Sun, 01 Feb 2015 07:00:39 -0800 (PST) MIME-Version: 1.0 X-Received: by 10.107.41.212 with SMTP id p203mr17899230iop.54.1422802839534; Sun, 01 Feb 2015 07:00:39 -0800 (PST) Received: by 10.50.20.229 with HTTP; Sun, 1 Feb 2015 07:00:39 -0800 (PST) Received: by 10.50.20.229 with HTTP; Sun, 1 Feb 2015 07:00:39 -0800 (PST) In-Reply-To: References: Date: Sun, 1 Feb 2015 11:00:39 -0400 Message-ID: From: Pieter Wuille To: Tamas Blummer Content-Type: multipart/alternative; boundary=001a1141f186d60419050e08197e X-Spam-Score: -0.6 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (pieter.wuille[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 1.0 HTML_MESSAGE BODY: HTML included in message -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1YHw0y-0008AY-Si Cc: Bitcoin Dev Subject: Re: [Bitcoin-development] var_int ambiguous serialization consequences X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Feb 2015 15:00:46 -0000 --001a1141f186d60419050e08197e Content-Type: text/plain; charset=ISO-8859-1 Hashes are always computed by reserializing data structures, never by hashing wire data directly. This has been the case in every version of the reference client's code that I know of. This even meant that for example a block of 999999 bytes with non-shortest length for the transaction count could be over the mazimum block size, but still be valid. As Wladimir says, more recently we switched to just failing to deserialize (by throwing an exception) whenever a non-shortest form is used. On Feb 1, 2015 1:34 AM, "Tamas Blummer" wrote: > I wonder of consequences if var_int is used in its longer than necessary > forms (e.g encoding 1 as 0xfd0100 instead of 0x01) > > This is already of interest if applying size limit to a block, since > transaction count is var_int but is not part of the hashed header or the > merkle tree. > > It could also be used to create variants of the same transaction message > by altered representation of txIn and txout counts, that would remain valid > provided signatures validate with the shortest form, as that is created > while re-serializing for signature hashing. An implementation that holds > mempool by raw message hashes could be tricked to believe that a modified > encoded version of the same transaction is a real double spend. One could > also mine a valid block with transactions that have a different hash if > regularly parsed and re-serialized. An SPV client could be confused by such > a transaction as it was present in the merkle tree proof with a different > hash than it gets for the tx with its own serialization or from the raw > message. > > Tamas Blummer > Bits of Proof > > > > ------------------------------------------------------------------------------ > Dive into the World of Parallel Programming. The Go Parallel Website, > sponsored by Intel and developed in partnership with Slashdot Media, is > your > hub for all things parallel software development, from weekly thought > leadership blogs to news, videos, case studies, tutorials and more. Take a > look and join the conversation now. http://goparallel.sourceforge.net/ > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > --001a1141f186d60419050e08197e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Hashes are always computed by reserializing data structures,= never by hashing wire data directly. This has been the case in every versi= on of the reference client's code that I know of.

This even meant that for example a block of 999999 bytes wit= h non-shortest length for the transaction count could be over the mazimum b= lock size, but still be valid.

As Wladimir says, more recently we switched to just failing = to deserialize (by throwing an exception) whenever a non-shortest form is u= sed.

On Feb 1, 2015 1:34 AM, "Tamas Blummer"= ; <tamas@bitsofproof.com>= ; wrote:

I wonder of consequences if var_int is used = in its longer than necessary forms (e.g encoding 1 as 0xfd0100 instead of 0= x01)

This is already of interest if applying size = limit to a block, since transaction count is var_int but is not part of the= hashed header or the merkle tree.

It could also b= e used to create variants of the same transaction message by altered repres= entation of txIn and txout counts, that would remain valid provided signatu= res validate with the shortest form, as that is created while re-serializin= g for signature hashing. An implementation that holds mempool by raw messag= e hashes could be tricked to believe that a modified encoded version of the= same transaction is a real double spend. One could also mine a valid block= with transactions that have a different hash if regularly parsed and re-se= rialized. An SPV client could be confused by such a transaction as it was p= resent in the merkle tree proof with a different hash than it gets for the = tx with its own serialization or from the raw message.

Tamas Blummer
Bits of Proof

--------------------------------------------------------------------= ----------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is you= r
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a<= br> look and join the conversation now. http://goparallel.sourceforge.net/
_______= ________________________________________
Bitcoin-development mailing list
Bitcoin-develo= pment@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-de= velopment

--001a1141f186d60419050e08197e--