* [bitcoin-dev] Trying to patch Core ZMQ "rawtx" topic to only publish unconfirmed transactions: How?
@ 2021-11-26 16:56 Ali Sherief
2021-11-29 9:32 ` 0xB10C
0 siblings, 1 reply; 5+ messages in thread
From: Ali Sherief @ 2021-11-26 16:56 UTC (permalink / raw)
To: bitcoin-dev
[-- Attachment #1: Type: text/plain, Size: 2079 bytes --]
This has also been posted on Bitcointalk forum: https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261 I have republished it here hoping someone more knowledgeable can post some insight about this.
----
It appears that the ZeroMQ topic I'm listening to, "rawtx", not only emits a raw transaction when it appears on the mempool, but once it's already confirmed too.
This messes with my software, causing it to add txids, addresses, etc. a second time inside arrays (this means that the same transaction is received twice in total).
Array de-duping is not a viable solution long-term (because the array will quickly grow to be big eventually and then this has to happen every time a new element is added), so I'm trying to nip the problem from the source by instructing Core to only publish unconfirmed bitcoin transactions.
According to https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra , it is not possible to configure this from a configuration or command-line option. The source code must directly be edited. But since the codebase has changed greatly, the proposed solution no longer works.
----
So basically, I know that something inside src/zmq/zmqnotificationinterface.cpp needs to be patched, but I'm not sure which function, or how to do it. Because I only need unconfirmed transactions to be published on ZeroMQ rawtx and not confirmed ones, it's one of the following functions that I need to patch for my own build:
CZMQNotificationInterface::TransactionRemovedFromMempool
void CZMQNotificationInterface::BlockDisconnected
Both of these call NotifyTransaction() method which I assume fires a message on "rawtx" channel.
In the Stack Exchange question I linked above, Jonas Schnelli suggested adding an `if (!pblock)` check, but that was several years ago and the function he was referencing no longer exists.
But I still wonder if the pblock check is still applicable in the present day (i.e. if it's indicating a block the transaction is inside).
- Ali Sherief
[-- Attachment #2: Type: text/html, Size: 3020 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [bitcoin-dev] Trying to patch Core ZMQ "rawtx" topic to only publish unconfirmed transactions: How?
2021-11-26 16:56 [bitcoin-dev] Trying to patch Core ZMQ "rawtx" topic to only publish unconfirmed transactions: How? Ali Sherief
@ 2021-11-29 9:32 ` 0xB10C
2021-11-29 14:10 ` LORD HIS EXCELLENCY JAMES HRMH
0 siblings, 1 reply; 5+ messages in thread
From: 0xB10C @ 2021-11-29 9:32 UTC (permalink / raw)
To: Ali Sherief, Bitcoin Protocol Discussion
Hi Ali,
I've run into this multiple times myself. I've opened a draft PR [0]
adding a rawmempooltx publisher.
You're right. In zmq/zmqnotificationinterface.cpp the
CZMQNotificationInterface is notified about TransactionAddedToMempool.
Currently, this calls NotifyTransaction() (the publisher with the rawtx
topic) and NotifyTransactionAcceptance() (the publisher with the
sequence topic)[1]. I've added a call to a new
NotifyMempoolTransaction() function (the publisher with the rawmempooltx
topic).
I'd find a mempool transaction publisher with both the raw transaction
and transaction fee useful too. However, this requires changes to the
chain notifications in interfaces/chain.h.
[0]: https://github.com/bitcoin/bitcoin/pull/23624
[1]:
https://github.com/bitcoin/bitcoin/pull/23624/files#diff-ac4b2d3a8de2c4dd41ad9d75505ea6ce4dc87a476710a9ebee8acf9bebf5cca2L146-L148
Best,
0xB10C
On 11/26/21 5:56 PM, Ali Sherief via bitcoin-dev wrote:
>
> This has also been posted on Bitcointalk
> forum: https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261
> <https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261> I
> have republished it here hoping someone more knowledgeable can post
> some insight about this.
> ----
> It appears that the ZeroMQ topic I'm listening to, "rawtx", not only
> emits a raw transaction when it appears on the mempool, but once it's
> already confirmed too.
>
> This messes with my software, causing it to add txids, addresses, etc.
> a second time inside arrays (this means that the same transaction is
> received twice in total).
>
> Array de-duping is not a viable solution long-term (because the array
> will quickly grow to be big eventually and then this has to happen
> every time a new element is added), so I'm trying to nip the problem
> from the source by instructing Core to only publish unconfirmed
> bitcoin transactions.
>
> According to
> https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra
> <https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra>
> , it is not possible to configure this from a configuration or
> command-line option. The source code must directly be edited. But
> since the codebase has changed greatly, the proposed solution no
> longer works.
>
> ----
>
> So basically, I know that something inside
> src/zmq/zmqnotificationinterface.cpp needs to be patched, but I'm not
> sure which function, or how to do it. Because I only need unconfirmed
> transactions to be published on ZeroMQ rawtx and not confirmed ones,
> it's one of the following functions that I need to patch for my own build:
>
> CZMQNotificationInterface::TransactionRemovedFromMempool
> void CZMQNotificationInterface::BlockDisconnected
>
> Both of these call NotifyTransaction() method which I assume fires a
> message on "rawtx" channel.
>
> In the Stack Exchange question I linked above, Jonas Schnelli
> suggested adding an `if (!pblock)` check, but that was several years
> ago and the function he was referencing no longer exists.
>
> But I still wonder if the pblock check is still applicable in the
> present day (i.e. if it's indicating a block the transaction is inside).
>
> - Ali Sherief
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [bitcoin-dev] Trying to patch Core ZMQ "rawtx" topic to only publish unconfirmed transactions: How?
2021-11-29 9:32 ` 0xB10C
@ 2021-11-29 14:10 ` LORD HIS EXCELLENCY JAMES HRMH
2021-11-29 14:13 ` Ali Sherief
0 siblings, 1 reply; 5+ messages in thread
From: LORD HIS EXCELLENCY JAMES HRMH @ 2021-11-29 14:10 UTC (permalink / raw)
To: Ali Sherief, Bitcoin Protocol Discussion, 0xB10C
[-- Attachment #1: Type: text/plain, Size: 5075 bytes --]
Wasn't this already not a problem because you can check if it was confirmed? The transaction is not finalised in the mempool it is just speculation of a transaction, so it makes sense to emit when the transaction is confirmed. Just already check..
> It appears that the ZeroMQ topic I'm listening to, "rawtx", not only
> emits a raw transaction when it appears on the mempool, but once it's
> already confirmed too.
KING JAMES HRMH
Great British Empire
Regards,
The Australian
LORD HIS EXCELLENCY JAMES HRMH (& HMRH)
of Hougun Manor & Glencoe & British Empire
MR. Damian A. James Williamson
Wills
et al.
Willtech
www.willtech.com.au
www.go-overt.com
duigco.org DUIGCO API
and other projects
m. 0487135719
f. +61261470192
This email does not constitute a general advice. Please disregard this email if misdelivered.
________________________________
From: bitcoin-dev <bitcoin-dev-bounces@lists•linuxfoundation.org> on behalf of 0xB10C via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org>
Sent: Monday, 29 November 2021 8:32 PM
To: Ali Sherief <ali@notatether•com>; Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation.org>
Subject: Re: [bitcoin-dev] Trying to patch Core ZMQ "rawtx" topic to only publish unconfirmed transactions: How?
Hi Ali,
I've run into this multiple times myself. I've opened a draft PR [0]
adding a rawmempooltx publisher.
You're right. In zmq/zmqnotificationinterface.cpp the
CZMQNotificationInterface is notified about TransactionAddedToMempool.
Currently, this calls NotifyTransaction() (the publisher with the rawtx
topic) and NotifyTransactionAcceptance() (the publisher with the
sequence topic)[1]. I've added a call to a new
NotifyMempoolTransaction() function (the publisher with the rawmempooltx
topic).
I'd find a mempool transaction publisher with both the raw transaction
and transaction fee useful too. However, this requires changes to the
chain notifications in interfaces/chain.h.
[0]: https://github.com/bitcoin/bitcoin/pull/23624
[1]:
https://github.com/bitcoin/bitcoin/pull/23624/files#diff-ac4b2d3a8de2c4dd41ad9d75505ea6ce4dc87a476710a9ebee8acf9bebf5cca2L146-L148
Best,
0xB10C
On 11/26/21 5:56 PM, Ali Sherief via bitcoin-dev wrote:
>
> This has also been posted on Bitcointalk
> forum: https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261
> <https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261> I
> have republished it here hoping someone more knowledgeable can post
> some insight about this.
> ----
> It appears that the ZeroMQ topic I'm listening to, "rawtx", not only
> emits a raw transaction when it appears on the mempool, but once it's
> already confirmed too.
>
> This messes with my software, causing it to add txids, addresses, etc.
> a second time inside arrays (this means that the same transaction is
> received twice in total).
>
> Array de-duping is not a viable solution long-term (because the array
> will quickly grow to be big eventually and then this has to happen
> every time a new element is added), so I'm trying to nip the problem
> from the source by instructing Core to only publish unconfirmed
> bitcoin transactions.
>
> According to
> https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra
> <https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra>
> , it is not possible to configure this from a configuration or
> command-line option. The source code must directly be edited. But
> since the codebase has changed greatly, the proposed solution no
> longer works.
>
> ----
>
> So basically, I know that something inside
> src/zmq/zmqnotificationinterface.cpp needs to be patched, but I'm not
> sure which function, or how to do it. Because I only need unconfirmed
> transactions to be published on ZeroMQ rawtx and not confirmed ones,
> it's one of the following functions that I need to patch for my own build:
>
> CZMQNotificationInterface::TransactionRemovedFromMempool
> void CZMQNotificationInterface::BlockDisconnected
>
> Both of these call NotifyTransaction() method which I assume fires a
> message on "rawtx" channel.
>
> In the Stack Exchange question I linked above, Jonas Schnelli
> suggested adding an `if (!pblock)` check, but that was several years
> ago and the function he was referencing no longer exists.
>
> But I still wonder if the pblock check is still applicable in the
> present day (i.e. if it's indicating a block the transaction is inside).
>
> - Ali Sherief
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
bitcoin-dev@lists•linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[-- Attachment #2: Type: text/html, Size: 8243 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [bitcoin-dev] Trying to patch Core ZMQ "rawtx" topic to only publish unconfirmed transactions: How?
2021-11-29 14:10 ` LORD HIS EXCELLENCY JAMES HRMH
@ 2021-11-29 14:13 ` Ali Sherief
0 siblings, 0 replies; 5+ messages in thread
From: Ali Sherief @ 2021-11-29 14:13 UTC (permalink / raw)
To: bitcoin-dev
[-- Attachment #1: Type: text/plain, Size: 5521 bytes --]
Theoretically that would be the desired outcome for me but this change is going to be implemented as part of a casino which must display the status of new deposits that are made, even when they are unconfirmed to now. Hence why I need to receive the unconfirmed messages.
- Ali Sherief
On Mon, Nov 29, 2021 at 5:10 PM, LORD HIS EXCELLENCY JAMES HRMH <willtech@live•com.au> wrote:
> Wasn't this already not a problem because you can check if it was confirmed? The transaction is not finalised in the mempool it is just speculation of a transaction, so it makes sense to emit when the transaction is confirmed. Just already check..
>
>> It appears that the ZeroMQ topic I'm listening to, "rawtx", not only
>> emits a raw transaction when it appears on the mempool, but once it's
>> already confirmed too.
>
> KING JAMES HRMH
> Great British Empire
>
> Regards,
> The Australian
> LORD HIS EXCELLENCY JAMES HRMH (& HMRH)
> of Hougun Manor & Glencoe & British Empire
> MR. Damian A. James Williamson
> Wills
>
> et al.
>
> Willtech
> www.willtech.com.au
> www.go-overt.com
> duigco.org DUIGCO API
> and other projects
>
> m. 0487135719
> f. +61261470192
>
> This email does not constitute a general advice. Please disregard this email if misdelivered.
> ---------------------------------------------------------------
>
> From: bitcoin-dev <bitcoin-dev-bounces@lists•linuxfoundation.org> on behalf of 0xB10C via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org>
> Sent: Monday, 29 November 2021 8:32 PM
> To: Ali Sherief <ali@notatether•com>; Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation.org>
> Subject: Re: [bitcoin-dev] Trying to patch Core ZMQ "rawtx" topic to only publish unconfirmed transactions: How?
>
> Hi Ali,
>
> I've run into this multiple times myself. I've opened a draft PR [0]
> adding a rawmempooltx publisher.
>
> You're right. In zmq/zmqnotificationinterface.cpp the
> CZMQNotificationInterface is notified about TransactionAddedToMempool.
> Currently, this calls NotifyTransaction() (the publisher with the rawtx
> topic) and NotifyTransactionAcceptance() (the publisher with the
> sequence topic)[1]. I've added a call to a new
> NotifyMempoolTransaction() function (the publisher with the rawmempooltx
> topic).
>
> I'd find a mempool transaction publisher with both the raw transaction
> and transaction fee useful too. However, this requires changes to the
> chain notifications in interfaces/chain.h.
>
> [0]: https://github.com/bitcoin/bitcoin/pull/23624
> [1]:
> https://github.com/bitcoin/bitcoin/pull/23624/files#diff-ac4b2d3a8de2c4dd41ad9d75505ea6ce4dc87a476710a9ebee8acf9bebf5cca2L146-L148
>
> Best,
> 0xB10C
>
> On 11/26/21 5:56 PM, Ali Sherief via bitcoin-dev wrote:
>>
>> This has also been posted on Bitcointalk
>> forum: https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261
>> <https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261> I
>> have republished it here hoping someone more knowledgeable can post
>> some insight about this.
>> ----
>> It appears that the ZeroMQ topic I'm listening to, "rawtx", not only
>> emits a raw transaction when it appears on the mempool, but once it's
>> already confirmed too.
>>
>> This messes with my software, causing it to add txids, addresses, etc.
>> a second time inside arrays (this means that the same transaction is
>> received twice in total).
>>
>> Array de-duping is not a viable solution long-term (because the array
>> will quickly grow to be big eventually and then this has to happen
>> every time a new element is added), so I'm trying to nip the problem
>> from the source by instructing Core to only publish unconfirmed
>> bitcoin transactions.
>>
>> According to
>> https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra
>> <https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra>
>> , it is not possible to configure this from a configuration or
>> command-line option. The source code must directly be edited. But
>> since the codebase has changed greatly, the proposed solution no
>> longer works.
>>
>> ----
>>
>> So basically, I know that something inside
>> src/zmq/zmqnotificationinterface.cpp needs to be patched, but I'm not
>> sure which function, or how to do it. Because I only need unconfirmed
>> transactions to be published on ZeroMQ rawtx and not confirmed ones,
>> it's one of the following functions that I need to patch for my own build:
>>
>> CZMQNotificationInterface::TransactionRemovedFromMempool
>> void CZMQNotificationInterface::BlockDisconnected
>>
>> Both of these call NotifyTransaction() method which I assume fires a
>> message on "rawtx" channel.
>>
>> In the Stack Exchange question I linked above, Jonas Schnelli
>> suggested adding an `if (!pblock)` check, but that was several years
>> ago and the function he was referencing no longer exists.
>>
>> But I still wonder if the pblock check is still applicable in the
>> present day (i.e. if it's indicating a block the transaction is inside).
>>
>> - Ali Sherief
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[-- Attachment #2: Type: text/html, Size: 8623 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [bitcoin-dev] Trying to patch Core ZMQ "rawtx" topic to only publish unconfirmed transactions: How?
@ 2021-11-27 13:42 Prayank
0 siblings, 0 replies; 5+ messages in thread
From: Prayank @ 2021-11-27 13:42 UTC (permalink / raw)
To: ali; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 777 bytes --]
Hi Ali,
Not sure if this is exactly what you are looking for but maybe trying to solve this I might also learn few things:
Save zmqpubsequence=tcp://127.0.0.1:28332 in bitcoin.conf
Run bitcoind
Run this python script: https://pastebin.com/raw/tNp2x5y3
You will see results like this:
https://i.imgur.com/xKzFJbl.png
https://i.imgur.com/gpsTTHZ.png
A - Accepted, C- Connect (block) and R- Removal in the above screenshots
If you are looking for unconfirmed transactions printed in sequence I think this should help. Since transactions can be printed twice (accept,remove) in this case as well, python script can be modified to manage this IMO.
Other alternatives can be debug=mempool and reading debug.log for changes without polling.
--
Prayank
A3B1 E430 2298 178F
[-- Attachment #2: Type: text/html, Size: 1465 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-11-29 14:14 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-26 16:56 [bitcoin-dev] Trying to patch Core ZMQ "rawtx" topic to only publish unconfirmed transactions: How? Ali Sherief
2021-11-29 9:32 ` 0xB10C
2021-11-29 14:10 ` LORD HIS EXCELLENCY JAMES HRMH
2021-11-29 14:13 ` Ali Sherief
2021-11-27 13:42 Prayank
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox