* Re: [Bitcoin-development] performance testing for bitcoin [not found] ` <CAEC9zAYrMHHEyyTx1QVHoGSJU3fFypB0Hx4K-VFoUn0hp4Z7JA@mail.gmail.com> @ 2012-10-03 17:38 ` Ian Miers 2012-10-03 17:43 ` Jeff Garzik 0 siblings, 1 reply; 5+ messages in thread From: Ian Miers @ 2012-10-03 17:38 UTC (permalink / raw) To: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 836 bytes --] Whats the best way to get performance numbers for modifications to bitcoin ? Profiling it while running on testnet might work, but that would take a rather long time to get data. Is there anyway to speed this up if we only needed to provide relative performance between tests. (in a sense a fast performance regression test). At least in theory, one possibility would be to replay real bitcoin transactions to a test-net-in-a-box network (of maybe a few nodes) that has the real blockchain history loaded into it but then operates at a far reduced hash difficulty in order facilitate running quickly. Are there existing techniques/scripts for some kind of perf testing with traffic that at least approximates real world traffic ? The test infrastructure that ships with bitcoin does not appear to provide this. Thanks, Ian > > [-- Attachment #2: Type: text/html, Size: 1084 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Bitcoin-development] performance testing for bitcoin 2012-10-03 17:38 ` [Bitcoin-development] performance testing for bitcoin Ian Miers @ 2012-10-03 17:43 ` Jeff Garzik 2012-10-03 17:57 ` Ian Miers 0 siblings, 1 reply; 5+ messages in thread From: Jeff Garzik @ 2012-10-03 17:43 UTC (permalink / raw) To: Ian Miers; +Cc: bitcoin-development On Wed, Oct 3, 2012 at 1:38 PM, Ian Miers <imiers1@jhu•edu> wrote: > Whats the best way to get performance numbers for modifications to bitcoin ? > Profiling it while running on testnet might work, but that would take a > rather long time to get data. > Is there anyway to speed this up if we only needed to provide relative > performance between tests. (in a sense a fast performance regression test). You have to be specific about what you're measuring, because "performance" is vague. You can measure many aspects of blockchain performance by importing blocks via -loadblock=FILE. Other performance measurements like "how fast does a block relay through the network" cannot be as easily measured. -- Jeff Garzik exMULTI, Inc. jgarzik@exmulti•com ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Bitcoin-development] performance testing for bitcoin 2012-10-03 17:43 ` Jeff Garzik @ 2012-10-03 17:57 ` Ian Miers 2012-10-04 9:55 ` Joel Joonatan Kaartinen 0 siblings, 1 reply; 5+ messages in thread From: Ian Miers @ 2012-10-03 17:57 UTC (permalink / raw) To: Jeff Garzik; +Cc: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 1327 bytes --] Script evaluation performance was what I was primarily concerned with. I'm fooling around with adding some new instruction types. The tricky part is that to test how that effects performance, you need to be able to intersperse transactions with the new instructions with existing ones. For accuracy, you'd like your simulated traffic to at least approximate the real world traffic. Also, is there any bench-marking / instrumentation in bitcoind ? Ian On Wed, Oct 3, 2012 at 1:43 PM, Jeff Garzik <jgarzik@exmulti•com> wrote: > On Wed, Oct 3, 2012 at 1:38 PM, Ian Miers <imiers1@jhu•edu> wrote: > > Whats the best way to get performance numbers for modifications to > bitcoin ? > > Profiling it while running on testnet might work, but that would take a > > rather long time to get data. > > Is there anyway to speed this up if we only needed to provide relative > > performance between tests. (in a sense a fast performance regression > test). > > You have to be specific about what you're measuring, because > "performance" is vague. > > You can measure many aspects of blockchain performance by importing > blocks via -loadblock=FILE. > > Other performance measurements like "how fast does a block relay > through the network" cannot be as easily measured. > > -- > Jeff Garzik > exMULTI, Inc. > jgarzik@exmulti•com > [-- Attachment #2: Type: text/html, Size: 1944 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Bitcoin-development] performance testing for bitcoin 2012-10-03 17:57 ` Ian Miers @ 2012-10-04 9:55 ` Joel Joonatan Kaartinen 2012-10-04 16:31 ` Ian Miers 0 siblings, 1 reply; 5+ messages in thread From: Joel Joonatan Kaartinen @ 2012-10-04 9:55 UTC (permalink / raw) To: Ian Miers; +Cc: bitcoin-development For script evaluation benchmarking, I don't think just a good approximation of real-world traffic is enough. You really need to benchmark the worst case scenarios, otherwise you could be creating a DoS vulnerability. - Joel ke, 2012-10-03 kello 13:57 -0400, Ian Miers kirjoitti: > Script evaluation performance was what I was primarily concerned > with. I'm fooling around with adding some new instruction types. > The tricky part is that to test how that effects performance, you need > to be able to intersperse transactions with the new instructions with > existing ones. For accuracy, you'd like your simulated traffic to at > least approximate the real world traffic. > > > > > Also, is there any bench-marking / instrumentation in bitcoind ? > > > Ian > On Wed, Oct 3, 2012 at 1:43 PM, Jeff Garzik <jgarzik@exmulti•com> > wrote: > On Wed, Oct 3, 2012 at 1:38 PM, Ian Miers <imiers1@jhu•edu> > wrote: > > Whats the best way to get performance numbers for > modifications to bitcoin ? > > Profiling it while running on testnet might work, but that > would take a > > rather long time to get data. > > Is there anyway to speed this up if we only needed to > provide relative > > performance between tests. (in a sense a fast performance > regression test). > > > You have to be specific about what you're measuring, because > "performance" is vague. > > You can measure many aspects of blockchain performance by > importing > blocks via -loadblock=FILE. > > Other performance measurements like "how fast does a block > relay > through the network" cannot be as easily measured. > > -- > Jeff Garzik > exMULTI, Inc. > jgarzik@exmulti•com > > > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev > _______________________________________________ Bitcoin-development mailing list Bitcoin-development@lists•sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Bitcoin-development] performance testing for bitcoin 2012-10-04 9:55 ` Joel Joonatan Kaartinen @ 2012-10-04 16:31 ` Ian Miers 0 siblings, 0 replies; 5+ messages in thread From: Ian Miers @ 2012-10-04 16:31 UTC (permalink / raw) To: Joel Joonatan Kaartinen; +Cc: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 2887 bytes --] Agreed, you need to consider DoS attacks. I have that in my mind under security and not performance. I'd imagine the test methodology would be completely different since you no longer are attempting to simulate real world traffic. Ian On Thu, Oct 4, 2012 at 5:55 AM, Joel Joonatan Kaartinen < joel.kaartinen@gmail•com> wrote: > For script evaluation benchmarking, I don't think just a good > approximation of real-world traffic is enough. You really need to > benchmark the worst case scenarios, otherwise you could be creating a > DoS vulnerability. > > - Joel > > ke, 2012-10-03 kello 13:57 -0400, Ian Miers kirjoitti: > > Script evaluation performance was what I was primarily concerned > > with. I'm fooling around with adding some new instruction types. > > The tricky part is that to test how that effects performance, you need > > to be able to intersperse transactions with the new instructions with > > existing ones. For accuracy, you'd like your simulated traffic to at > > least approximate the real world traffic. > > > > > > > > > > Also, is there any bench-marking / instrumentation in bitcoind ? > > > > > > Ian > > On Wed, Oct 3, 2012 at 1:43 PM, Jeff Garzik <jgarzik@exmulti•com> > > wrote: > > On Wed, Oct 3, 2012 at 1:38 PM, Ian Miers <imiers1@jhu•edu> > > wrote: > > > Whats the best way to get performance numbers for > > modifications to bitcoin ? > > > Profiling it while running on testnet might work, but that > > would take a > > > rather long time to get data. > > > Is there anyway to speed this up if we only needed to > > provide relative > > > performance between tests. (in a sense a fast performance > > regression test). > > > > > > You have to be specific about what you're measuring, because > > "performance" is vague. > > > > You can measure many aspects of blockchain performance by > > importing > > blocks via -loadblock=FILE. > > > > Other performance measurements like "how fast does a block > > relay > > through the network" cannot be as easily measured. > > > > -- > > Jeff Garzik > > exMULTI, Inc. > > jgarzik@exmulti•com > > > > > > > ------------------------------------------------------------------------------ > > Don't let slow site performance ruin your business. Deploy New Relic APM > > Deploy New Relic app performance management and know exactly > > what is happening inside your Ruby, Python, PHP, Java, and .NET app > > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > > http://p.sf.net/sfu/newrelic-dev2dev > > _______________________________________________ Bitcoin-development > mailing list Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > > [-- Attachment #2: Type: text/html, Size: 3979 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-10-04 16:31 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CAEC9zAbHO9y7Go4FhZFyOCKdm1js_eKCoz7iaOCh1Wu=B9OASw@mail.gmail.com>
[not found] ` <CAEC9zAYrMHHEyyTx1QVHoGSJU3fFypB0Hx4K-VFoUn0hp4Z7JA@mail.gmail.com>
2012-10-03 17:38 ` [Bitcoin-development] performance testing for bitcoin Ian Miers
2012-10-03 17:43 ` Jeff Garzik
2012-10-03 17:57 ` Ian Miers
2012-10-04 9:55 ` Joel Joonatan Kaartinen
2012-10-04 16:31 ` Ian Miers
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox