Dear Jameson,
Thank you for your questions. Answers inline below:
You've made many salient points, Shaolin, though I have a few questions:
1) How well does this model work under adversarial conditions? Fair point about signaling not being reliable, though it seems more vague in terms of safety given that you can't actually know what percentage of hashrate that is /not/ signaling for the soft fork has taken the necessary precautions to avoid mining an invalid block and potentially causing a hard fork. It's probably safe to say that if a flag-day soft fork is activated, there will be at least a few parties who will attempt to trigger a chain fork by crafting transactions that are valid via non-fork rules but invalid via the soft fork rules.
In a well designed soft fork, transactions under the old rules are non-standard by default and will not propagate or be mined. A miner would have to deliberately include the invalid transaction in a block and mine it. The invalid block would be rejected by the network costing the miner block reward and fees.
If >51% of the hashrate does not upgrade or does not take steps to protect themselves from invalid blocks, they will fork if someone produces an invalid block. Game theory suggests the incentive for those who do not wish to participate, would be to do so safely. There is no incentive to allow an attacker to cause you to split off from the network and it is trivial to prevent it.
There is a valid concern about "spy" mining and I cited a previous incident with BIP66 activation and we should be working towards solutions that remove the incentive to spy mine. "Weak blocks", where miners propagate their proposed blocks before solving the PoW may provide better incentives against spy mining, while delivering more (~no propagation delay and full validation, and thus more security).
2) If the flag day soft fork is activated with only a minority of hashrate support + safely opted-out hashrate, isn't it possible for the rest of miners to coordinate orphaning any soft fork compatible blocks to kill the soft fork chain? This would be a major difference from a miner-activated soft fork, correct? Unless perhaps many miners colluded to signal soft fork support while not actually supporting it...
The basic assumption in the Bitcoin system is that miners will remain honest because it is in their economic interest to do so. Of course 51% of the hashrate can censor the minority hash by orphaning blacklisted transactions or blocks. I am fairly certain it would be considered an attack by as well as being very conspicuous. A 51% attack would likely cause a dramatic loss in confidence in the Bitcoin system and adversely affect price. It is reasonable to assume miners would not do that because mining has to remain profitable. Additionally, such a scenario would draw much ire from users who may escalate demands for a PoW change.
It is assuming good-faith and that miners would not want to deny people the ability to opt into something they wanted. All that is required of miners is to upgrade their border node. Miners should update their software anyway for security reasons.
3) In terms of complexity for mining pool operators, how well does this model scale if there are N soft forks and the pool doesn't want to opt-in to any of them? Couldn't this result in those pool operators having to run not just one border node, but a multitude of "chained" border nodes if the soft forks are spread across different software implementations?
While BIP9 allows for 29 parallel deployments I think it is unrealistic to expect there would be such a high number of active parallel deployments at any one time: History shows soft forks take a minimum of 6 months design, consensus building, coding and testing before deployment. With such a high bar, I do not envisage more than a couple of parallel deployments at any given time. I also do not envisage "conflicting" soft forks, as that would not meet consensus from the technical community on the basis of safety and sanity. In any case, the deployment strategy of each soft fork should be considered on a case by case basis.
It seems to me that this type of user-driven approach would preferably be coupled with assurances from major Bitcoin wallets / exchanges / payment processors that they will not honor coins from a chain fork that results from invalid spends of outputs encumbered by soft fork rules. Though on the other hand, I don't see such an assurance being possible given that exchanges have an incentive to take the first mover advantage in listing a new coin.
Soft fork consensus proposals should be sane, uncontroversial and have a reasonably high bar in terms of technical consensus as we have seen with other soft forks to date. There is an implicit assumption in my text, that the decision to deploy a soft fork (regardless of the activation method) is based on a reasonable expectation that users will make use of the new feature. Hashrate signalling is not a vote, but a coordination trigger. Soft forks are backwards compatible and opt-in; so long as they are well written and bug free, users should at worst, be agnostic towards them because they have a choice whether to safely use the new feature or not, without preventing others' enjoyment of the feature. A controversial or unreasonable soft fork would not gain traction and I believe it would be fairly self evident.
In short, I do expect wide ecosystem collaboration as part of any deployment strategy, both hashrate or flag day based.
Many thanks for taking the time to read over and consider my thoughts and proposal. I would be happy to discuss more if you have any further questions or suggestions.