I think it makes more sense to construct entirely separate filters for the different types of elements and allow clients to download only the ones they care about. If there are enough elements per filter, the compression ratio shouldn't be much worse by splitting them up. This prevents the exponential blowup in the number of filters that you mention, Johan, and it works nicely with service bits for advertising different filter types independently.

So if we created three separate filter types, one for output scripts, one for input outpoints, and one for TXIDs, each signaled with a separate service bit, are people good with that? Or do you think there shouldn't be a TXID filter at all, Matt? I didn't include the option of a prev output script filter or rolling that into the block output script filter because it changes the security model (cannot be proven to be correct/incorrect succinctly).

Then there's the question of whether to separate or combine the headers. I'd lean towards keeping them separate because it's simpler that way.