Modern devices like smartphones and tablets do not have swap files. This design is chosen to ensure responsive, fluid UI that can avoid blocking on disk regardless of how much multi-tasking is done, but it creates ripples that impact everything else.
One implication of this is that on these devices, we cannot store all keys or transactions in memory forever. BIP 70 has an expiry field for PaymentRequests that we can use to allow us to eventually stop loading those keys into RAM - at that point payments to those keys would no longer be recognised. But there's no equivalent for refund addresses.
More generally, though we re-used the output structure to define the refund, we didn't (for some reason that I forgot) reuse PaymentDetails, even though the payment details for a refund are indeed PaymentDetails.
Though I am loathe to go back and redesign this part of BIP 70 so soon after we shipped v1, it seems to me like the refund feature may be hard to implement on phones if there's no time limit for when you can receive a refund. Otherwise a wallet has to be looking out for refunds for payments you may have made years ago. So perhaps we should add a new refund field that embeds a PaymentDetails structure instead of being just a list of outputs.
We could try and solve this problem some other way purely internally, by doing a kind of wallet-specific swapping process in which things like Bloom filters are calculated without all keys in them being held in memory at once (perhaps caching filters for old parts of the key chain on disk), so you can have "infinite" wallets, but eventually the huge Bloom filters that would result would hurt efficiency in other ways. So key expiry seems pretty fundamental to scalability.