Does increased OP_RETURN demand bias mempool-based fee estimation for ordinary payments (e.g., different elasticity/arrival patterns vs monetary txs)?
No, OP_RETURN transactions are not treated specially (positive or negative) when it comes to fees. All transactions are treated equally, with the only distinguishing factor being how much they are willing to pay to in fees. Regardless of whether the transaction has OP_RETURNs, inscriptions, or payments, all transactions are competing for the same resource: space in a block. Fee estimation needs to account for all demand of block space equally.
Could sustained OP_RETURN demand crowd out smaller monetary payments and reduce their on-chain viability, even if total fees rise?
Certainly, and we have seen similar behavior occur before with inscriptions. It’s all just supply and demand. The supply of block space is limited, so when the demand increases, whether that’s payments, OP_RETURNs, inscriptions, or something else, the price of that block space will increase as well. Ultimately, it comes down to whoever is willing to pay a higher price. Maybe those making payments are willing to pay more than those making OP_RETURNs. Maybe vice versa.
However, it should be noted that payment transactions are often smaller, and there are a number of coin selection strategies which can optimize for creating smaller transactions. Given that, it is probable that transactions that are simply payments will pay smaller absolute fees than transactions containing OP_RETURNs, even if the feerate is higher.
Is there analysis or simulation showing that higher OP_RETURN usage does not degrade settlement reliability or lead to more volatile fee dynamics that hurt adoption?
Past behavior on the network has shown that even when there is significant demand for block space that adoption has still increased. Just look at how many more people are into using Bitcoin since things like inscriptions have happened. In general, adoption has tended to go up regardless of what is happening on chain.
We also know from previous instances of high transaction volume that it can be hard to predict how much to pay in fees to get into a block. This is likely true regardless of whether that volume is from OP_RETURNs, inscriptions, payments, or something else. This may effect “settlement reliability” if what you mean by that is guessing how long it takes a transaction to be mined.
But there’s no reason to assume to increasing the OP_RETURN limit would result in increased transaction volume. The primary method of data insertion in the blockchain over the past several years has been via inscriptions. That method of data insertion is cheaper, and allows for more data, than OP_RETURNs. Increasing the OP_RETURN limit doesn’t change that, so those looking to insert large amounts of data are unlikely to switch to OP_RETURN. There is a very small set of users for which larger OP_RETURNs are interesting, but they would not have high (or really any appreciable amount of) transaction volume.