ChangeOperator should be used with matrices with the same sparsity pattern or in such a way that it does not trigger a reallocation. You can enforce the same pattern by augmenting your varf with a penalized term like 1e-16*(terms you have in your other varf but not on this one). Or you can assemble an initial dummy Mat with all your terms, and then reassemble the proper varf.