Former Tory minister Sir Jacob Rees-Mogg has said his new reality show is unlikely to “change many people’s minds” about him.