You need people not only voters and siders. If you give up half of your population you are already halved as a nation. Even though a portion of the population don't want you in power they are still useful in other ways. Like a larger base paying tax, conscripting for the military, doing public...