One Nation voters and older people are much more likely to be concerned about the number of migrants in Australia.