Every time I make or receive a phone call it goes straight to loud speaker

I have tried restarting my phone, restoring phone to factory settings, checking for updates, but nothing has helped fix the problem.