Stored speed dial numbers can be easily removed, and contacts that have been deleted are automatically removed from the speed dial numbers.

Android 10 (One UI 2.0)
Step 1:  Open the  Phone App
Step 2:  Open the  Menu
Step 3:  Tap on  Speed dial numbers
Step 4:  Tap on  Remove
  1. Open the Phone App
  2. Open the Menu
  3. Tap on Speed dial numbers
  4. Tap on Remove

Samsung Instructions