Trivia: During the fight scene between Jimmy and Dalton, Jimmy slams a log against Dalton's chest. The log you see here is a real one. Marshall Teague thought the log was a breakable prop and kept slamming it into Patrick Swayze's chest with each take. By the end, Swayze's chest was covered in bruises.
Trivia: When Wade Garrett (Sam Elliot) first shows up at the Double Deuce, there is a fight out back by the delivery truck. Morgan (Terry Funk) says 'none of your business, Dad' to Wade. Terry Funk is two months older than Sam Elliot in real life.
Answer: As we see throughout the film, Dalton is not confrontational, and always takes the path of least violence. He wouldn't confront them, he would wait for them to attack him.