|Using your manager's terminology, patting an employee on the rear is an illegal use of the hands. And, there's a huge penalty for that type of conduct.
Although many managers today are using sports comments as a seemingly colorful and creative way to communicate, this approach can miss the individual needs of many of the employees. In a word, sports talk is not for everyone.
The larger problem with your manager is that he is not only unaware of the impact that his language and style are having on you and your associates, he is also unaware of at least one very basic issue at work these days, namely unwanted touching. And, that's probably not the only issue that is missing from his playbook.
You and some of your associates should see if he would be willing to discuss ways to help the department run more effectively. In the context of such a discussion, he could be told that some of the employees do not respond particularly well to comments laced with sports references. A few sharp words about the issue of sexual harassment are in order as well. He needs to understand that there is more to management than fun and games.