|Comments from this interviewer regarding the appropriateness of your apparel could be appropriate or inappropriate, depending upon a number of factors. |
For example, if his comments referred directly or indirectly to aspects of your apparel that relate to your religion, then his words were totally inappropriate. In fact, they could even be construed as discriminatory.
At the other end of the continuum, his comments may have been little more than a suggestion to help you land a job in the future. In the context of a job interview, apparel that is overly casual or just plain weird can prevent an otherwise qualified individual from getting that second interview or job offer.
The fact that you have had many interviews and no one has given you this feedback does not really matter. It could even be argued that your attire is the reason you have had so many interviews, but apparently no offers.
Either way, the best guide is to wear clothing that is a notch more formal than what you would expect to find among employees who work at the company. A little checking before the interview is often worthwhile. If you have doubts about a particular article of clothing, set it aside to prevent similar doubts about you.