It is frustrating and annoying to be told to get to the point, especially when you believe that such a comment is pointless. However, before concluding that your manager has topped out on the egocentric scale, it is important to take an honest look at the answers you provide.
You mentioned that you do not wander all over the place, but even a little wandering may be too much for your manager. He obviously prefers communication that is brief, direct, and to-the-point. The closer your communication style is to his, the more he will appreciate your comments, as well as appreciate you.
However, if you truly believe that your comments to him are clear and succinct, then it is time for more assertive action. Your manager's behaviors may be based on his need to flex his power in front of you.
The next time he tells you to get to the point, use a businesslike tone, address him by name, and tell him that everything you are saying is important to accurately answer his question. Then stop talking. The result is that he is likely to give you a hearing. If you then provide some first-rate information, you can score points on many levels.
| 
