|It's somewhat ironic that you do not want to join your co-workers at a team-building session. Perhaps the presence of feelings such as yours induced management to implement such sessions in the first place.
Nonetheless, the prospect of playing some "little games" is not exactly captivating for most employees. People are very busy at work right now, and if they are going to be pulled away from the job for some educational experiences, it is important to sense that such exercises will be more than fun and games.
The fact is that many of today's best training programs are highly experiential and rely heavily on hands-on activities for the participants. This approach is particularly important in the team-building arena, since lectures or other traditional classroom strategies provide minimal opportunity for the level of group interaction, coordination, and problem-solving that is so important in the team-building process.
Granted that your company needs to provide more communication regarding the training program and its objectives, you would be well-advised to attend. If you turn your back on the company's training, the company is more likely to turn its back on you.