|Saying that you see your employees when you arrive and leave is similar to saying that you read a book by looking at the front and back covers. You are missing the best part.
Although you said you cannot get your job done if you spend time with your employees, it could be argued that you cannot get your job done if you don't spend time with them. Having "face-time" with your employees is a key part of your job. After all, managing includes leading, coaching, guiding, and following-up with your team. If you are permanently planted behind four walls, these essential managerial functions are locked out.
As for not getting the rest of your job done, you need not spend huge amounts of time wandering around. A few minutes each day can more than suffice. Secondly, if you look carefully at how you spend your hours at work, you will probably find more than a few minutes that could be spent more valuably with your team than on minor chores or extraneous non-work activities. Either way, with a little more time management, you should be able to easily manage a little more time for your employees.