It is harder than you might think to decide on what kinds of gifts to give visitors because people think their gift will affect whether or not a visitor returns. I would think the most important purpose of a visitor gift is to point the visitor to Jesus first and your church second!
Many people want to have something engraved with a cutesie cliche like "Happy Church is a little slice of heaven" or "Come to Happy Church to see the light" but I would caution against that. I would also caution against something with just the name of your church on it. Point them to Jesus using God's words. They're far more meaningful and powerful than anything you can come up with. If the gift is too small to have scripture and the name of the church on it, ask yourself what the goal of the gift is and what God would want you to do? HE is the one who brings people back. And really... if they found their way to the church once, I think they'll be able to find it again.
One of my favorites are refrigerator magnets. They can be made very inexpensively, customize easily, and can share scripture and church information. I printed these on ink jet printer and fit about 12 on a sheet.
I also like the idea of mugs but they're pretty expensive. Paper cups? Just kidding...