The recalls are really minor so those wouldn't bother me at all. I can't tell you if the pricing is worth it without knowing all the details and specs, the original MSRP, etc. But there are people who can run the VIN, and see what the wholesale value of the vehicle is.
Have you tried calling any junkyards / salvage yards around you? That's probably the cheapest way to find a replacement. If you are really luck you can find one in your color.
The only way to know is to give it a try. Be prepared to walk out though and leave the dealership. Tell them what you want, and when they say they can't do it.... leave. Wait an hour and they usually will call you back and tell you that they can do it, or at least do better than what they said.
Have you checked all the fuses? Your gut could be right. After that, yah I'd assume it was either a bad switch inside or the mirror motor. But, the mirror motor might be kind of expensive to replace just on a hunch without proving it.