I think you're right about it being primarily the listener's preference, and not based on your relative skill. I played a gig recently where I doubled on alto and tenor. Afterwards I asked a number of people which horn they preferred. The response was completely mixed, but almost everyone had a pretty strong opinion one way or the other. My conclusion was that some people like tenor and others like alto. It's just their preference.