The Chiayi Bus Rapid Transit BRT (http://www.cibus.com.tw/) system has three lines, with the main line linking the HSR and TRA railway stations. The trip between the two takes around 30 minutes $40, exact change - no charge if going directly between the HSR & TRA stations, but note that the BRT stop is on the rear side of the TRA station, so you need to take the pedestrian bridge to cross to the other side. The location of the TRA station is not obvious from the BRT stop. If coming from the direction of the HSR railway, after you get off at the stop, cross to the other side of the street at the intersection you have just passed. The "back station" area is under construction - bear to the right and you should find the pedestrian bridge.
Buses have announcements and signage in English as well as Chinese.
Taxis are plentiful, especially around the train stations, but no English is spoken.
Buses to Alishan costs $221NT one way, and can be bought at the bus station next to the train station. The ride takes 2.5 hours and can be quite windy.