You may take the trip each way with a 2 hour layover in Silverton or take a bus either way.  Using the bus one way allows you more time in Silverton.
Durango,CO railroad trip