If you are in the Seattle or Vancouver area, try to make it to the Victoria Island. It is closer to visit from Seattle, and you can take a morning ferry, which drops you off around 10:00 am. The last one going back is around 6:00 pm, but check the ferry schedule.
View of Victoria Harbour from the ferry
From Victoria Harbour, you can take a bus to Buschard Gardens. We decided to rent a car, as we were staying the night, and didn’t need to rush back to the docks. Parking is free, but there is an admission fee, and the hours vary depending on the season. Since we were there in June, the hours were opened longer, and we were also able to make a 3:30 pm reservations for Afternoon Tea.