Almost since the beginning, St. Andrews has had a library.

Records show that from approximately 1840 to 1961, there was a volunteer library but no permanent home. Estelle Caughey, in 1962, decided to change this and set out to give St. Andrews a real Library. She took courses, recruited other volunteers, sought book donations, found a permanent home for a library, established regular hours, and even ran children's programs. Disaster struck in 1973 when the building the library was in burned to the ground destroying almost the entire collection. Thanks to Mrs. Caughey and the other dedicated volunteers who worked tirelessly, the library was again in operation less than a year later.

Meanwhile, Henry Phipps and Sarah Juliette Ross had left a considerable Trust for a Library and Museum. They had purchased Chestnut Hall for the Museum and left instructions in their will for a library to be erected on the same property, and for a Ross Trust to be established to help support these two institutions. Building was completed in 1977, and in 1982 the Ross Memorial Library became one of the current 61 provincial libraries.