Before the residence was built, the Wardens of Kingston Penitentiary and their families lived in apartments provided within the walls of the penitentiary. This wing also housed the administration offices of the facility.

Built entirely by convict labour under the supervision of staff Trade Instructors, the house took nearly 3 years to build. The cost? Approximately $ 9,000.00 (in 1873 dollars). Designed in the Italianate Vernacular style by Acting Penitentiary Architect, Henry H. Horsey, it was constructed using a combination of local Kingston limestone quarried on the penitentiary reserve, with Ohio Sandstone accents. In the late 19th century, the residence became known as "Cedarhedge" in reference to the extensively manicured cedar hedges that once lined the driveway.

By all accounts, life at Cedarhedge was a luxurious one. At its peak, the property boasted apple orchards, greenhouses, a grape-vinery, and a conservatory. Inmate gardeners tended to the extensive terraced grounds each day, and some of the resident families are known to have had their own non-inmate servants within the home. The importance of the house is exemplified by at least two postcards that were produced for sale to the general public around 1900.