The Governor’s Residence was built in 1909 for the Hartford physician and industrialist George C.F. Williams. Situated at the corner of Prospect and Asylum Avenues in the City of Hartford, the house was designed in Georgian Revival style by the Boston-based architectural firm of Andrews, Jacques & Rantoul. In 1916, the Hartford architects Smith & Bassett designed the north and south wing additions.

A member of the Williams family lived in the house until 1940 when it became too expensive to maintain. The house then stood vacant for several years, though it was briefly used as a convalescent unit by the Hartford Hospital.

The idea of having a residence for the chief executive of the state was first suggested in 1943 by Francis S. Murphy, the publisher of the now-defunct Hartford Times. A search committee was formed, and after reviewing 30 properties the committee decided that the old Williams estate, even though it was in need of repairs, would serve admirably as the official residence. The state acquired it by paying $39,500 to the Williams estate in lieu of back taxes. Another $75,000 was allocated for renovations that were undertaken by Hartford architect Carl J. Malmfeldt. The work took over two years to complete.

The first occupants were Governor Raymond E. Baldwin and his family, who moved in during September 1945. The original restoration was less than successful and featured mismatched furniture in a wide variety of styles. According to Governor Baldwin, the legislature had furnished his house politically, parceling out patronage to merchants in various towns by allowing each one to purchase some of the furniture. The results were such that Mrs. Baldwin told her husband that she would not move in. A neighbor, Beatrice Fox Auerbach, owner of G. Fox & Co., came to the rescue. She removed the offending furniture and brought in her staff of decorators to furnish the Residence with 18th century antiques and handcrafted reproductions. The finest pieces were made by Hartford cabinet maker Nathan Margolis, whose shop had already executed prestigious commissions for John D. Rockefeller, the DuPonts, and Yale University. Soon after the Residence was furnished, the governor invited the public to tea and 5,000 people showed up (the governor, however, had strained his back and was unable to leave his bed to greet the guests).