In 1818 the vestry began to seek subscriptions for a new brick building to replace the old stone one. Among the subscribers was James Buchanan, later President of the United States.
Although he himself was a Presbyterian, his niece Harriet Lane was a member of Saint James. In 1819 the old building was razed to the ground and the site cleared.