From Where Does Idea of Christian Nationalism Come?

 To quote the Library of Congress website:

It is no exaggeration to say that on Sundays in Washington during the administrations of Thomas Jefferson (1801-1809) and of James Madison (1809-1817) the state became the church. Within a year of his inauguration, Jefferson began attending church services in the House of Representatives. Madison followed Jefferson's example, although unlike Jefferson, who rode on horseback to church in the Capitol, Madison came in a coach and four. Worship services in the House--a practice that continued until after the Civil War--were acceptable to Jefferson because they were nondiscriminatory and voluntary. Preachers of every Protestant denomination appeared. (Catholic priests began officiating in 1826.) As early as January 1806 a female evangelist, Dorothy Ripley, delivered a camp meeting-style exhortation in the House to Jefferson, Vice President Aaron Burr, and a "crowded audience." Throughout his administration Jefferson permitted church services in executive branch buildings. The Gospel was also preached in the Supreme Court chambers.

Jefferson's actions may seem surprising because his attitude toward the relation between religion and government is usually thought to have been embodied in his recommendation that there exist "a wall of separation between church and state." In that statement, Jefferson was apparently declaring his opposition, as Madison had done in introducing the Bill of Rights, to a "national" religion. In attending church services on public property, Jefferson and Madison consciously and deliberately were offering symbolic support to religion as a prop for republican government.