John Jay's correspondence is available at Liberty Fund. There is a lot of great stuff there.
Jay is a Founder, generally conceded as orthodox Christian. And he does offer some quotations that sound "Christian Nation" like. Certainly, Christian Nationalists have used (abused?) those quotations from Jay. Rev. Robert Jeffress recently used Jay's quotation to justify why he would vote for the "Christian" Rick Perry over the "non-Christian" Mormon Mitt Romney.
John Jay was a lifelong Anglican, a church warden, and was involved in the reformation of Anglicanism to Episcopalianism.
While investigating the exact religious beliefs of as many American Founders I could, I began to think very hard about the concept of creedalism, and that various Founders, in some way affiliated with churches connected to doctrinally orthodox creeds, could disbelieve at least some of what their churches officially endorsed.
In this debate over who exactly believed what, we look for smoking guns to "settle" the issue. Some use, in the absence of smoking gun quotations, formal affiliation to churches and their official doctrines, as shorthand for what exactly a Founder believed. Indeed, if one held an official position in said churches -- church warden, vestryman, godfather, etc. -- then one probably DID swear an oath to said church's official doctrines.
Enter the concept of an "Anglican Whig." According to Anglicanism's official doctrines, the King of England was both head of church AND state. And when Anglican church MEMBERS swore oaths they swore to remain loyal to the Crown. There was a term for the oath and doctrinal fundamentalists in the Anglican Church in late 18th Century America: Tories.
In order to claim America's Founders as orthodox Christians via their church affiliation, Christian Nationalists have to turn them into oath fundamentalists and argue something like if you are not an oath fundamentalist, you are a liar or a hypocrite. See for instance, Christian Nationalist Richard Gardiner making this argument.
Accordingly, because they violated their high church Tory oaths when they rebelled against the Crown, Anglican Whigs such as Washington, Jefferson, Madison, Henry, Jay and others were "liars when they swore to God to adopt the confessions of their churches when they became members of these churches" to use Dr. Gardiner's exact words. (Again, I don't make that judgment; Christian Nationalists like Dr. Gardiner do.)
John Jay, unlike Washington and Madison, and like Jefferson was fairly explicit in detailing what he believed, religiously, in his private letters. (Jefferson, the good Anglican he [not!], rejected every single tenet of orthodox Christianity.)
As I've written before, John Jay, though more apparently "biblical" and "Christian" than other Founders, was no "oath fundamentalist" regarding his Anglicanism-Episcopalianism. Christian Nationalists may enjoy Jay's letter to Samuel Miller, Feb. 18, 1822, when he wrote:
"In forming and settling my belief relative to the doctrines of Christianity, I adopted no articles from creeds, but such only as, on careful examination, I found to be confirmed by the Bible."
"No creed but the Bible" may make Jay sound like a good late 18th Century Quaker, but it does not represent what orthodox Anglicanism or Episcopalianism stood for at that time. Anglicanism, and then Episcopalianism, in its official doctrines endorsed not just "the Bible" as revelation, but also the Athanasian Creed and 39 Articles.
And indeed, when the Bible is unmoored from orthodox Trinitarian creeds, heresy in the form of biblical non-Trinitarianism inevitably results as it did in Jay's case. In his private letters, Jay's endorsement of "the Bible" but no "creeds" led him to seem not only utterly unconcerned with Anglican doctrine, but also to doubt the Trinity, or at least the understanding of the Trinity as represented by the Athanasian Creed (what his church, what he took official oaths to, endorsed). From the same letter to Samuel Miller, Feb. 18, 1822:
It appeared to me that the Trinity was a Fact fully revealed and substantiated, but that the quo modo was incomprehensible by human Ingenuity. According to sundry Creeds, the divine Being whom we denominate the second Person in the Trinity had before all worlds been so generated or begotten by the first Person in the Trinity, as to be his coeval, coequal and coeternal Son. For proof of this I searched the Scriptures diligently -- but without Success. I therefore consider the Position of being at least of questionable Orthodoxy.