Swedenborgs are an eccentric sect in Christendom because of their unorthodox Christology. Their Wiki entry states:
Swedenborg explicitly rejected the common explanation of the Trinity as a Trinity of Persons, which he said was not taught in the early Christian Church. Instead he explained in his theological writings how the Divine Trinity exists in One Person, in One God, the Lord Jesus Christ. Swedenborg also rejected the Protestant doctrine of salvation through faith alone, since he considered both faith and charity necessary for salvation, not one without the other. The purpose of faith, according to Swedenborg, is to lead a person to a life according to the truths of faith, which is charity.
The founder of the church was a bit "odd" to lightly put it:
At the age of fifty-six he entered into a spiritual phase, in which he experienced dreams and visions. This culminated in a spiritual awakening, where he claimed he was appointed by the Lord to write a heavenly doctrine to reform Christianity. He claimed that the Lord had opened his eyes, so that from then on he could freely visit heaven and hell, and talk with angels, demons, and other spirits. For the remaining 28 years of his life, he wrote and published 18 theological works, of which the best known was Heaven and Hell (1758) , and several unpublished theological works.
It was because of their heretical beliefs they were persecuted and sought refuge in America. The interesting thing about the Swedenborg's initial letter to Washington is that they explicitly identified their "heresy" to him:
Neither in this address can we, was it expected, enter into a detail of the profession of our faith; but we are free to declare that we feel ourselves among the number of those who have occasion to rejoice that the word literally is spiritually fulfilling; that a new and glorious dispensation or fresh manifestation of Divine Love hath commenced in our Land; when as there is but one Lord, so is his name becoming one throughout the earth; & that the power of Light or truth and righteousness is in an eminent Degree, universally prevailing, and even triumphing over the powers of Darkness; when Priestcraft & Kingcraft, those banes of human felicity, are hiding their diminished heads, and equality in State, as well as in Church, proportionally to mind, are considered the true criterion of the majesty of the people. -- Oh! Sir, could we, without being charged with adulation, pour out the fullness of our souls to the enlightened conduct of him who stands chief among the foremost of men, what a volume of truth might we deservedly offer to the name of Washington, on the altar of Liberty uncircumscribed! Allow us, by the first opportunity, to present to your Excellency, among other Tracts, the Compendium of the New Church, signified by the New Jerusalem, as the readiest mean to furnish you with a just idea of the Heavenly Doctrines.
That the Lord Jesus, whom alone we acknowledge as "the true God & eternal Life," will preserve you long to reign in the hearts of the people, and lastly to shine as a gem of the brightest lustre, a Star of the first magnitude of the unfading mansions above, is the fervent aspiration of your faithful citizens and affectionate Brethren. Done in behalf of the members of the Lord's New Church at Baltimore, this 22d Day of January 1793 -- 37.
In his response, there is not a shred of indication that Washington has any problems with their beliefs:
To the members of the New Church at Baltimore.
It has ever been my pride to mind the approbation of my fellow citizens by a faithful and honest discharge of the duties annexed to those Stations to which they have pledged to place me; and the dearest rewards of my Services have been those testimonies of esteem and confidence with which they have honored me. But to the manifest interpretation of an over-ruling Providence, and to the patriotic exertions of United America, are to be ascribed those events which have given us a respectable rank among the nations of the earth. --
We have abundant reason to rejoice that in this land the light of truth and reason has triumphed over the power of bigotry and superstition and that every person may here worship God according to the dictates of his own heart. In this enlightened Age & in this Land of equal liberty it is our boast, that a man's religious tenets, will not forfeit his protection of the Laws, nor deprive him of the right of attaining & holding the highest offices that are known in the United States.
Your Prayers for my present and future felicity were received with gratitude; and I sincerely wish, Gentlemen, that you may in your social and individual capacities, taste those blessings which a gracious God bestows upon the Righteous.
I don't want to read too much into GW's response. He probably didn't share their unorthodox Christology. Washington corresponded with many churches, the overwhelming majority of them orthodox Christian, and similarly gives them positive, approving, responses.
Peter Lillback constructs an argument from his correspondence which goes something like this: 1) Washington frequently corresponded with Christian churches, all of them orthodox. 2) Washington uses language which praises these orthodox Christian churches and the work they do, something no Deist could do. 3) Washington must have been Christian like the churches with whom he corresponded, not Deist.
The problem with his logic is, though most of the churches were orthodox Christian, not all of them were. Washington corresponded with Roman Catholics, Jews, the Swedenborgians, Freemasons, and the Universalist Church and gave similar positive responses. (Catholics are orthodox Christian, yet few pious orthodox Protestants of the Founding era thought Roman Catholicism was a legitimate path to God.)
We could just as logically conclude that Washington was simply creedally indifferent. What Gregg Frazer calls theistic rationalism (the religion of the key Founders like Washington) is really just a form of radical latitudinarianism which is mainly concerned that "religion" teaches there is an overriding Providence who governs the affairs of man and who will ultimately reward good and punish evil. The rest (things like the Trinity) are just unimportant points. The test of whether these religions are sound is whether they produce virtue. And since almost all of them do, they are almost all sound, even non-Judeo-Christian ones.
It's true that American Presidents generally, including orthodox Christian ones like President Bush, when they correspond with various religious sects try to use accommodating and accepting language. (Bush went so far as to call "Allah" a legitimate title for God when speaking to Muslims, something that really ticked off his base.) But that's only because they follow in the tradition that GW et al. established. Before Washington et al. changed things around, heads of state in Western Christian countries who were always connected with and sometimes were heads of churches, though they might tolerate dissenting sects (most didn't), would never speak as though dissenters were legitimate paths to God or express such indifference towards dissenting creeds.