In those first years when settlers came to Horse Neck, the larger area  including what is now Caldwell, they walked to the nearest church, the Mountain Society at Orange, to worship with Rev. Jedediah Chapman. This journey took a half day each way. Later Rev. Chapman  joined by Elizabethtown minister, Rev. James Caldwell, and Rev. Joseph Grover of Morris County traveled to Horse Neck to preach the Gospel and perform baptisms, weddings and funerals at the homes, barns and burying grounds of the little settlement. The Horse Neck residents wanted their own church, but Rev. Chapman and Rev. Caldwell, both fervent patriots, urged the people to first fight for their freedom from Great Britain and postpone their plan for a church.

The dream was not long deferred. In the midst of the Revolution, nine Horse Neck men and Rev. James Caldwell met on February 21, 1779 to organize a committee to acquire land on which they could build a church and a parsonage. On July 17, 1779, a tract of land lying in what is now the center of the Borough of Caldwell, was given to the First Presbyterian Church in Horse-Neck "for the purpose of erecting a proper building and buildings for the support and convenience of the public worship of Almighty God, and for the support and comfort of such minister of the Gospel of the Presbyterian denomination... and also for the use of a place for burial for said Congregation and Inhabitants."[1]

Rev. Caldwell served as a Deputy Quartermaster for General George Washington during the Revolution. His wife, Hannah Ogden Caldwell, was killed by a British soldier as she held an infant in her arms in her home during the Battle of Springfield. Rev. Caldwell was himself murdered in 1781. The members of the church named their parish Caldwell in his honor and the NJ Legislature followed that example when they created Caldwell Township in 1798.

A plaque was placed on a boulder in front of the church by the NJ Society, Sons of the American Revolution on November 24, 1924. The plaque recognizes Rev. Caldwell’s contributions to the community:

 

 

“ There are times when it is righteous to fight as well as to pray.”

"Rev. James Caldwell, in whose honor this community was named,early espoused the country's cause in the War for American Independence. Chaplain - Deputy Quartermaster - General in Washington's army. Sterling patriot- fearless preacher- Martyr on Freedom's Alter."

 

The cemetery, affectionately known as the "Old Burying Ground” of the First Presbyterian Church at Caldwell is known to have been used from at least circa 1788 through 1881. The cemetery contains the graves of twenty veterans of the Revolutionary War:[2]

Veteran of Revolution	Date  of Death	Age at Death
Daniel Beach	Dec. 31, 1824	81
Zadoc Crane	Feb. 11, 1841	83th. year
Samuel Crane, Esq.	Feb. 28, 1811	64th. year
Rev. John Duryea	Octo. 2, 18316	84
Stephen Fordham	Nov. 29, 1829	75
John Gould	Aug. 25, 1807	71st. year
Joseph Gould	Dec. 7, 1810	74th. year
Robert Gould	July 23, 1825	71
Timothy Gould	Feb. 19, 1841	89th. year
Gen. William Gould	Feb. 12, 1847	90
Rev. Stephen Grover	June 22, 1836	78th. year
Joseph Harrison	March 1, 1837	89
Jacob Kent	June 3, 1841	85
Enos Martin, Esq.	Sept. 1, 1810	47
David Ogden	Aug. 13, 1790	36
John Ogden	Aug. 27, 1797	60
Simeon Ogden	Apr. 2, 1790	26 yrs. 11 mos.
Caleb Pierson	March 6, 1801	63rd. year
William Sanford	Nov. 8, 1811	45th. year
Josiah Steele	Oct. 22, 1836	76

Rev. Stephen Grover was the first minister of the First Presbyterian Church at Caldwell. Grover was a student at Dartmouth College in NH when the Revolution began. He left school and served in the 21st Regiment of the Continental Army in Connecticut. After military service he returned and graduated with honors from Dartmouth in 1786. His brother was Rev. Joseph Grover of the Parsippany Presbyterian Church, which likely affected his decision to come to NJ after his graduation. The First Presbyterian Church at Caldwell extended an offer to Rev Stephen Grover to become pastor in August 1787 and he served in this capacity until 1833. Rev. Grover was highly esteemed by the community; more than two dozen Caldwell infants were named after him. One of these was Stephen Grover Cleveland, better known as President Grover Cleveland, born in the Church Manse in 1837.

He was buried on Jun 25 1836 in the Old Burying Ground, Grave 3, Row 14-12.

Also in the Old Burying Ground is a tall stone marking the grave of General William Gould. Gould served as a private in the New Jersey Militia during the Revolution and fought at the Battle of Monmouth. He later served during the Whiskey Rebellion of 1794 and during the War of 1812. Thereafter, he was made a Brevet General. Gould was also a member of the New Jersey legislature from 1805 to 1817.[3]

The Old Burying Ground was replaced by Prospect Hill Cemetery at Caldwell in 1866, although family members continued to be buried there until 1881, when the church sexton, Abraham Doremus Personette was laid to rest next to his wife, Joanna Williams Personette.

In the early 1980’s the Old Burying Ground was almost razed to create a parking lot. Resistance from descendants of those buried there included Rachel A. Farrington, Ruth Crane Shepard and Personette Gould Baldwin, then one of the Church’s oldest and longest members. The debate convinced the Church to spare the Old Burying Ground and today it is now receives cremated remains. Damage done by time, weather, neglect and vandals has eliminated many of the stones, but the Old Burying Ground remains a poignant reminder of the history of Caldwell and its people.