Looks like a mash-merge. The Samuel Handy who married Mary Dennis was not also married to Comfort Wisher. http://espl-genealogy.org/MilesFiles/site/p898.htm#i89706
In fact, Comfort Wisher (Wishart) seems to have married Samuel's brother Thomas. But they didn't have a son James either. http://espl-genealogy.org/MilesFiles/site/p265.htm#i26492
James is also not a misplaced HanBy (only one James, in the 19th century, and he died young).
So where *did* he come from? The closest match, birthdate-wise, is this d00d: Maj. James Martin (of Benjamin) Handy1,2
M, b. 20 November 1750, d. 14 December 1802
http://espl-genealogy.org/MilesFiles/site/p979.htm#i97831
Parents: Major Benjamin Handy, Elizabeth (of Col. James) Martin
(Benjamin was one of Ebenezer's kids, who hasn't been added yet.)