Jim,
James Dudley's father was not Col. Richard Robert Dudley, but rather Richards brother William Dudley. There parents where EDWARD DUDLEY AND ELIZABETH PRITCHARD.
WILLIAM b.1621 Bristol, Eng-1677 VA m. Elizabeth Cary b. 1621 VA was Justice of Middlesex 1676
JAMES DUDLEY 1659-1702 m. 1) Mary WELCH 2) Elizabeth 3) Ann Fleet dau. of Wm FLEET, with Mary WELCH he had James b/a 1679 (who had James who m. Jean STAUNTON) and Robert (who had Robert). With Elizabeth he had Sarah and Elizabeth, and with Ann Fleet he had Mary DUDLEY b. 2/24/1692 (who married John BERRY on 1710) and William DUDLEY b. 8/12/1698 d. 1/15/1760 (who married Judith JOHNSON and had Anne b. 1728 and Robert DUDLEY b.1730 who married Joyce and had Gayle, Joyce, Robert, Ambrose, Peter, James and William).
Source is http://genforum.genealogy.com/dudley/messages/726.html