Col. James Octavius Lee married Margaret Harvey, daughter of John Harvey of Castle Semple, in 1816, whereupon Lee changed his name by the addition of Harvey.
His parents are not Harvey according to: https://www.ucl.ac.uk/lbs/person/view/10541 from the Centre for the Study of the Legacies of British Slavery.