James Harris, Earl of Malmesbury, 1746-1820