Ogilvie, James, Earl of Seafield, 1750-1811