De Morgan, family