I haven't researched his family line but his great-great (?) grandfather was a writer of the most bigoted tracts against blacks that you can find. His family were big in the founding of Virginia which is one of the most historically bigoted states in the nation.

William Byrd.

http://en.wikipedia.org/wiki/William_Byrd_I

http://en.wikipedia.org/wiki/William_Byrd_II

http://en.wikipedia.org/wiki/William_Byrd_III

http://en.wikipedia.org/wiki/Harry_F._Byrd