Search for Joseph L. McWilliams:

1850 Campbell County, Georgia, 10th Dist., 15, Oct 1850, pg. 444 (image 90):
584/584 Robt. McWilliams, 22, m, Far(mer), $350, Ireland
Polly M., 22, f, SC
JOSEPH L. 5/12, m, GA
Note: Polly M. is probably Mary in 1860--either the information giver or the innumerator didn't do too good with their ages!

1860 Campbell County, Georgia, Campbellton PO, 5 June 1860, pg. 7&8/193 & 194 (image 7-8):
48/48 Robert McWilliams, 30, m, $1600/$500, Ireland
Mary, 30, f, (the innumerator left all the places of birth off for rest of family!)
JOSEPH, 10, m,
Tatonia, 4, f,
Margaret, 2, F,
Jane Foster, 11, f
Note: On the same page there is also a John McWilliams, 42, Ireland with wife Nancy 38
Children:Andrew, Nancy, Peter, William, Marion, James

1880 District 731, Cambell Co, Georgia, pg 513C:
MCWILLIAMS, Robert Self <1828> 52, M W IREL GA
Polly M., McWilliams, wife, f, m, w, 52, SC
Wm. M. McWilliams, son, m, s, w, 19, GA
Margaret S., McWilliams, dau., f, s, w, 20 GA

A search of Campbell Co, Georgia GenWeb marriage records gave me this possibility:
Mary M. Foster to Robert McWilliams, 4 Jan 1848 Campbell CO
(Polly was often a nickname for Mary!)

If anyone has further information on this family I would appreciate your comments. I have been unable to locate Robert or Joseph in the 1870 census.