Why is slavery my fault?

Ok, I'm white and there was a guy in my business cl who's black and he said slavery was my fault because my ancestors had slaves. (That's what he said) anyway I straight up told him that my family never owned slaves because they came to America from Ireland in the 1880's or from Germany in the 1880's and that a member of my family has fought in every war from WWI up until today. And after all this the teacher pulled me aside and said I shouldn't try to offend people like that. Do you think this is fair?