The answer is simple. I would be much more concerned with the moral and ethical character of the people around my son.
I'm 20 and In a serious relationship with with a 41 year old man.. I agree with not exploiting the relationship too soon to the kids, no matter what the ages are. I've known late 20s-early 30s women who would consider a man her age to not be mature or responsible enough. Just because this lady is younger than your husband does not mean she is a bad person. Maybe the girl is very mature, you don't know I hope it takes you less time to figure that out than it took me. I have a 4 year old as well, and she can recite her whole day to me.