I think the problem here is that a lot of people use his charges as a means of discrediting his art, as if they're interchangeable. Same for his fanbase too, which I find frustrating (although I'm not that big of a fan of him). Almost every time XXX is brought up, I see people reference his charges and the fact that a substantial part of his fanbase are white teenagers who are fans of anime, as if there is anything wrong with being white, a teenager, or a fan of anime. So what if a certain group of people like his music? I'm wlling to bet a lot of artists have their own special demographics - unless a fan came up to you and spit in your face, I find disregarding an artist just because of their fanbase is a little immature.
Also, Andrew isn't saying that you have to ignore the fact that XXX is a shitty person, I think he feels it's unfair that people have his wrongdoings and musical identity in the same place. You have the right to not listen an artist who happens to be a fucked up person, but what they do in real life and the character they portray in their music are not the same, no matter how "alike" they may seem.