Downvotes are used on bad answers. So it may be that you get downvoted because of something you said, or the wrong answer, but honestly points are easy to come buy. Just answer a bunch of things and at some point you'll find you have 3 upvotes and two downvotes, you made ten points. Downvotes will happen, accept that, move on, it won't change your life.
If you feel you were unjustly spammed downvotes then you should comment on the ban report page, but I don't think that's the case here.