Me or You?

Is it better to protect yourself or others? Whose spiritual development has more priority? If I know someone is hurting me and I tolerate it, am I allowing myself to be hurt or am I hurting the other person by allowing them to continue their behavior and thus, adding up bad karma points? Are you responsible for the behavior of others or only yourself??