I think the most likely answer is that it would round up to do exactly one point of damage. I know that attacks that do under 1 damage or 0 damage by the calculation formula round up to 1 HP damage, so most likely this would also apply to attacks doing negative damage. We never see negative damage in legit gameplay though, so the only way to test this would be a modding to give some attack negative power or some such. I think the best answer to this would be "We don't know", because negative damage is simply impossible in legit gameplay as far I know.