Personally I would the send swapper a polite and friendly email explaining that the book was in good condition when you posted it and ask whether there was any evidence that the package was damaged. If you are feeling generous you could ask if they would like to choose a "free book" from you list as compensation (but that is totally up to you and you are not required to do this).
In any event, if they don't up your ratings, you just need to get a few more good swaps done and the low star rating will sink and not affect your overall score.
Let us know how you get on.
Welcome to the forum!