If you click on the "Remove this swap from my active list" next to the relevant swap, you will then go through the process to move it into Uncompleted swaps. I think (from memory) it asks you for a reason why you're moving it into uncompleted swaps but am not sure whether it also asks you to leave feedback for the other swapper. If it does, you can still leave 5* and explain that the swap was cancelled but communications with the other swapper were excellent - that way it won't affect their rating but will still remove the swap from your active list.
The other way of doing this in these circumstances is to email the other swapper and say that you'd like to leave 5* even though the swap was cancelled and would they mind doing the same for you, given that it was dealt with amicably on both sides. That way you both benefit from the feedback. Many people seem to do this when swaps are cancelled but they are otherwise happy with how things were handled.