What is correct quantity ?
Now in your system or which is provided by vendor.
If vendor send the correct credit memo and you have entered wrong, then you should cancel the old credit memo instead of doing debit memo.
If your system is correct and vendor send wrong, then tell your vendor to send the correct credit memo.