Malamar isn't bad, it's a great Pokemon!
The move set is fine, but you should EV train and IV breed any Pokemon you use against other real-life Trainers.
Malamar @ Razor Claw
EVs - 128/252/128/0/0/0
Nature - Adamant
Ability - Contrary
I love the fact that with the Razor Claw, the critical hit ratio of PsychoSlash is raised even more. It works great!
The most important thing to look for when IV breeding is the correct Nature, and Pokemon you use in-game generally don't have the correct nature. To catch a proper natured Inkay you will need-
2) An Abra/Ralts with Synchronize(Adamant Nature).
First go to route 8 and keep the Abra/Ralts fainted in the front of your party. Now all Pokemon that appear have a 50% chance of having Adamant Nature. Catch a few Inkays with the Quick Balls and go to your PC, withdraw any one which has an Adamant Nature and give it an Everstone and deposit it in the Day Care.
Try and get a 6IV Japanese Ditto. It will take a bit of time(try offering shinies or rare Pokemon), but you will eventually get it.
Now give that Ditto a Destiny Knot and Deposit it in the Day Care.
Now, when the man at the Day Care turns around, talk to him to receive an Egg. Get 5 eggs and then go to Kiloude City. Keep cycling until you hatch all of the eggs and then go to the IV Judge in the Kiloude City Pokemon Center. He will tell you what IVs your Pokemon is perfect in. Keep hatching eggs until you get one the has the IVs - 31/31/31/x/31/31(the Special Defense IVs are optional to get, but they will help in battle). Also, make sure that it has Contrary. If it doesn't, then use an Ability Capsule on it. They are available at the Battle Maison for 200BP.
Use Super Training and Core Training to EV train your newly bred Inkay. Give it 252 Attack EVs. You don't have to worry as the limit is 252. But the HP and Defense EVs are both 128, so use a pen and paper to keep track of how many EVs you have given it.
The reason your Malamar is not performing well is because it was used in-game so it has an awkward EV spread and it is not IV bred. Malamar is not a bad Pokemon. It is just up to the Trainer to use it properly.
Hope I helped,