I doubt Sony would be too worried - they'd sell more replacement PSPs

However, using the PSP while it's charging shouldn't cause any problems at all. All it's doing is redirecting some of the charging current towards operation. Unless there's a major power spike which may well fry it anyway (mine usually charges off USB so no mains voltage to worry about), it should be fine.
But yeah, if it doesn't go with the battery but does when it's plugged in, then it'd be a dead battery, which happens - _might_ be covered by warranty depending on company policy. It could also be a faulty charger assuming you're using the correct one, you could try buying one of the inexpensive USB charging cables and see if that works, they're pretty handy anyway.
Try finding someone with a meter and make sure that there is in fact 5Volts (centre is positive, it might output as high as 6.5V) coming out of the adaptor plug.
The power adaptor for the PSP is rated at 2A - that's a heckuva lot of current for such a small gadget, how much does the PSP actually draw at peak!? (333Mhz, UMD going, max brightness...). Not that it matters, better to have a higher rating than risk blowing it up with excessive drain