I have to agree with Sam. You could look like a highly-rated member, but only have 3 or 4 ratings. Averages usually aren't the most reliable source of information in low numbers, especially when only 5 options can be given as values (1-5 stars).
If anything, the rating of the member would have to be hidden on their member page / answers until they've allocated many ratings (10-20, maybe?), which would probably be more trouble than it's worth to try to work toward, since quite a few people overlook even voting on answers frequently.
Points are instant, easy to rank, and proportion well between other members, the user's number of answers, and the length in which they've been on the site; there's simply many more factors included in points that can make you think, "Is this user a reliable source, or not?".
You can wait until Captain answers, of course, but I have a funny feeling that his answer will be similar to something like this.