How is my average rating worked out?
The mean of every rating given to every note you have shared. A note with no ratings does not count against you — it simply is not in the calculation.
One poor rating on one note moves the average very little once you have several notes with ratings.