You should see your doctor at any potential sign of infertility. Not only can you receive help with fertility, but a small percentage of male infertility is actually due to testicular cancer. This is often missed by primary doctors.
With any abnormal results–including exams that change from one year to the next–you should schedule an appointment with a specialist. Like other cancers, prostate cancer is best treated when it is detected early.