Should i be worried?

Go back to the doc. You have a really high temp and swollen lymph nodes, your body is trying to fight an infection!