I gave my Aadhaar card to a service provider for proving my identity. Can anyone harm me by knowing and misusing my Aadhaar number?

No. Just, by knowing your Aadhaar number, no one can harm you. To prove your identity, Aadhaar number is verfied/authenticated by agencies through various modes as prescribed under Aadhaar Act, 2016.