Can I lose a badge?

In principle yes, because every badge reads a live number rather than being awarded once. Deleting the notes that earned you Librarian would take Librarian with them.

In practice nothing is taken away for inactivity or for anything you have not done deliberately.