Delete font library without opening Font Book OR restore Font Book completely

This is driving me nuts. I created a Font Library or Font Collection (I don't even remember exactly anymore) in Font Book.

I was silly enough to add a very large amount of non-previously-installed fonts into such Library/Collection (around 1000 of them). It might also be relevant to add that all those fonts are stored in an external hard drive. So when I added all those fonts, my computer just froze.

Now, the problem, is that I cannot even start Font Book anymore. Whenever I try to launch it, my computer would just freeze. My guess is that it is trying to configure all the fonts but is unable to finish. I already tried leaving it on for several hours with the Font Book open, but it did not solve my problem.

I think that I should just delete the Library/Collection without launching the Font Book and restore its default settings during the process.

How can I achieve this? Has anyone encountered such problem before?