If you use a poor source of entropy for the signatures, then reusing the same bitcoin address will let someone deduce the private key. An Android RNG bug caused the theft of a bunch of bitcoins in this manner. If the victims had always used new change addresses, their coins would have been safe.
This is because a bitcoin is protected by two forms of cryptography: the Elliptic Curve Digital Signature Algorithm (ECDSA) and a hash of the public key.
Public keys are broadcast only when spending to match the signature. As long as you have never spent bitcoins, all people know is the RIPEMD160 hash of the SHA256 hash of the public key.
To crack a bitcoin you need to break both forms of cryptography, breaking either one on its own is very hard (if good RNG is used), breaking both at the same time is exponentially harder.
When you use a bitcoin you have to expose the public key, this means you are now only protected by one form of cryptography (ECDSA).