Single authentication methods such as password, smart card, or biometric authentication suffer from their own weaknesses. Thus, combined authentication methods have been proposed recently. Unfortunately, even combined authentication methods are exposed to replay attacks, user impersonation attacks, server masquerading attacks, or stolen smart card attacks. To minimize the range of such attacks, we propose a security model that combines smart card authentication and biometric authentication using a modified public key cryptography. The modified public key cryptography transfers a public key only to the opposite entity not to public. The proposed security model can withstand the above-mentioned attacks. In particular, the insider attack can be resisted even in cases where the secret values stored in any two of three parties of a system are compromised. Such tolerance is enabled by modified public keys which are not revealed to the third party.