Most of the existing authentication schemes for mobile communication are static in nature, and principally dependent on strength of authenticating identifiers for users identity. The acceptance of all the transactions of a user under a single authentication level is vulnerable. We propose a novel transaction based authentication scheme (TBAS) for mobile communication using cognitive agents. The proposed approach provides range of authentication based on mobile transaction sensitivity, and users behaviors. The TBAS uses mobile agents to gather user behaviors, and static agents for detecting transaction sensitivity, user history analysis, and for choosing appropriate authentication actions. The method has been simulated using the agent factory framework for cognitive agents generation, and their communication. The performance analysis, and the simulation of the proposed system shows that, there is a considerable reduction in the security cost compared to regular session based authentication schemes. By combining transaction based authentication with behavior analysis the authentication attacks can be effectively identified.