Most existing computer and network systems authenticate a user only at the initial login session. This could be a critical security weakness, especially for high-security systems because it enables an impostor to access the system resources with the initial access permission. To address this security flaw, Continuous authentication of the user is necessary, to continuously monitors and authenticates the user throughout the session. Existing continuous authentication schemes primarily uses hard biometric traits, which is inconvenient to the user. To mitigate this problem, a new continuous user authentication scheme is designed to authenticate the user irrespective of their posture in front of the system. The system continuously monitors the user by using soft biometrics (color of user's clothing and facial skin) along with hard biometrics. It automatically registers soft biometric traits every time the user logs in and fuses soft biometric matching with the conventional face biometric authentication.