In this paper, we describe the challenges in using Personal Identity Verification (PIV) cards and PIV-like cards as federated identities to authenticate to US Federal government facilities and systems. The current set of specifications and policies related to the implementation and use of PIV cards leave a number of gaps in terms of trust and assurance. This paper identifies these gaps and proposes approaches to address them towards making the PIV card the standardized, interoperable, federated identity credential envisioned within Homeland Security Presidential Directive 12 (HSPD-12). Categories and Subject Descriptors K.6.5 {Management of Computing and Information Systems]: Security and Protection General Terms Management, Security, Standardization. Keywords Authentication, Smart cards, PKI, Assurance, Federal Bridge Certification Authority, Authorization.