With the proliferation of IoT devices, network device identification is essential for effective network management and security. Many exhibit performance degradation despite the potential of machine learning-based IoT device identification solutions. Degradation arises from the assumption of static IoT environments that do not account for the diversity of real-world IoT networks, as devices operate in various modes and evolve over time. In this paper, we evaluate current IoT device identification solutions using curated datasets and representative features across different settings. We consider key factors that affect real-world device identification, including modes of operation, spatio-temporal variations, and traffic sampling, and organise them into a set of attributes by which we can evaluate current solutions. We then use machine learning explainability techniques to pinpoint the key causes of performance degradation. This evaluation uncovers empirical evidence of what continuously identifies devices, provides valuable insights, and practical recommendations for network operators to improve their IoT device identification in operational deployments