You can no longer finish a coffee run or a simple subway commute without encountering some kind of electronic system loaded with artificial intelligence (AI) attempting to “guide” you or tell you what to do.

As companies focus more intently on trying to interpret and influence customer behavior, our encounters and experiences with these systems continue to grow. They tell us which local coffee shop to get coffee at, what gym membership to get to lose some extra pounds, what movie to catch on a Friday night and what time to go to bed.

These regular tips seem like “well-meaning” recommendations and reminders meant to push us toward a better life, but it’s not as straightforward as it’s made out to be. Behind the friendly counsel are for-profit organizations trying to continually adjust and adapt customer behaviors, to steer them towards making preferred choices that ensure business remains profitable.

Mitch Ashley, VP and practice lead at the Futurum Group calls this constant digital shadowing a kind of fraud that culminates in the creation of our “digital twins”.

“Fraud isn’t just whether my credit card is valid, or do I have a valid certificate in my wallet. It’s also finding out do I shop at that store. Am I in the town where the purchase is being made?,” said Ashley during an Ignite talk at Security Field Day in October.

The information about our habits and traits captured during every moment of our interfacing with these applications is continually streamed to a processing center where it is analyzed – typically with AI or ML – to predict our daily decisions – but more importantly to map that digital self of ours.

“Imagine a future that’s already being created right now where our biometric behavior analysis is based on a “digital twin” of ourselves, our behaviors captured in some digital form,” he said.

Businesses put to use AI analytics of not just static information like fingerprints and facial recognition, but “information in motion” that can help curate better experiences and be used as mechanisms to prompt us towards impulse purchases. Think of all the activewear, running shoes or ear pods you didn’t plan to buy until a feed popped up on your screen, and you ended up ordering them.

“All of that is part of creating our digital footprint online and guess what – it’s also being captured as part of our digital transactions and starting to be used in behavioral analysis of what our world looks like,” Ashley cautions.

An even more frightening aspect is how these bytes of information are handled post-collection. All the data that makes up our personal and psychological profile are gold for identity theft, and that risk begins as soon as it lands in the information banks of the companies collecting them.

In 2024, Meta Platforms Ireland (Meta IE) was fined an unbelievable €251m, the largest General Data Protection Regulation (GDPR) penalty of 2024, for a regulatory violation that caused a massive data breach affecting 29 million Facebook users in 2018.

The Government Accountability Office states that, usually, customers are unaware of the potential privacy risks and biases that arise from increased personal information collection.

A 2024 Vercara consumer survey of 1000 adults finds that a third (30%) of consumers report being, in some form or another, affected by data exposure following online shopping.

Theft of personal information can have deep-reaching impacts depending on the scale of the attack. As seen over and over in data breaches, PII information such as names, addresses, phone numbers and financial details of customers are leaked on the web. Think of the LoanDepot breach which impacted 16.9 million people last year, or the National Public Data breach which compromised 2.9 billion records, or the Change Healthcare incident which affected 100 million customer records.

While businesses in the aftermath are seen grappling with the fallout, adequate measures are not adopted across the board to prevent such incidents in the first place. Meanwhile, the collection of personal data has become instinctual and “business as usual” for companies.

Consumer data when mined with AI provides strong indicators for whether a company is going to have a robust sales season ahead of time, how customers relate and engage with a brand, and so on, all of which are critical intel for recognizing loyal customers, personalizing interactions and managing customer relationship.

“I think this is an area where we think about identity and fraud and all the things that are so important to us in the security world for us to go back to school and start to think what should we challenge,” Ashley said.

For more such insightful talks, be sure to catch the other Ignite Talks from the Security Field Day event at Techfieldday.com.