By Mohammad Ismail, VP of EMEA, Cequence Security
Comparison shopping is a proven and accepted practice within the retail industry and doing this via bots is nothing new, with price comparison sites often being used to offer the consumer the best price on a specific item. At the same time, retailers have also used price comparison technology to ensure their products are offered at competitive prices, spawning a new generation of pricing intelligence tools. But what neither of these approaches takes into account is the risk of exploitation of these techniques by malicious bots.
Attackers are now routinely using these processes to disrupt or monopolise supply. Search abuse, for instance, sees bots used to find, locate, and even buy goods on behalf of the user with respect to highly desirable items. Sneakerbot, Nikebot and Ticketbot are all examples of bots that perform this function. However, automated search bots are not always innocuous in their behaviour.
Typically, these bots will have a detrimental effect on the retailer as search queries can target every single web application URI (uniform resource identifier) across all locations and many of the queried items don’t exist, creating further strain on the retailer’s infrastructure. These bots are easily detectable because their search patterns are too fast and too perfect to be human and the queries may come from a location that doesn’t match the geography they are searching.
Search, scrape and scalp
Another example of malicious behaviour can be found in content scraping. Bots need to copy web content and web developers will use Robot.txt files to indicate where these bots should and should not scrape. However, malicious bots can and do ignore these directives. What’s more, automated scraping can be performed deliberately as a form of attack. We have observed numerous instances where content scraping targeted non-existent URIs or items in an attempt to bog down a website. We’ve also seen various masking or evasive techniques used to disguise and attack, such as by spoofing or forging a browser, making it harder to track the attack.
Scalping bots can also cause disruption. Used to secure purchases associated with flash sales, in-demand ticket sales, and limited-edition merchandise, these bots swarm websites, completing purchases much faster than a human could, leading to inventory issues, infrastructure strain, and disgruntled customers. These bots are notoriously difficult to detect because they hide behind residential proxy IP addresses, so appear to be legitimate. It therefore becomes nigh impossible to block these attacks using IP-based security tools without also blocking bona fide human customers.
Retailers have tried a variety of tactics to stop these attacks. These range from creating a ‘waiting room’ for both bots and customers to buy time for detection and deter bots, to disabling mobile applications during sales to force consumers to the website, to launching sales at off-peak periods and even overnight so that the bot swarms that overwhelm the site don’t impact the core business. Needless to say, these are all approaches that either create friction or cost the business by monopolising resource while doing little to address the core problem: the bots.
Good versus bad bots
It’s an issue made more complex by the fact that the business needs to be able to distinguish between good and bad bots. Good bots perform legitimate search and browsing activity, which is a necessary part of ecommerce, while grey bots are largely neither particularly beneficial nor harmful, while bad bots have a negative effect on the business. Distinguishing one from the other is becoming increasingly important in a world where bot traffic has now overtaken human traffic and where nearly a third of all traffic is now attributed to bad bots.
In the retail environment, however, the default approach has always been to facilitate access to reduce friction, which is why more often than not processes are set up to ‘allow’ rather than ‘deny’. Retailers, therefore, now need to learn how to validate bots before they grant these permissions, and that means looking at the wider behavioural context and intent of the bot. This is harder than it sounds because malicious scraper bots, for instance, will often use the same toolkits and common libraries as legitimate crawlers, even down to the user agent strings in their requests.
Behaviour-based defence
To be effective against today’s sophisticated and in many cases custom-coded bots, retailers need to adopt solutions that employ machine learning and behavioural analysis to identify and separate the malicious traffic and bots from the good. This form of monitoring looks at the actions of the bot rather than just what it is claiming to be, and assigns the bot a unique behavioural fingerprint which can then be tracked. Should the bot then change tactics to evade detection, they can still be monitored.
In addition to detection, it’s also vital that the retailer has some means of arresting the attack. Logging, rate limiting, deception, and blocking are all viable ways to mitigate the bot attack from progressing and the approach taken will depend upon the veracity of the attack, the tactics, techniques and procedures used, and the tolerance levels of the business. Both detection and mitigation functions should be offered in combination; otherwise, the response can be delayed, enabling bots to get through.
Bots are undoubtedly weaponising comparison shopping and continue to be a major threat to online retail, but the reality is that malicious bot attacks are about to become much more of a problem. Humans are using agentic AI bots to handle tasks for them – comparing, recommending and purchasing on an autonomous basis, without the need for human intervention, and that will mean that retailers will have to let them in. Those without the capability to effectively determine good from bad will then find it very difficult to defend and protect their ecommerce platforms so risk being overrun or missing out on a lucrative new revenue stream. So make no mistake, bots will be the new customer and retailers will need to determine if they’re to be trusted.
