For Recommendation Engine Development, AWS is Recommended

Industries such as retail and media/entertainment are rife with competition. To increase market share, companies in these sectors must do more than provide products and services that satisfy the needs of their customers.

They must create experiences that exceed expectations, facilitate brand loyalty, and position these organizations as the “go-to” sources for what their stakeholders want now – and might want in the future. To do so requires an understanding of human behavior – specifically that of the customers. And it requires the ability to use those insights to shape behaviors and drive choices.

That’s what a recommendation engine offers.

The Logic and Challenges of Recommendation Engines

Almost everyone is familiar with examples of recommendation engines, including those used by Amazon, Netflix, Walmart, YouTube, eBay, and Pandora, among many others. In simple terms, a recommendation engine is a type of data filtering system that uses machine learning (ML) and data on customer behavior and history to recommend the most relevant items to a particular user or customer.

ML employs massive amounts of data, which may include customer purchase history, preferences, and search behavior, to train an algorithm so it can generate appropriate recommendations based on that data. When new information about the customer becomes available, the system incorporates it and offers updated recommendations. ML also uses data from similar customers and customers that have exhibited similar behaviors and choices to further inform its recommendations.

A number of statistics attest to the value of recommendation engines:

• The click-through rate of personalized recommendations is twice that of non-personalized recommendations.
• Personalized product recommendations dramatically increase AOV (average order value).
• Shoppers who engage with AI-powered product recommendations have a 26% higher average order value (AOV).
• When shoppers click on product recommendations, the chance they’ll complete the sale nearly quadruples. That likelihood continues to increase the more they engage with the suggested products.
• Up to 31% of eCommerce site revenue is generated from personalized product recommendations.

The Challenges of Recommendation Engine Development

It all sounds practical and easy. But that’s not necessarily the case when developing and implementing a recommendation engine and integrating it into an app. There are numerous factors that come into play. These include determining what data to use, how to collect it, how to structure it, and how and where to store it.

Some knowledge of ML and/or data science may be required. You must know how to train the data, how to determine if the outputs are accurate and how to integrate the recommendation engine within a real-time application.

There’s also the matter of determining the platform to use for hosting and powering it, as well as the technologies that comprise it.

The Benefits of AWS for Recommendation Engine Development

Fortunately, AWS offers a variety of resources to help ─ starting with the AWS Cloud and its well-known benefits. There are also numerous tools you can use to build recommendation engines and enable ML. Among them:

• Amazon Neptune is a fully managed graph database provided by AWS. It provides fast, graph-based queries on connected data.
• AWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development.
• Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to prepare build, train, and deploy ML models quickly.
• Amazon Forecast is a time-series forecasting service based on ML and built for business metrics analysis.

ClearScale’s Choice: Amazon Personalize

ClearScale has been involved in developing numerous AWS recommendation engines on the cloud for clients across many industries. In addition to the AWS resources cited, among the tools we’ve found most beneficial is Amazon Personalize.

Amazon Personalize is an API-based tool. It automates and accelerates the complex machine learning tasks integral to building, training, refining, and deploying personalization models. Among the reasons we’ve made it integral to our tool kit for building recommendation engines are because it:

• Automates and accelerates the complex ML tasks integral to building, training, refining, and deploying personalization models
• Leverages the same ML modeling that Amazon uses in its retail business to drive personalization
• Trains, tunes, and deploys custom ML models, while provisioning the necessary infrastructure and managing the full ML pipeline
• Provides customers with results on a pay-as-you-use basis
• Keeps customer data private at all times

Amazon Personalize in Action

ClearScale has used Amazon Personalize in several client engagements. Among them was the development of a smart recommendation engine (SRE) prototype for PBS that’s capable of making high-quality suggestions to viewers based on a multitude of factors. The work included:

• Determining which data sources to feed into the ML models. And incorporating contextual information from Google Analytics to gain a more comprehensive understanding of viewer behavior
• Implementing a data pipeline starting with AWS Glue, a serverless cloud-native solution to crawl, validate, and transform data from diverse sources
• Orchestrating processes using AWS Step Functions, which allows the client to benefit from automated stateflow management and exceptions handling.
• Creating  the initial version of the smart recommendation engine based on Amazon Personalize; using Amazon FSx for Lustre to achieve a better throughout and increase performance for ML jobs that use model training data from S3, and integrating Amazon SageMaker Studio as the development environment ML engineers use to maintain models
• Selecting the core recipes (which are Amazon Personalize algorithms fine-tuned for specific use cases) for the SRE and building four models based on different requirements per recommendations input and output

The result: PBS now has an effective MLOps platform and recommendation system that it can build on.

Big Data

For another client, ClearScale created a prototype that includes a search engine component that re-ranks results within a session. Therefore it always presents customers with high-quality feeds. It can also deliver personalized notifications. Services such as Amazon Personalize, Amazon Pinpoint, and Amazon Elasticsearch Service figure prominently in the solution. But there’s more to it than ML-driven functions.

ClearScale designed the recommendation engine to be serverless. With no infrastructure to buy and maintain, the client can quickly scale on demand and only pay for the resources used.

ClearScale also used an Infrastructure as Code (IaC) approach. That allows for automatically managing, monitoring, and provisioning resources rather than manually configuring discrete hardware devices and operating systems. If the app should fail for any reason, the client can quickly and easily redeploy the architecture.

The new recommendation engine is fully automated and managed by AWS, so the client doesn’t have to maintain an in-house data science team to take advantage of its two decades of Amazon intelligence.

Get Personal

To learn more about how ClearScale has used AWS Personalize and other AWS ML resources to assist clients in building recommendation engines and other ML-powered solutions, read our case studies. You can also learn more about our work with retail clients. And if you’d like to consult with us on an ML-based recommendation engine project, contact us now:

Call us at 1-800-591-0442
Send us an email at sales@clearscale.com
Fill out a Contact Form
Read our Customer Case Studies