The Challenges of Developing Search Functionality

Posted on September 11, 2023
The Challenges of Developing Search Functionality

Introduction

What is the common essential feature among apps we use in our daily lives, such as Coupang, Baemin, and Yanolja? It's search functionality. In Coupang, we search for products; in Baemin, we search for food, and in Yanolja, we search for travel options. We also use search on platforms like Google and Naver to find restaurants or shop. Search has become an indispensable part of our daily lives, and it's a must-have feature in all modern applications.

However, developing search functionality is by no means an easy task, as backend developers and product managers are well aware. Creating a proper search functionality involves significant challenges. Users expect Google-level search quality even within applications, and as that expectation rises, the technical requirements become more complex.

So, what are the technical requirements that make developing search functionality difficult?

There are two primary methods for developing search functionality: using the search capabilities (specifically, query capabilities) provided by a database and adding a separate search engine to synchronize with the database.

Using Database Query Capabilities

The advantage of the first approach is that it allows for relatively simple development in the early stages of a service. It leverages the query and indexing capabilities provided by the database, eliminating the need for additional solutions. Developers can work with the database's query language, resulting in a smooth learning curve.

For example, in a hotel reservation app, you can use a simple SQL query to find hotels in the Pangyo area and provide them to the user. In the early stages of a service when user requirements are not complex, database search capabilities are sufficient.

However, as search requirements become more demanding, significant limitations and performance issues arise. There are functional limitations such as handling typos, synonyms, and scoring calculations. For instance, if a stock market app doesn't handle synonyms, users must enter "NAVER" instead of "네이버" to find relevant information. Any slight typo might result in "no results found."

Moreover, as search criteria become more complex, the complexity of queries increases. What if a user wants to search for a 5-star hotel with an antique atmosphere that allows pets and provides breakfast in Pangyo? As users require more specific search results, query length grows, and you may need to create indexes for numerous conditions. Building these indexes consumes resources, and not using indexing leads to performance issues as the database scans all records to find matching results.

In conclusion, development time increases significantly due to testing, debugging, and other factors. Even when the functionality is complete, it can negatively impact service speed and more.

As your service expands, you'll ultimately require a separate search engine.

Connecting a Search Engine to the Database

Databases and search engines are fundamentally different software designed for distinct purposes.

Databases primarily focus on data integrity, consistency, and durability in data processing. They emphasize balanced performance for read and write operations, concurrency, availability, and recoverability in disaster scenarios. Querying data is relatively less critical, except for specialized databases designed for handling massive data extraction (OLAP databases).

In contrast, search engines are designed to find the most relevant results for user intentions within stored data. They assume features like typo correction, synonym handling, searching for related but not identical results, and ranking search results. Therefore, the database's capabilities are inherently limited.

Due to this trade-off relationship between the two software types, it's advisable to store and process data in databases while handling search functionality separately in search engines. However, this choice leads to a complex development challenge, and hiring skilled backend developers capable of developing and maintaining search functionality becomes a crucial task.

First and foremost, adding a search engine means dealing with two different systems. You'll need to learn and use the search engine's language, resulting in a learning curve. Every time you deploy your service, you must test both systems simultaneously. API changes will occur as well. The backend's dependencies will deepen, and product development speed will gradually slow down. There will be a dramatic drop in development productivity.

Above all, the most challenging issue is synchronizing the database and search engine. To search for data stored in the database through the search engine, both systems must have identical data. Storing the same data in duplicate already raises inefficiencies.

Moreover, you'll need to establish mechanisms to synchronize new events like user registrations and adding products. These mechanisms might involve creating data pipelines based on messaging systems like Apache Kafka or using separate package connectors. Building these solutions takes time and adds costs.

Once the synchronization process is deployed, you'll need to monitor and manage it. You have to track whether synchronization occurs at the same speed as transactions in the database, ensure there are no security issues, and troubleshoot in real-time. If an issue arises, you'll have to determine whether it's a database or search engine problem. Naturally, you'll need to purchase solutions like Datadog. If, by any chance, the search engine's indexing speed falls significantly behind the database's, you'll have to re-synchronize from scratch.

This complexity affects application performance, deteriorates user experiences, decreases retention, lowers conversion rates, and ultimately leads to customer churn.

Is there an easier way to develop search functionality?

Using only database query capabilities is insufficient for implementing proper search functionality. Integrating a search engine with the database requires specialized search development skills and increases development complexity. So, is it truly impossible to enhance the user experience with high-quality search during the early stages of a service?

Read more

검색, 그게 왜 중요하죠?

웹과 모바일 애플리케이션에서 검색은 긍정적인 사용자 경험을 형성하는 핵심기능 입니다. 특히 커머스 서비스에서 검색은 사용자 경험 향상을 넘어, 회사 매출에 직접적인 영향을 미치는 요소 입니다. 상품정보와 콘텐츠의 양이 폭발적으로 증가하는 상황에서 고객이 입력한 검색어에에 부합되는 정보를 적시에 제공하는 검색품질은 애플리케이션과 웹사이트의 성공과 실패를 가를 중요한 요인이 되었습니다. 일반적으로 커머스 서비스에서 상품을 검색하는 고객은 지불의사가 높은 강력한 잠재고객으로 판단합니다. 사용자들이 검색하는 행위와 검색결과에 반응하는 모든 행동에는 고객 자신들의 구매의도, 니즈, 지불의사가 모두 나타나 있다고 보고 있습니다. 통계상으로도 검색을 수행한 사용자의 구매전환율이 그렇지 않은 고객에 비해 2배 이상이며, Total MAU에서 검색을 사용하는 유저의 비율은 20% 이하지만 검색을 한 번 이상 수행한 사용자들에게서 발생하는 매출이 절반이상인 것으로 알려져 있습니다. 또한 첫 검색에 실패한 사용자의 이탈율이 높게 나타나며, 재검색 고객의 구매전환율은 매우 낮게 나타납니다. 즉, 검색은 고객의 지갑을 열기 위한 강력한 수단일 뿐만 아니라 서비스의 지속가능성에도 큰 영향을 미치는 요소 입니다. 그렇다면 검색은 구체적으로 어떻게 고객리텐션과 매출증대, 서비스 개선에 기여할까요?

By Tim Yang|2023-09-10

Tags:

Copyright © 2024 Aeca, Inc.

Made with ☕️ and 😽 in San Francisco, CA.