In today's fast-growing and dynamic global business landscape, Human Resource (HR) teams (especially from cross-border organizations) face ever-increasing challenges in efficiently maintaining legal and financial compliance. As a result, Gonex (opens new window), the world's leading AI-native HR compliance service, was developed to assist global HR teams in overcoming these challenges and streamline their operations.
Leveraging the power of large language modeling, Gonex can rapidly process multiple HR-specific data types and automate complex international human capital workflows, such as onboarding and offboarding, calculating and paying salaries, and calculating and paying country-specific employee taxes.
Gonex utilizes machine learning techniques to analyze employee job performance (based on key performance metrics) as well as employee satisfaction, providing HR professionals and decision-makers with data-driven and actionable insights.
Note:
Gonex was among the early adopters of AI technology for knowledge base management.
Because of the complexity of our AI-based model and to continuously deliver on our promises, providing our customers with the highest quality service, we needed to upgrade our database to a vector database, one that offers high availability and multi-tenancy. After evaluating different options, we went with MyScale (opens new window), aligning seamlessly with our requirements in terms of availability, customer data isolation, cost-effectiveness, and data maintenance.
# What We Need from a Vector Database
Gonex operates in over one hundred countries and geographic sub-regions within a specific country, making it a versatile choice for global enterprises. Leveraging the capabilities of generative AI, Gonex offers users a seamless user experience that transcends language and professional knowledge barriers through natural language interactions.
Therefore, we needed to upgrade to a vector database to continue providing a top-notch user experience as well as the highest customer service. And to find the vector database that perfectly meets Gonex's diverse and demanding requirements, we identified the following critical aspects that are pivotal to our decision-making process:
# Data Management
Our LLM's knowledge base includes many different categories, such as policies, information interpretation, case analysis, and more. When splitting metadata, we must utilize individual fields to handle policy information for different countries or regions. Additionally, since we update our knowledge base daily, we require ease of use to update our LLM's historical data.
# Accurately Retrieve Massive Data
To enable efficient national-level content querying for our customers, we use countries as a dimension to manage information, sometimes even further subdividing into specific regions. While content maintenance poses minimal challenges, the primary hurdle is quickly and accurately retrieving relevant knowledge base information based on customer queries.
# Data Isolation
Gonex's offering is a consolidated SaaS platform for multiple clients (tenants) that share services. Each client's data includes sensitive information, such as employee lists and details, other HR data, financial data, and user details. Therefore, it is imperative to ensure data privacy between tenants to prevent data leakage and unauthorized access by implementing physical or logical mechanisms to isolate multi-tenant data.
# High Availability and Cost Effectiveness
As the vector database is a relatively new concept, more than the available documentation and community support for open-source solutions is needed to support the cost-effective implementation of a self-hosted solution. Therefore, we had to consider proprietary vector database solutions.
The paramount consideration in choosing a vector database solution is the guarantee of high availability to ensure uninterrupted business operations. Our selected database must consistently deliver highly reliable services and establish a resilient database environment that meets our exacting requirements consistently over time.
At the same time, maintenance costs are a key consideration when choosing a vector database. We needed to consider several factors: hardware, software, backup systems, security and compliance, customer support, and performance enhancement, with the overarching goal of maintaining these costs within a justifiable range.
In summary, we hoped to identify a vector database solution that balances cost and high availability, ensuring optimal support for our business requirements.
Because of the complexity of our requirements and the newness of the vector database industry, we were extremely cautious when selecting a suitable database solution. Around April 2023, based on the requirements described above, we embarked on our journey of discovery. However, we soon discovered that viable options were limited. We conducted research and testing on databases such as Redis 6.x, Pinecone, and Milvus, but encountered various issues with each of them. They all fell short of our requirements, whether viewed from the perspective of maintenance costs or convenience.
# Why We Chose MyScale
In our moment of uncertainty, we discovered the MyScale vector database solution, offering us a ray of hope and leaving an indelible mark on our collective business minds with its remarkable filter search and RBAC (Role-Based Access Control) (opens new window) features. MyScale stood out as a cloud-based vector database solution meticulously optimized for LLM-based AI applications.
What sets MyScale (opens new window) apart from other proprietary vector databases like Pinecone, Milvus, Qdrant, and Weaviate is it's founded on the open-source SQL-compatible ClickHouse database, empowering the MyScale to deliver rapid and scalable performance not only for vector search but also for filtered search. It also seamlessly combines complex SQL statements and vector search queries, including advanced operations like joining vector search results to another database table.
MyScale facilitates efficient data retrieval and scalability while providing essential features like filter search and RBAC. MyScale is exceptionally well-suited to our business model, with its integrated capabilities enabling us to store and maintain granular data for multiple countries, ensuring seamless data categorization and metadata maintenance.
# Key Benefits MyScale Brings to Gonex
Having explored the reasons behind our choice, let's dive into how Gonex harnesses MyScale to achieve operational excellence and meet our unique business needs.
# 1. Data Maintenance
Leveraging MyScale's multi-table structure and robust intent recognition engine, we've gained the capability to format and filter information according to crucial attributes like intent, country/region, and issue type. Using country as a primary dimension, we created multiple data tables, each supporting vector data storage with multiple labels. This approach facilitates retrieval based on vector similarity and enhances knowledge-base accuracy through label-based categorization, achieving an impressive accuracy rate of 99.9%.
# 2. Data Isolation
Many of Gonex’s clients are SaaS enterprises; therefore, MyScale’s multi-tenancy feature (opens new window) ensures that each tenant’s data is stored and managed independently within a multi-tenant architecture, even when sharing the same physical infrastructure and resources.
MyScale also offers a variety of multi-tenant strategies, allowing us the flexibility to choose a strategy based on specific requirements. Consequently, we can offer our clients data isolation options ranging from table-level to instance-level isolation, ensuring data security and privacy for all of our clients.
# 3. Technical Support
MyScale provides comprehensive support across multiple channels compared to open-source vector database solutions. We can engage with MyScale’s technical experts by email, telephone, and other communication apps like Discord (opens new window). Kudos to MyScale because our customer support engagements are typically responded to and resolved within an hour, promptly addressing our technical queries and issues.
# Our Future with MyScale
We plan to continue our partnership with MyScale, supplying our users with compliant data storage services. Moreover, given our diverse user base spanning multiple countries/regions, we plan to deploy MyScale globally in our major data centers, such as San Francisco, Singapore, Frankfurt, Sydney, and Sao Paulo.
Furthermore, we are developing an AI-based global talent recruitment application known as ATS or Applicant Tracking System, offering comprehensive recruitment services, covering everything from resume collection and analysis to interview guidance and onboarding support. Lastly, we intend to migrate our entire vector data storage services to MyScale to support our evolving business requirements and continue delivering an exceptional user experience to all our customers.