toc/projects/cubefs/cubefs-adopter-interview-netease.md

CubeFS Adopter Interview - NetEase

Interview date: Auguest 1st, 2024 Interviewee: Rio Zhang, Operations Engineer, NetEase

Organization Intro

Can you give us an overview of your organization and what it does?

The big data team of NetEase Games. Business scenarios are:

1. Business functions require the data analysis and log retrieval capabilities of ElasticSearch. A storage-computing separation scheme with flexible and adjustable positions is expected.
2. In the scenario of AI machine training and learning, NFS was used before, and there are challenges in reliability and performance.
3. In the log storage scenario of K8s, the disk space of nodes is limited. With the help of CubeFS, remote storage can be realized.

CubeFS is used for data analysis and log retrieval, including:

1. Game operation logs and game server logs.
2. For scoring and recommendation based on game data, storage-computing separation is required.

Motivation

Compared with other products in this space (proprietary and open), what drew you to the project?

We made the selection in 2020. At that time, there were not many comparable options. Compared with proprietary solutions, we were more inclined to open source because it can be maintained by ourselves and has better data privacy.

In fact, there were not many open source distributed storage solutions at that time. The relatively well-known Ceph is known to be difficult to maintain and requires a lot of manpower. Practice has proved that as a user of CubeFS, not much manpower is required.

Usage Scenario

How long has your organization used the project?

We have been using CubeFS for 4 years. Started with v2.1.0 in July, 2020.

What were the main motivations to adopt the project and which key features do you use today?

The initial motivation was to achieve storage-computing separation for ElasticSearch. Moreover, CubeFS had been widely implemented on a large scale at JD.com, reliability was ensured.

As the project evolves and business needs arise:

• We use the Erasure Code subsystem to back up and load ElasticSearch snapshots through the S3 interface. Compared with the previous replica system, storage is reduced by 50%.

• Machine learning requires high-performance and highly available storage. After testing, CubeFS meets the requirements and is put into use. We mainly use the replica subsystem, FUSE, and S3 interfaces. For public cloud acceleration, we use the cfs-bcache cache.

• We store the logs generated by K8s pods to avoid the host Kubelet entering disk pressure status. In this case, we mainly use the replica subsystem and the FUSE interface.

What is the current level of usage (pre-production, production) and scale?

Pre-production environment: about 20 servers, storing 100TB of data, mainly test environment and intranet office network equipment, SLA requirements are not so high.

Production Environment: More than 200 servers, including replica systems and Erasure Code systems, storing 15PB of user data

What version is used and what is your update cadence with the project?

One feature release upgrade per year, CubeFS is usually stable, no fatal bugs. For bugfix releases, we upgrade as needed. The upgrade process is done by automated scripts, which does not require attendance, and administrators only step in to fix problems when they occur.

Can you walk me through what your experience was in either adopting it outright or integrating it with your existing services and applications? What challenges did you experience with the project?

Most of the time, we integrate with existing services via the FUSE interface, with a small number of services natively supporting S3. The main challenges encountered in these projects were:

• CubeFS does not work according to posix semantics in some corner cases, such as the behavior when reading and writing is stuck or reporting errors, but they were fixed after the community maintainer debugged and located them together. In this process, the community was responsive and positively dealing with problems.
• There were many problems before CubeFS v2.4.0, and in the later version we mainly met operational and maintenance issues, such as ease of use, etc. We can deal with them by ourselves and actively communicate with the community.

Did you find the information in the repo or the project docs valuable to your implementation? If so, where did you find the information and what specifically was useful?

Yes, we did find the information valuable. We often look for source code logic based on printed logs or running parameters. The project manual is mainly used to find usage parameters and common FAQs. The documentation is relatively complete, and in most scenarios, we can find the needed information in the documents. The official account of the community has source code analysis articles, which are of great help in familiarizing ourselves with the implementation of the project.

Has your implementation of the project provided measurable value? Such as reducing manual activities, faster integrations, supported federation/multi-cloud, ease of use, cost savings, etc

Compared to the previous maintenance of ElasticSearch on traditional physical disks, we used LVM to rent out to multiple users. The disks were not fully consumed but the CPUs were, resulting in a large amount of waste of resource fragmentation. Moreover, the IO cannot be adjusted, and hardware failures have a large impact. We expanded and promoted the reduced storage capacity to other businesses such as AI training and K8s logging.

Improvements in problem challenges:

• Reduction in resource waste. Previously, the usage rate was that only 60% of the CPU was actually consumed and only 39% of the disk was consumed. After using CubeFS, the usage rate is 80%, with almost no resource fragmentation. Disks do not need to be set up with RAID 10 to reduce the failure rate. Using CubeFS has increased storage efficiency by 100%.
• Compared to three replicas, Erasure Code reduces the consumption of storage space by nearly 100% (saving 50% of space).
• Reduction in operation and maintenance manpower. Before using CubeFS, every time a server fails, it requires manual handling, and all involved business layers need to be modified accordingly. After using CubeFS, handling server failures does not require changes in the business layer.

Solution to problems: CubeFS provides a high-performance storage solution for the AI training scenario.

Do you have any future plans regarding the project? More involvement, feature requests, expansion, etc

We hope CubeFS can be further improved in hybrid cloud acceleration and provide kernel client support.

• For the hybrid cloud scenarios:
  • Workloads are deployed in IDC + public cloud or only running in the public cloud, resources need to be released after completion. With the data source in IDC, a caching scheme is used to accelerate data access on the public cloud.
  • Currently, the cache and client have a 1:1 deployment relationship. The cache cannot be reused, resulting in high resource consumption. We hope the community can consider providing a shared cache solution.
• Regarding the kernel space client:
  • Currently, the FUSE client is in user space and has limited performance. We hope the community can consider providing a kernel space client to offer higher performance.

Perception

What is your perception in terms of the project’s

• Community openness
• Governance
• Community growth potential
• Maintainer diversity and ladder
• Maintainer response

The community has done relatively well in the areas above. The community has shown that it values users and will provide targeted user support in the community, such as answering questions. The community is also active, with regular updates on the WeChat official account.

How are you participating in the project community?

We mainly submit code and issues on GitHub and communicate with developers through issues and WeChat groups.

Did you need to engage with the community members or maintainers? If so, what was the context of the engagement, which communication channels did you use and did it reach an acceptable outcome?

We mainly interact with community members through WeChat and GitHub issues. The purpose is to solve usage problems and discuss some optimization methods.

It can meet the need for communicating with community members.

Project Strengths

In your opinion, what are the overall strengths of the project?

Relatively low maintenance cost and few open source competitors.

Project Improvements

Is there something you feel that holds the project back from reaching its ultimate potential?

The low performance ceiling of the user-space interface of FUSE holds the project back from reaching its ultimate potential. When using the FUSE interface for writing to ElasticSearch, it cannot achieve a very high QPS. Our current workaround is to write to a physical SSD first and then convert to CubeFS storage through ElasticSearch's hot and cold separation mechanism. In fact, using a physical SSD is not storage-compute separation.

If FUSE could provide merged writes, it would meet the performance requirements. Given the current actual situation where FUSE may not be easy to modify, considering the kernel-space approach could be an option.

In your opinion, what can the project do better?

The project could do better by finding several medium and large companies to jointly research and use it and contribute to the community. In my view, smaller companies are more inclined towards commercial cloud solutions. Apart from commercial cloud solutions, storage developed by companies with R&D capabilities will be closed source or even be based on secondary development of an open source component.