2 posts tagged with "rabbitmq"

Memphis is an open-source message queuing and streaming platform that has gained a lot of traction in recent years due to its powerful features and ease of use. It was created to solve the problems of scaling, reliability, and performance in large-scale distributed systems. In this article, i will take a deep dive into Memphis and compare it with RabbitMQ and Kafka, which are similar technology stacks. I will also show you an example of how to use Memphis in detail and discuss the limitations that still exist.

Overview of Memphis​

Memphis is a highly performant messaging system that is designed to be easy to use and highly scalable. It is built on top of the Rust programming language and leverages the power of modern hardware to deliver exceptional performance. Memphis is designed to be highly fault-tolerant and can handle the loss of nodes in a cluster without any loss of data or service interruption. It uses a peer-to-peer architecture that allows for seamless horizontal scaling.

Characteristics of Memphis​

One of the biggest strengths of Memphis is its ease of use. It comes with a simple and intuitive API that makes it easy to get started with. It also has an excellent documentation that makes it easy to learn and troubleshoot. Memphis is highly reliable and can deliver messages with low latency, making it ideal for use cases that require real-time data processing.

When compared to RabbitMQ and Kafka, Memphis is faster and can deliver messages with lower latency. It is also easier to use and has a lower learning curve than both RabbitMQ and Kafka. Memphis also supports a wide range of protocols, including HTTP, MQTT, and AMQP, which makes it highly versatile.

Comparison with RabbitMQ​

RabbitMQ is a popular message broker that is widely used in enterprise environments. It is built on top of the Erlang programming language and is designed to be highly reliable and scalable. RabbitMQ is known for its advanced features, such as message routing, clustering, and federation.

When compared to Memphis, RabbitMQ is more complex and has a steeper learning curve. It is also slower than Memphis and has higher latency when delivering messages. RabbitMQ has more advanced features than Memphis, but these features come at the cost of increased complexity.

Comparison with Kafka​

Kafka is a distributed streaming platform that is widely used for building real-time data pipelines and streaming applications. It is built on top of the Java programming language and is designed to be highly scalable and fault-tolerant. Kafka is known for its high throughput and low latency, making it ideal for use cases that require real-time data processing.

When compared to Memphis, Kafka is more complex and has a steeper learning curve. It is also slower than Memphis and has higher latency when delivering messages. Kafka has more advanced features than Memphis, such as the ability to store and process streams of data over extended periods of time, but these features come at the cost of increased complexity.

Example of using Memphis​

How to use Memphis, let's look at an example with code. Suppose i have two applications that need to communicate asynchronously. The first application produces messages, while the second application consumes messages. To use Memphis for this, i first need to set up a Memphis broker and create a queue for the messages. Here's how to do it.

const { memphis } = require('memphis');

const broker = new memphis.Broker();

const queue = broker.createQueue('example_queue');

// Producer application code
const producer = broker.createProducer('example_queue');
producer.send({ message: 'Hello, World!' });

// Consumer application code
const consumer = broker.createConsumer('example_queue');
consumer.on('message', (msg) => {
  console.log(`Received message: ${msg.content.toString()}`);
});

This code creates a Memphis broker, creates a queue called 'example_queue', and sets up a producer and consumer for the queue. The producer sends a message to the queue, while the consumer listens for messages and prints them to the console. This is just a simple example, but it demonstrates how easy it is to use Memphis for asynchronous communication between applications.

Another example of how to use Memphis to send message from a producer to a consumer.

use memphis::client::MemphisClient;

let client = MemphisClient::new("localhost:5555").unwrap();

let message = "Hello, world!";
let topic = "my-topic";

client.send(topic, message).unwrap();

This example creates a Memphis client and sends a message to the "my-topic" topic.

Limitations of Memphis​

While Memphis is an excellent messaging system, there are still some limitations that exist.

One of the main limitations is its lack of support for some messaging patterns such as publish-subscribe and topic-based messaging. This can make it difficult to implement certain types of applications that require these patterns.

Another limitation is its lack of support for dynamic routing, which can make it challenging to route messages to specific destinations dynamically. Memphis also has a smaller community.

Conclusion​

Memphis is a reliable and high-performance messaging system that is easy to use and has several strengths when compared to other similar messaging systems such as RabbitMQ and Kafka. Although it does have some limitations, they are constantly working to improve Memphis and make it more versatile and robust.

It is still a beta version and has many limitations, but I think it has many strengths and features a new technology stack that is easy to use. If you try it once, I recommend you to try it because you will be able to apply it quickly when the official version is relesed someday.

배경​

현업에서 업무를 하다 보면 배정 관련해서 데이터베이스 쿼리를 통해 가장 적게 배정 받은 순서로 플래너를 배정하고 안심번호를 배정하는 경우가 있습니다. 이때 요청이 많을 경우 문제가 발생합니다.

예를 들어 배정이 가장 적게 된 사람을 검색하고 그 사람은 이제 배정 됐음을 데이터베이스를 통해 업데이트를 하게 된다면, 이 프로세스를 진행하는 동안 수 많은 고객들이 데이터베이스가 업데이트 되기도 전에 같은 프로세스를 돌면서 중복 업데이트, 중복 배정의 문제가 발생됩니다.

설계​

솔루션을 활용한 방법(rabbitmq, kafka, redis, …)​

순서​

1. 백엔드로 요청이 들어오면 배정을 위해 필요한 데이터를 준비하고 그 순서를 큐에 쌓아둡니다.
2. 쌓아둔 큐를 규칙에 따라 cron이 돌면서 가져와 순차적으로 배정됩니다.
3. 이 때 큐의 규칙은 FIFO로 설정해 먼저 쌓인 요청부터 처리하는 방식이 신뢰도 있게 동작할 수 있습니다.

결론​

메시지 큐 관련 솔루션을 활용해 배정이 필요한 부분을 쌓아둔 뒤, 일정 규칙에 맞게 cron을 돌려 쌓아둔 배정을 순차적으로 처리하는 방식입니다.

MySQL user-level lock 기능을 활용하는 방법​

소개​

MySQL은 사용자 레벨에서 데이터베이스를 조작할 수 있는 Lock 기능을 아래와 같이 제공합니다.

GET_LOCK(str, timeout): Lock의 이름을 설정하고 해당 락 프로세스 수행이 오래 걸릴 시를 대비해 timeout도 설정합니다.

IS_FREE_LOCK(str): GET_LOCK에서 설정된 이름을 조회해 현재 이 Lock이 사용중이라면 0, Lock이 풀렸다면 1, 만약 오류가 있다면 null 을 반환합니다.

IS_USED_LOCK(str): 이름에서 보는 것과 같이 IS_FREE_LOCK과 반대로 동작합니다.

RELEASE_ALL_LOCKS(): 해당 세션에서 설정한 모든 락을 풀어줍니다.

RELEASE_LOCK(str): Lock의 이름을 찾아 Lock을 풀어줍니다.

순서​

동시성에 대해 문제가 발생하는 위의 그림과 같이 lock을 걸고 insert, update 를 처리한 후에 lock을 release 해주어 이를 해결하는 방법입니다.

1. 배정이 가장 적게 된 사람을 검색하고 배정 됐음을 업데이트 하는 프로세스를 코드 실행 상 가장 근접하게 구성합니다.
2. 1번에서 구성된 프로세스 앞 단에 GET_LOCK 쿼리를 하고 로직 수행 후 RELEASE_LOCK을 수행합니다.

결론​

솔루션을 활용하는 방법 보다 서버 구성의 복잡도를 줄일 수 있고 백엔드 서버 내에서 코드적으로 처리할 수 있는 이점이 있습니다.