Error Handling in Kafka Consumer for API Calls
Consumer API allows you to fetch data from Kafka. The group of payload are processed in consumer to be send to SINK system. Sink system is your target such as database, API, queue or file.
In this blog, we will deep dive how to design the data flow in case you call an API from Kafka Consumer.
SYNC API CALL in Kafka Consumer
When you make a SYNC API call, your code block will wait till the response come from API. After than, depends on the return, the code block will proceed to the logic
As you see the consumer calls an API and wait for the response. Depends on the backend process of the API, it takes sometime(mili-second to seconds) and return response to the Consumer. Once the consumer gets the response, it can start to process another payload
ASYNC API CALL in Kafka Consumer
ASYNC is different from the previous one.When you call ASYNC API, the API return a response that your request is accepted and to be processed in the backend.
As you see, the API returns the response but it doesn’t give an idea whether the data is processed successfully, just inform you that your request is received. The backend process takes some time, you need to make second call to identify the status of processing.
How to tackle errors for SYNC call
Steps in the workflow
- Event is consumed from Kafka Topic
- Event is processed based on the logic
- API is called and expect the following responses;
Success response
API returns success response and consumer can fetch another message from Kafka to be processed
Invalid data
Sent data is not valid and couldn’t be accepted by API. In this case, consumer can send this data to error queue to be investigated later and error should be logged.After than, consumer can fetch another message from Kafka
API down
API is not responding to the consumer due to some reason. The consumer should retry to call API with some time interval.Another important point is to note that the consumer shouldn’t fetch any more messages from Kafka since the target system (API) is down. Once the API is available, the consumer can resume fetching and processing.
How to tackle errors for ASYNC call
Steps in the workflow
- Event is consumed from Kafka Topic
- Event is processed based on the logic
- API is called and expect the following responses;
Success response
API returns success response and consumer can fetch another message from Kafka to be processed. The point is that consumer doesn’t know whether the data is successfully processed, the information only shows the request is received by the API.
API down
Same as SYNC. API is not responding to the consumer due to some reason. The consumer should retry to call API with some time interval.Another important point is to note that the consumer shouldn’t fetch any more messages from Kafka since the target system (API) is down. Once the API is available, the consumer can resume fetching and processing.
Invalid data
Same as SYNC. Sent data is not valid and couldn’t be accepted by API. In this case, consumer can send this data to error queue to be investigated later and error should be logged.After than, consumer can fetch another message from Kafka
So far, we tackled how to interpret the return message. Return message only shows whether the request is received by target system. In order to check the result of process, you need to run another process which is shown as a component 2.
Why ASYNC API couldn’t return the final response?
In general, ASYNC API is used;
- in case don’t want to overload to the target system with huge amount of request
- in case internal process take some time (batch process)
As you see in the following picture, API receive the request and publish the event to the queue. Depends on the database performance, the event is processed in some time.
Let’s check how we identfy whether the data is processed in the database;
Steps in the component 2
For ASYNC API process, the target system can provide another API to check the data is processed sometime later.
- Call to the check API in order to control whether the data is successfully processed.
As you see in the picture, Control API allow you to query the result of the CRUD task. Hence, you can verify the flow.
Summary
As you see, there are different ways of using the API like SYNC and ASYNC API. In this blog, we reviewed how to use these API in terms of error handling.
