Kafka direct和receiver

Kafka direct和receiver

Receiver Approach

Receiver是使用kafka的high level的consumer API来实现的。Receiver从kafka中获取数据都是存储在Spark Executor内存中的，然后Spark Streaming启动的job会去处理那些数据

然而这种方式很可能会丢失数据，如果要启用高可靠机制，让数据零丢失，就必须启动Spark Streaming预写日志机制（WAL，Write Ahead Log）。该机制会同步地接受到Kafka数据写入分布式文件系统（比如HDFS）上的预写日志中。所有底层节点出现了失败，也可以使用预写日志的数进行恢复

注意事项kafka中topic的partition与spark中RDD的partition是没有关系的，因此，在KafkaUtils.createStream()中，提高partition的数量，只会增加Receiver的数量，也就是读取Kafka中topic partition的线程数量，不会增加Spark处理数据的并行度。可以创建多个Kafka输入DStream，使用不同的consumer group和topic，来通过多个receiver并行接受数据。如果基于容错的文件系统，比如HDFS，启用了预写日志机制，接收到的数据都会被复制一份到预写日志中。因此，KafkaUtils.createStream()中，设置的持久化级别是StorageLevel.MEMORY_AND_DISK_SER。

Direct Approach

它会周期性的查询kafka，来获取每个topic + partition的最新offset，从而定义每一个batch的offset的范围。当处理数据的job启动时，就会使用kafka简单的消费者API来获取kafka指定offset的范围的数据。

1. 简化并行读取 ：如果要读取多个partition，不需要创建多个输入DStream然后对它们进行union操作。Spark会创建跟kafka partition一样多的RDD partition，并且会并行从kafka中读取数据。所以在kafka partition和RDD partition之间有一个一对一的映射关系。
2. 高性能不依赖于Receiver，不需要开启WAL机制 ：如果要保证数据零丢失，基于Receiver的机制需要开启WAL机制，这种方式其实很低效，因为数据实际上被copy了2份，kafka自己本身就有可靠的机制，会对数据复制一份，而这里又复制一份到WAL中。基于Direct的方式， ，只要kafka中做了数据的复制，那么就可以通过kafka的副本进行恢复。
3. 有且仅有一次的事务机制 ：基于Receiver的方式，是使用Kafka High Level的API在zookeeper中保存消费过的offset的。这是消费kafka数据的传统方式，这种方式配合着WAL机制可以保证数据零丢失，但是无法保证数据制备处理一次且仅有一次，可能会两次或者更多，因为spark和zookeeper可能是不同步的。
4. 降低资源 ：Direct不需要Receivers，其中Executors全部参与到计算任务中；而Receiver-based则需要专门的Receivers来读取kafka数据且不参与计算。因此相同的资源申请，Direct能够支持更大的业务。
5. 降低内存 ：Receiver-based的Receiver与其他Executor是异步的，并持续不断接收数据，对于小业务量的场景还好，如果遇到大业务量时，需要提高Receiver的内存，但是参与计算的Executor并无需那么多的内存。而Direct因为没有Receiver，而是在计算时读取数据，然后直接借宿那，所以对内存的要求很低。实际应用中我们可以把原先的10G降至现在的2-4G左右。
6. 不会出现数据堆积 ：Receiver-based方法需要Receivers来异步持续不断的读取数据，因此遇到网络、存储负载等因素，导致实时任务出现堆积，但Receivers却还在持续读取数据，此种情况很容易导致计算崩溃。Direct则没有这种顾虑，其Direct在触发batch计算任务时，才会读取数据并计算。队列出现堆积并不会引起程序的失败。

基于direct的方式，使用kafka的简单api，Spark Streaming自己就负责追踪消费的offset，并保存在checkpoint中。Spark自己一定是同步的，因此可以保证数据是消费一次且仅消费一次。

%23%23%20Receiver%20Approach%0AReceiver%E6%98%AF%E4%BD%BF%E7%94%A8kafka%E7%9A%84high%20level%E7%9A%84consumer%20API%E6%9D%A5%E5%AE%9E%E7%8E%B0%E7%9A%84%E3%80%82Receiver%E4%BB%8Ekafka%E4%B8%AD%E8%8E%B7%E5%8F%96%E6%95%B0%E6%8D%AE%E9%83%BD%E6%98%AF%E5%AD%98%E5%82%A8%E5%9C%A8Spark%20Executor%E5%86%85%E5%AD%98%E4%B8%AD%E7%9A%84%EF%BC%8C%E7%84%B6%E5%90%8ESpark%20Streaming%E5%90%AF%E5%8A%A8%E7%9A%84job%E4%BC%9A%E5%8E%BB%E5%A4%84%E7%90%86%E9%82%A3%E4%BA%9B%E6%95%B0%E6%8D%AE%0A%0A%E7%84%B6%E8%80%8C%E8%BF%99%E7%A7%8D%E6%96%B9%E5%BC%8F%E5%BE%88%E5%8F%AF%E8%83%BD%E4%BC%9A%E4%B8%A2%E5%A4%B1%E6%95%B0%E6%8D%AE%EF%BC%8C%E5%A6%82%E6%9E%9C%E8%A6%81%E5%90%AF%E7%94%A8%E9%AB%98%E5%8F%AF%E9%9D%A0%E6%9C%BA%E5%88%B6%EF%BC%8C%E8%AE%A9%E6%95%B0%E6%8D%AE%E9%9B%B6%E4%B8%A2%E5%A4%B1%EF%BC%8C%E5%B0%B1%E5%BF%85%E9%A1%BB%E5%90%AF%E5%8A%A8Spark%20Streaming%E9%A2%84%E5%86%99%E6%97%A5%E5%BF%97%E6%9C%BA%E5%88%B6%EF%BC%88WAL%EF%BC%8CWrite%20Ahead%20Log%EF%BC%89%E3%80%82%E8%AF%A5%E6%9C%BA%E5%88%B6%E4%BC%9A%E5%90%8C%E6%AD%A5%E5%9C%B0%E6%8E%A5%E5%8F%97%E5%88%B0Kafka%E6%95%B0%E6%8D%AE%E5%86%99%E5%85%A5%E5%88%86%E5%B8%83%E5%BC%8F%E6%96%87%E4%BB%B6%E7%B3%BB%E7%BB%9F%EF%BC%88%E6%AF%94%E5%A6%82HDFS%EF%BC%89%E4%B8%8A%E7%9A%84%E9%A2%84%E5%86%99%E6%97%A5%E5%BF%97%E4%B8%AD%E3%80%82%E6%89%80%E6%9C%89%E5%BA%95%E5%B1%82%E8%8A%82%E7%82%B9%E5%87%BA%E7%8E%B0%E4%BA%86%E5%A4%B1%E8%B4%A5%EF%BC%8C%E4%B9%9F%E5%8F%AF%E4%BB%A5%E4%BD%BF%E7%94%A8%E9%A2%84%E5%86%99%E6%97%A5%E5%BF%97%E7%9A%84%E6%95%B0%E8%BF%9B%E8%A1%8C%E6%81%A2%E5%A4%8D%0A%0A%E6%B3%A8%E6%84%8F%E4%BA%8B%E9%A1%B9%0Akafka%E4%B8%ADtopic%E7%9A%84partition%E4%B8%8Espark%E4%B8%ADRDD%E7%9A%84partition%E6%98%AF%E6%B2%A1%E6%9C%89%E5%85%B3%E7%B3%BB%E7%9A%84%EF%BC%8C%E5%9B%A0%E6%AD%A4%EF%BC%8C%E5%9C%A8KafkaUtils.createStream()%E4%B8%AD%EF%BC%8C%E6%8F%90%E9%AB%98partition%E7%9A%84%E6%95%B0%E9%87%8F%EF%BC%8C%E5%8F%AA%E4%BC%9A%E5%A2%9E%E5%8A%A0Receiver%E7%9A%84%E6%95%B0%E9%87%8F%EF%BC%8C%E4%B9%9F%E5%B0%B1%E6%98%AF%E8%AF%BB%E5%8F%96Kafka%E4%B8%ADtopic%20partition%E7%9A%84%E7%BA%BF%E7%A8%8B%E6%95%B0%E9%87%8F%EF%BC%8C%E4%B8%8D%E4%BC%9A%E5%A2%9E%E5%8A%A0Spark%E5%A4%84%E7%90%86%E6%95%B0%E6%8D%AE%E7%9A%84%E5%B9%B6%E8%A1%8C%E5%BA%A6%E3%80%82%0A%E5%8F%AF%E4%BB%A5%E5%88%9B%E5%BB%BA%E5%A4%9A%E4%B8%AAKafka%E8%BE%93%E5%85%A5DStream%EF%BC%8C%E4%BD%BF%E7%94%A8%E4%B8%8D%E5%90%8C%E7%9A%84consumer%20group%E5%92%8Ctopic%EF%BC%8C%E6%9D%A5%E9%80%9A%E8%BF%87%E5%A4%9A%E4%B8%AAreceiver%E5%B9%B6%E8%A1%8C%E6%8E%A5%E5%8F%97%E6%95%B0%E6%8D%AE%E3%80%82%0A%E5%A6%82%E6%9E%9C%E5%9F%BA%E4%BA%8E%E5%AE%B9%E9%94%99%E7%9A%84%E6%96%87%E4%BB%B6%E7%B3%BB%E7%BB%9F%EF%BC%8C%E6%AF%94%E5%A6%82HDFS%EF%BC%8C%E5%90%AF%E7%94%A8%E4%BA%86%E9%A2%84%E5%86%99%E6%97%A5%E5%BF%97%E6%9C%BA%E5%88%B6%EF%BC%8C%E6%8E%A5%E6%94%B6%E5%88%B0%E7%9A%84%E6%95%B0%E6%8D%AE%E9%83%BD%E4%BC%9A%E8%A2%AB%E5%A4%8D%E5%88%B6%E4%B8%80%E4%BB%BD%E5%88%B0%E9%A2%84%E5%86%99%E6%97%A5%E5%BF%97%E4%B8%AD%E3%80%82%E5%9B%A0%E6%AD%A4%EF%BC%8CKafkaUtils.createStream()%E4%B8%AD%EF%BC%8C%E8%AE%BE%E7%BD%AE%E7%9A%84%E6%8C%81%E4%B9%85%E5%8C%96%E7%BA%A7%E5%88%AB%E6%98%AFStorageLevel.MEMORY_AND_DISK_SER%E3%80%82%0A%0A%0A%0A%23%23%20Direct%20Approach%0A%E5%AE%83%E4%BC%9A%E5%91%A8%E6%9C%9F%E6%80%A7%E7%9A%84%E6%9F%A5%E8%AF%A2kafka%EF%BC%8C%E6%9D%A5%E8%8E%B7%E5%8F%96%E6%AF%8F%E4%B8%AAtopic%20%2B%20partition%E7%9A%84%E6%9C%80%E6%96%B0offset%EF%BC%8C%E4%BB%8E%E8%80%8C%E5%AE%9A%E4%B9%89%E6%AF%8F%E4%B8%80%E4%B8%AAbatch%E7%9A%84offset%E7%9A%84%E8%8C%83%E5%9B%B4%E3%80%82%E5%BD%93%E5%A4%84%E7%90%86%E6%95%B0%E6%8D%AE%E7%9A%84job%E5%90%AF%E5%8A%A8%E6%97%B6%EF%BC%8C%E5%B0%B1%E4%BC%9A%E4%BD%BF%E7%94%A8kafka%E7%AE%80%E5%8D%95%E7%9A%84%E6%B6%88%E8%B4%B9%E8%80%85API%E6%9D%A5%E8%8E%B7%E5%8F%96kafka%E6%8C%87%E5%AE%9Aoffset%E7%9A%84%E8%8C%83%E5%9B%B4%E7%9A%84%E6%95%B0%E6%8D%AE%E3%80%82%0A%0A1.%20%E7%AE%80%E5%8C%96%E5%B9%B6%E8%A1%8C%E8%AF%BB%E5%8F%96%EF%BC%9A%E5%A6%82%E6%9E%9C%E8%A6%81%E8%AF%BB%E5%8F%96%E5%A4%9A%E4%B8%AApartition%EF%BC%8C%E4%B8%8D%E9%9C%80%E8%A6%81%E5%88%9B%E5%BB%BA%E5%A4%9A%E4%B8%AA%E8%BE%93%E5%85%A5DStream%E7%84%B6%E5%90%8E%E5%AF%B9%E5%AE%83%E4%BB%AC%E8%BF%9B%E8%A1%8Cunion%E6%93%8D%E4%BD%9C%E3%80%82Spark%E4%BC%9A%E5%88%9B%E5%BB%BA%E8%B7%9Fkafka%20partition%E4%B8%80%E6%A0%B7%E5%A4%9A%E7%9A%84RDD%20partition%EF%BC%8C%E5%B9%B6%E4%B8%94%E4%BC%9A%E5%B9%B6%E8%A1%8C%E4%BB%8Ekafka%E4%B8%AD%E8%AF%BB%E5%8F%96%E6%95%B0%E6%8D%AE%E3%80%82%E6%89%80%E4%BB%A5%E5%9C%A8kafka%20partition%E5%92%8CRDD%20partition%E4%B9%8B%E9%97%B4%E6%9C%89%E4%B8%80%E4%B8%AA%E4%B8%80%E5%AF%B9%E4%B8%80%E7%9A%84%E6%98%A0%E5%B0%84%E5%85%B3%E7%B3%BB%E3%80%82%0A2.%20%E9%AB%98%E6%80%A7%E8%83%BD%EF%BC%9A%E5%A6%82%E6%9E%9C%E8%A6%81%E4%BF%9D%E8%AF%81%E6%95%B0%E6%8D%AE%E9%9B%B6%E4%B8%A2%E5%A4%B1%EF%BC%8C%E5%9F%BA%E4%BA%8EReceiver%E7%9A%84%E6%9C%BA%E5%88%B6%E9%9C%80%E8%A6%81%E5%BC%80%E5%90%AFWAL%E6%9C%BA%E5%88%B6%EF%BC%8C%E8%BF%99%E7%A7%8D%E6%96%B9%E5%BC%8F%E5%85%B6%E5%AE%9E%E5%BE%88%E4%BD%8E%E6%95%88%EF%BC%8C%E5%9B%A0%E4%B8%BA%E6%95%B0%E6%8D%AE%E5%AE%9E%E9%99%85%E4%B8%8A%E8%A2%ABcopy%E4%BA%862%E4%BB%BD%EF%BC%8Ckafka%E8%87%AA%E5%B7%B1%E6%9C%AC%E8%BA%AB%E5%B0%B1%E6%9C%89%E5%8F%AF%E9%9D%A0%E7%9A%84%E6%9C%BA%E5%88%B6%EF%BC%8C%E4%BC%9A%E5%AF%B9%E6%95%B0%E6%8D%AE%E5%A4%8D%E5%88%B6%E4%B8%80%E4%BB%BD%EF%BC%8C%E8%80%8C%E8%BF%99%E9%87%8C%E5%8F%88%E5%A4%8D%E5%88%B6%E4%B8%80%E4%BB%BD%E5%88%B0WAL%E4%B8%AD%E3%80%82%E5%9F%BA%E4%BA%8EDirect%E7%9A%84%E6%96%B9%E5%BC%8F%EF%BC%8C%E4%B8%8D%E4%BE%9D%E8%B5%96%E4%BA%8EReceiver%EF%BC%8C%E4%B8%8D%E9%9C%80%E8%A6%81%E5%BC%80%E5%90%AFWAL%E6%9C%BA%E5%88%B6%EF%BC%8C%E5%8F%AA%E8%A6%81kafka%E4%B8%AD%E5%81%9A%E4%BA%86%E6%95%B0%E6%8D%AE%E7%9A%84%E5%A4%8D%E5%88%B6%EF%BC%8C%E9%82%A3%E4%B9%88%E5%B0%B1%E5%8F%AF%E4%BB%A5%E9%80%9A%E8%BF%87kafka%E7%9A%84%E5%89%AF%E6%9C%AC%E8%BF%9B%E8%A1%8C%E6%81%A2%E5%A4%8D%E3%80%82%0A3.%20%E6%9C%89%E4%B8%94%E4%BB%85%E6%9C%89%E4%B8%80%E6%AC%A1%E7%9A%84%E4%BA%8B%E5%8A%A1%E6%9C%BA%E5%88%B6%EF%BC%9A%E5%9F%BA%E4%BA%8EReceiver%E7%9A%84%E6%96%B9%E5%BC%8F%EF%BC%8C%E6%98%AF%E4%BD%BF%E7%94%A8Kafka%20High%20Level%E7%9A%84API%E5%9C%A8zookeeper%E4%B8%AD%E4%BF%9D%E5%AD%98%E6%B6%88%E8%B4%B9%E8%BF%87%E7%9A%84offset%E7%9A%84%E3%80%82%E8%BF%99%E6%98%AF%E6%B6%88%E8%B4%B9kafka%E6%95%B0%E6%8D%AE%E7%9A%84%E4%BC%A0%E7%BB%9F%E6%96%B9%E5%BC%8F%EF%BC%8C%E8%BF%99%E7%A7%8D%E6%96%B9%E5%BC%8F%E9%85%8D%E5%90%88%E7%9D%80WAL%E6%9C%BA%E5%88%B6%E5%8F%AF%E4%BB%A5%E4%BF%9D%E8%AF%81%E6%95%B0%E6%8D%AE%E9%9B%B6%E4%B8%A2%E5%A4%B1%EF%BC%8C%E4%BD%86%E6%98%AF%E6%97%A0%E6%B3%95%E4%BF%9D%E8%AF%81%E6%95%B0%E6%8D%AE%E5%88%B6%E5%A4%87%E5%A4%84%E7%90%86%E4%B8%80%E6%AC%A1%E4%B8%94%E4%BB%85%E6%9C%89%E4%B8%80%E6%AC%A1%EF%BC%8C%E5%8F%AF%E8%83%BD%E4%BC%9A%E4%B8%A4%E6%AC%A1%E6%88%96%E8%80%85%E6%9B%B4%E5%A4%9A%EF%BC%8C%E5%9B%A0%E4%B8%BAspark%E5%92%8Czookeeper%E5%8F%AF%E8%83%BD%E6%98%AF%E4%B8%8D%E5%90%8C%E6%AD%A5%E7%9A%84%E3%80%82%0A4.%20%E9%99%8D%E4%BD%8E%E8%B5%84%E6%BA%90%EF%BC%9ADirect%E4%B8%8D%E9%9C%80%E8%A6%81Receivers%EF%BC%8C%E5%85%B6%E4%B8%ADExecutors%E5%85%A8%E9%83%A8%E5%8F%82%E4%B8%8E%E5%88%B0%E8%AE%A1%E7%AE%97%E4%BB%BB%E5%8A%A1%E4%B8%AD%EF%BC%9B%E8%80%8CReceiver-based%E5%88%99%E9%9C%80%E8%A6%81%E4%B8%93%E9%97%A8%E7%9A%84Receivers%E6%9D%A5%E8%AF%BB%E5%8F%96kafka%E6%95%B0%E6%8D%AE%E4%B8%94%E4%B8%8D%E5%8F%82%E4%B8%8E%E8%AE%A1%E7%AE%97%E3%80%82%E5%9B%A0%E6%AD%A4%E7%9B%B8%E5%90%8C%E7%9A%84%E8%B5%84%E6%BA%90%E7%94%B3%E8%AF%B7%EF%BC%8CDirect%E8%83%BD%E5%A4%9F%E6%94%AF%E6%8C%81%E6%9B%B4%E5%A4%A7%E7%9A%84%E4%B8%9A%E5%8A%A1%E3%80%82%0A5.%20%E9%99%8D%E4%BD%8E%E5%86%85%E5%AD%98%EF%BC%9AReceiver-based%E7%9A%84Receiver%E4%B8%8E%E5%85%B6%E4%BB%96Executor%E6%98%AF%E5%BC%82%E6%AD%A5%E7%9A%84%EF%BC%8C%E5%B9%B6%E6%8C%81%E7%BB%AD%E4%B8%8D%E6%96%AD%E6%8E%A5%E6%94%B6%E6%95%B0%E6%8D%AE%EF%BC%8C%E5%AF%B9%E4%BA%8E%E5%B0%8F%E4%B8%9A%E5%8A%A1%E9%87%8F%E7%9A%84%E5%9C%BA%E6%99%AF%E8%BF%98%E5%A5%BD%EF%BC%8C%E5%A6%82%E6%9E%9C%E9%81%87%E5%88%B0%E5%A4%A7%E4%B8%9A%E5%8A%A1%E9%87%8F%E6%97%B6%EF%BC%8C%E9%9C%80%E8%A6%81%E6%8F%90%E9%AB%98Receiver%E7%9A%84%E5%86%85%E5%AD%98%EF%BC%8C%E4%BD%86%E6%98%AF%E5%8F%82%E4%B8%8E%E8%AE%A1%E7%AE%97%E7%9A%84Executor%E5%B9%B6%E6%97%A0%E9%9C%80%E9%82%A3%E4%B9%88%E5%A4%9A%E7%9A%84%E5%86%85%E5%AD%98%E3%80%82%E8%80%8CDirect%E5%9B%A0%E4%B8%BA%E6%B2%A1%E6%9C%89Receiver%EF%BC%8C%E8%80%8C%E6%98%AF%E5%9C%A8%E8%AE%A1%E7%AE%97%E6%97%B6%E8%AF%BB%E5%8F%96%E6%95%B0%E6%8D%AE%EF%BC%8C%E7%84%B6%E5%90%8E%E7%9B%B4%E6%8E%A5%E5%80%9F%E5%AE%BF%E9%82%A3%EF%BC%8C%E6%89%80%E4%BB%A5%E5%AF%B9%E5%86%85%E5%AD%98%E7%9A%84%E8%A6%81%E6%B1%82%E5%BE%88%E4%BD%8E%E3%80%82%E5%AE%9E%E9%99%85%E5%BA%94%E7%94%A8%E4%B8%AD%E6%88%91%E4%BB%AC%E5%8F%AF%E4%BB%A5%E6%8A%8A%E5%8E%9F%E5%85%88%E7%9A%8410G%E9%99%8D%E8%87%B3%E7%8E%B0%E5%9C%A8%E7%9A%842-4G%E5%B7%A6%E5%8F%B3%E3%80%82%0A6.%20%E4%B8%8D%E4%BC%9A%E5%87%BA%E7%8E%B0%E6%95%B0%E6%8D%AE%E5%A0%86%E7%A7%AF%EF%BC%9AReceiver-based%E6%96%B9%E6%B3%95%E9%9C%80%E8%A6%81Receivers%E6%9D%A5%E5%BC%82%E6%AD%A5%E6%8C%81%E7%BB%AD%E4%B8%8D%E6%96%AD%E7%9A%84%E8%AF%BB%E5%8F%96%E6%95%B0%E6%8D%AE%EF%BC%8C%E5%9B%A0%E6%AD%A4%E9%81%87%E5%88%B0%E7%BD%91%E7%BB%9C%E3%80%81%E5%AD%98%E5%82%A8%E8%B4%9F%E8%BD%BD%E7%AD%89%E5%9B%A0%E7%B4%A0%EF%BC%8C%E5%AF%BC%E8%87%B4%E5%AE%9E%E6%97%B6%E4%BB%BB%E5%8A%A1%E5%87%BA%E7%8E%B0%E5%A0%86%E7%A7%AF%EF%BC%8C%E4%BD%86Receivers%E5%8D%B4%E8%BF%98%E5%9C%A8%E6%8C%81%E7%BB%AD%E8%AF%BB%E5%8F%96%E6%95%B0%E6%8D%AE%EF%BC%8C%E6%AD%A4%E7%A7%8D%E6%83%85%E5%86%B5%E5%BE%88%E5%AE%B9%E6%98%93%E5%AF%BC%E8%87%B4%E8%AE%A1%E7%AE%97%E5%B4%A9%E6%BA%83%E3%80%82Direct%E5%88%99%E6%B2%A1%E6%9C%89%E8%BF%99%E7%A7%8D%E9%A1%BE%E8%99%91%EF%BC%8C%E5%85%B6Direct%E5%9C%A8%E8%A7%A6%E5%8F%91batch%E8%AE%A1%E7%AE%97%E4%BB%BB%E5%8A%A1%E6%97%B6%EF%BC%8C%E6%89%8D%E4%BC%9A%E8%AF%BB%E5%8F%96%E6%95%B0%E6%8D%AE%E5%B9%B6%E8%AE%A1%E7%AE%97%E3%80%82%E9%98%9F%E5%88%97%E5%87%BA%E7%8E%B0%E5%A0%86%E7%A7%AF%E5%B9%B6%E4%B8%8D%E4%BC%9A%E5%BC%95%E8%B5%B7%E7%A8%8B%E5%BA%8F%E7%9A%84%E5%A4%B1%E8%B4%A5%E3%80%82%0A%0A%E5%9F%BA%E4%BA%8Edirect%E7%9A%84%E6%96%B9%E5%BC%8F%EF%BC%8C%E4%BD%BF%E7%94%A8kafka%E7%9A%84%E7%AE%80%E5%8D%95api%EF%BC%8CSpark%20Streaming%E8%87%AA%E5%B7%B1%E5%B0%B1%E8%B4%9F%E8%B4%A3%E8%BF%BD%E8%B8%AA%E6%B6%88%E8%B4%B9%E7%9A%84offset%EF%BC%8C%E5%B9%B6%E4%BF%9D%E5%AD%98%E5%9C%A8checkpoint%E4%B8%AD%E3%80%82Spark%E8%87%AA%E5%B7%B1%E4%B8%80%E5%AE%9A%E6%98%AF%E5%90%8C%E6%AD%A5%E7%9A%84%EF%BC%8C%E5%9B%A0%E6%AD%A4%E5%8F%AF%E4%BB%A5%E4%BF%9D%E8%AF%81%E6%95%B0%E6%8D%AE%E6%98%AF%E6%B6%88%E8%B4%B9%E4%B8%80%E6%AC%A1%E4%B8%94%E4%BB%85%E6%B6%88%E8%B4%B9%E4%B8%80%E6%AC%A1%E3%80%82