pyspark.sql.datasource.DataSourceStreamReader.reportLatestOffset#

DataSourceStreamReader.reportLatestOffset()[source]#

Returns the most recent offset available. The information is used to report the latest offset in the streaming query status. The source can return None, if there is no data to process or the source does not support to this method.

New in version 4.2.0.

Returns

dict or None: A dict or recursive dict whose key and value are primitive types, which includes Integer, String and Boolean. Returns None if the source does not support reporting latest offset.

Examples

>>> def reportLatestOffset(self):
...     return {"partition-1": {"index": 100}, "partition-2": {"index": 200}}