I found the answer: the little drawer icon on the right is the 'mark read' icon. Click that and then you can select what to mark read. Clicking 'all' will mark everything read.