According to educational theories, you're actually helping yourself to learn and remember the material in the hard read (the easy read, too) by doing this. Swapping books forces you to try to pick up the strings of thought when you start reading the book again, which forces you to remember what you were thinking when you stopped reading. As you get back into the book, you'll remember more and more of those strings. The next time you put the book down, you'll be able to remember more about it as a result.