If the URLs didn't change when you changed that setting, change Wiki URL Format back to the first option, then change it back to your desired choice.
And I think I understand what you mean now. Under the area's list of pages, you want it to only show items that are book-type, and not other types, but you still want to allow non-book-types to be created there? There is not currently a way to apply a default filter (or any filters) on those types of content lists. You will have to request such a feature by opening a feature-request, or you should move the chapter-pages to a sub-area that is hidden from users by setting a display order of 0.