I have a strong preference for smaller stories, though I sometimes struggle to make them smaller. Because of that (and the desire to be flexible) I'm willing to entertain other options. Though everytime I allow myself to do larger stories I keep coming back to wishing for smaller stories. So here's my attempt to expose my reasoning and consider what prompts moving away from smaller stories.
How I imagine stories being broken down to smaller ones has always been a whole team iterative approach. During a grooming session the Product Owner, after ensure things are good for the upcoming sprint, will bring up an upcoming story with the team. The team asks questions and works with the Product Owner to refine the acceptance criteria and figure out if there are some significant outstanding questions that need to be answered. Out of this discussion some smaller stories might be created to help answer the questions that came up or perhaps do some part of the work that can be done in isolation - or maybe there's enough details to get a skeleton of work that will provide additional information. While some stories might have been carved off, the rest of the story still remains and stays further down in the backlog waiting for another time to revisit. It can take several sessions for a story to be broken down and it's even possible that a story that was initially broken off was just a clean separator to reduce risk and now it needs to be broken down further. Keep on a regular cadence of viewing the story until it's been broken down enough and gets to done.
Reality doesn't always give you the time to revisit a story multiple times - sometimes the work is the most critical thing you can do. However, the sentiment of the above thinking seems like it should still be there - just shorter iterations.
Benefits I see with smaller stories...
Less variance
The more things there are involved in a single story the more variance there will be. It just seems that you're more likely to be able to see everything that needs to get done when there's less to do. With a large story it's much more likely that you'll gloss over parts of it thinking of it as one thing, but once you start digging into it you realize there's more to unpack. That's not to say that smaller stories are a cure all of this - we're human and will miss things - it's just trying to help decrease the variance.
I find variance important because it helps with developing a cadence - this ties into morale which I'll touch on later.
Greater understanding of the story
Having multiple conversations around a story allows for repeating the details. This might play into repetition learning, but not every story keeps the same focus. Some need to change as questions are answered or feedback is obtained. I think the greater value is the ability to get your subconscious in on the problem. This isn't always going to pay off, but giving the opportunity for it to surface other connections is important.
Even without either of those in play by having to break things down you're required to get a better understanding of the problem. Maybe even shining a light into some areas that would've been problematic, but you couldn't see until you got to smaller chunks of work.
The biggest counter I have to this thinking is the concern of distracting the team or wasting their time. It certainly wouldn't be appreciated if a stories keep getting shelved after they go through a couple iterations. I also don't think having a handful of stories in this process would be good. Though I've only experienced it with one story and I think that there's room for some experimentation with this.
Faster feedback
Feedback doesn't just come from the end user. It also comes in the form of passing tests, integrated code, and even non-broken code. I completely agree that it is important to keep track of what is important to the end user and releasing something that doesn't provide them with what they want shouldn't be announced. That doesn't mean it shouldn't be released to get what feedback there is for it.
I've met many people who think that Continuous Deployment is a goal worth attaining. At the heart of this is the need for small batches (various articles from: Atlassian, SD Times, SAFe) so that the release is quick and easy to troubleshoot in case of issues. For me this all ties together - Continuous Deployment is just another way to get faster feedback on your code in production.
Morale
How many times have you had a story carry into another sprint and while you knew there were good reasons why it didn't get closed you were just sick of working on it. Yes, you found an interaction that you hadn't considered and it needs to be tested. Yes, it would be better if you refactored it to be more encapsulated. Everything makes sense to your rational mind, but you're just done with it and need a break.
I don't recall having this feeling with small stories. Whenever I've been able to break things down I've felt a greater sense of movement and less dread about the release of the work. I'll concede that your milage may vary and this is really more of a gut feeling - though I know of several people who've lamented the carry over story and how much it stinks.
The flip side...
New story is top priority
While this certainly makes it harder to walk away from and let it simmer, you could just make faster iterations. Maybe visit the story in the morning and revisit in the afternoon or the next day. I don't think this changes the process - just makes it easier to fall back to previous habits. :)
Breaking up (a story) is hard...
When I'm thinking of breaking up a story I want something that can show progress, but not necessarily something that should be turned on for customers. This is something I still feel I require practice at. A flowchart was shared with a couple teams I've been on, but I'll confess to feeling intimidated by it and never giving it a fair shake. A different take is Tracer Bullet Development, which tries framing in the overall architecture and letting the details be filled in later. Another example that I suspect many will find extreme (I haven't tried it, but keep wanting to) is breaking things down to around one hour stories, by focusing on acceptance criteria.
In the end, this really comes down to the fact that to get the work done you need to break it down regardless. You either do it ahead of time with smaller stories or you do it later when you're splitting up the tasks (...or I guess you can even do it later when you're doing the coding). Given the reasons I like smaller stories makes me think I want to figure this out and not back away from it.
Managing the additional stories
This one intimidates me. Having stepped into the Product Owner role once and trying to keep stories small and up to date, I didn't enjoy this. Which, incidentally, is why I've never pushed for one hour stories as I linked to above. The idea of trying to manage that number of stories seemed ridiculous - which makes me think I'm still missing something or making something too complex.
Though, at its heart, this seems more of a tradeoff between managing stories or managing tasks. This seems to be driven because managing tasks is easier than stories - they are often created ad-hoc by the team, where stories are often managed in some application. Maybe it would be good to explore that and consider if there is a way to make things easier to manage?
Another variation on this challenge is stopping a story when it should be stopped. I've had a number of cases where the team gets on a roll and realizes that they forgot to finish a story and the code is now blended with the next one. In some cases this is because the stories aren't well broken down and to be able to release the code it required the additional work. Other times it's excitement or just trying to help out the team by working on the next item. While this isn't against small stories per se, it does create confusion and make people wonder why they shouldn't keep things as one big story to avoid this.
Conclusion
The reality is that not every story needs this type of treatment. Some are straight forward and have little ambiguity that would prevent them from being worked on right away. It's those other ones that require you to be a little more thoughtful on how you get them started.
I'm sure I'm not covering all of the thinking around this topic, but this is what's currently bouncing around my head. Hopefully it will provoke some comments or get me to talk with others about it, so I can continue to challenge my built in preference.
