TL;DR
Changing your method of reporting will not actually address the underlying process issues implied by your original problem statement. Your old reporting process was actually the better one. You should go back to it, but adapt your team's process to optimize for achieving the Sprint Goal rather than for velocity or trend-line reporting.
Remaining Time: An Often Misleading Metric
And then, we were using the Sprint Burndown chart in JIRA, set to show Remaining Time instead of Remaining Story Points. By doing so, the Burndown chart was going down and was really showing progress. If we were to display the chart by Remaining Story Points, it would've been a flat line until almost the end of the sprint.
This is a tool-driven project smell. You're "hours remaining" is really just a variation of consumed story points, which is a misfeature I've discussed elsewhere. In other words, hours remaining is based on initial time estimate - hours expended but there's actually no guarantee in your system that the end result will equal zero. Put another way, you have three possible states with your current process:
initial time estimate - hours expended < 0
You've overestimated your story. This isn't a tragedy, but may impact the resources available for other stories/tasks.
initial time estimate - hours expended = 0
You correctly estimated your story. Yay!
initial time estimate - hours expended > 0
You underestimated your story. It is likely it will be incomplete at the end of the iteration.
Only in the second case will your "percentage complete" calculation really be accurate. The rest of the time, you are simply creating a false impression of progress where "x% of the work is y% done." Don't do that!
Work in Progress Limits and "Swarming"
Your problem is a process problem, and is likely being caused by ignoring (or simply not having) Work-in-Progress (WIP) limits. While a team can work on more than one story at a time if the stories meet INVEST criteria and are independent of one another, having team members working on stories independently of one another (instead of as a team) often defeats the purpose of having a cross-functional team. It also frequently results in other queuing and integration problems, especially towards the end of a Sprint.
In general, team members should be actively collaborating on stories, which means that as a rule of thumb your WIP limits should be around half your team size. Given a team of seven people, plus or minus two, that means your WIP limit should be roughly three stories in progress at any one time.
Since the team only succeeds if the Sprint Goal is met, doing more stories as once doesn't actually benefit anyone. It's much better to use techniques like pairing or swarming (where the whole team "swarms" over a story) to maximize resources allocated to getting a story to 100% of the Definition of Done as quickly as possible. When you work this way, your burn-down chart will more accurately show user stories or story points completed towards the Sprint Goal, and the chart will correctly burn down, up, or flatlined in a more sensible way throughout the Sprint.
NB: Remember, even if you don't complete 100% of the stories, you may still meet the Sprint Goal! The point of a Sprint isn't to inflate velocity, it's to deliver on the Sprint Goal. Optimize for that instead.
Decompose Stories and Tasks Further
If you're dealing with user stories that span the whole Sprint, they're probably too large. However, that's a judgement call; it's certainly possible to have larger stories, so long as the story fits into a single Sprint.
Tasks, however, should be roughly 1/2 to 2 days in size so that they are clearly done or not-done. If a task isn't on-track to be done within the time-box, it's worth reviewing the task to see if it needs:
- Re-estimation of the task or story.
- If there's hidden or unplanned work that wasn't accounted for.
- If there are resource constraints or process bottlenecks that are impacting delivery of the task.
- If there are unexpected dependencies.
- Replanning, either within the Sprint if it's minor and won't impact the Sprint Goal, or as part of an Early Termination and a return to Sprint Planning.
While stories can theoretically span an entire Sprint, tasks that require extensive periods of time are most often the result of improper granularity and a lack of task-level planning during the second half of Sprint Planning or subsequent just-in-time planning sessions within the Sprint. While implementation decisions should always be deferred as late as possible, Scrum doesn't do away with task-level planning; it just postpones it until the last responsible moment.
Use Metrics Properly
Finally, remember that velocity is a capacity-planning metric, not a measure of productivity. Likewise, burn charts are tools for identifying process bottlenecks or hidden process problems, not status reporting or task-level schedule estimation tool.
What the chart actually tells you is whether you are likely to complete all planned work within the time box, or whether your estimation or planning is out of reasonable tolerance. In other words, it's a check on your process, not a measure of productivity! In your specific case, the old burn-down chart was clearly telling you that you were not completing stories with a reliable cadence, and were likely experiencing "end of Sprint integration madness" as all the suddenly-completed stories were marked done and then (hopefully) integrated at the last minute.
Recommendations
I would strongly recommend going back to the old burn-down chart, and revising your process to incorporate WIP limits and swarming to fix the process problems, rather than using a new (and potentially misleading) reporting tool because you like the trend lines better. Fix the process, because working features are the real goal, not trend lines or pretty reports.
